Commit e40969d
Fix: Add User-Agent to redirect checker
Some sites (e.g. atlasmag.dk) block requests without User-Agent,
causing "Could not get url after redirects" errors in crawler.
Also added timeout to prevent hanging.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>1 parent 1910d39 commit e40969d
1 file changed
+3
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
177 | | - | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
178 | 180 | | |
179 | 181 | | |
180 | 182 | | |
| |||
0 commit comments