How to crawl a staging, test or development sites with Screaming Frog



Ignore robots.txt. Your dev environment will probably be disallow

Setup at: Configuration > robots.txt

Might be useful to set up a custom robots.txt

Timeout & Errors

Set response timeout higher. Staging, test or dev servers are usually slower. In addition be patient with 5xx errors

Setup at: Configuration > Spider > Advanced

If you want to render JavaScript AJAX timeouts should be be higher, too.

Setup at: Configuration > Spider > Rendering

Go slow

Ask your devs what the environment can handle. Default is 5 Threads which should be fine in most cases.

Setup at: Configuration > speed

© 2020 Tobias Willmann