SEO crawlers and site audit tools can trigger MODX Cloud's Web Application Firewall (WAF) because their behaviour—rapidly crawling pages, opening many concurrent connections—can look similar to automated attacks. This guide explains what to expect and how to avoid disruptions.
Why these tools get blocked
The WAF protects all MODX Cloud sites from malicious bots, scrapers, and vulnerability scanners. SEO and audit tools often share characteristics with these threats:
- They make large numbers of requests in a short time
- They open multiple concurrent connections
- They follow links automatically, similar to how some attack tools probe for vulnerabilities
- They sometimes use generic or unrecognised user-agent strings
This doesn't mean the tools are doing anything wrong—it's a byproduct of how they work.
Tools we've already allowlisted
We've already configured access for many commonly used SEO and audit services, particularly those that publish static IP lists or provide a publicly accessible IP list endpoint we can poll on a schedule. If your tool is in this category, it should work without any action on your part.
If you're using a hosted tool and aren't sure whether it's already allowlisted, contact support and ask. Include the tool name and the URL of the site you're auditing.
Tools that may need manual allowlisting
Some tools don't publish static IP lists or have IP ranges that change frequently. These may need to be handled differently.
Desktop clients (e.g. Screaming Frog) We can configure access using a custom user-agent string that you set in the tool. Contact support for instructions specific to your tool.
Hosted tools with no published IP list We can often configure access using a custom user-agent string if the tool supports it. Contact support.
Site search services (e.g. Elasticsearch-based tools) If the service uses a unique user-agent string per client or hostname, we can allowlist it. Contact support and include the URL of the site and the name of the service.
How to request allowlisting
We recommend contacting us before running large crawls or audits. It's faster than dealing with a blocked scan mid-run.
Email support@modxcloud.com with the following:
- The name of the tool or service
- The URL of the site you're auditing
- Static IP addresses or a link to the tool's published IP list, if available
- The user-agent string the tool uses, if you know it
We'll confirm what we can configure and let you know if we need anything else. Most requests are handled quickly.
Custom user-agent strings
For desktop tools like Screaming Frog that allow you to set a custom user-agent string, using a unique value makes it easier for us to reliably identify your traffic. When contacting support, we can advise you on a suitable format.
Note: user-agent strings are spoofable, so they work best as part of a broader allowlisting approach rather than as the only identifier. We use them in combination with other signals.
General tips for running scans
- Reduce crawl speed and concurrency settings in your tool where possible—most tools let you throttle requests
- Schedule large crawls for off-peak hours where you can
- Contact support in advance if you're running a particularly large or fast audit
Already blocked?
If you're seeing a "Security Verification Required" screen or your scan has been interrupted, email support@modxcloud.com with your IP address (or a screenshot of the block screen) and the URL you were auditing. We can restore access and configure ongoing allowlisting at the same time.