Where have my robots.txt controls gone!?
On October 19, 2017, we have simplified how we handle and serve requests for robots.txt. How your Cloud responds to robots.txt requests is no longer configured in the MODX Cloud dashboard.
All new Clouds from this date will have the new behavior by default.
Any Clouds created prior to this date will continue to behave as previously configured, provided there have been no changes to the Cloud's Web Rules or Domains/Subdomains.
When an older Cloud's Web Rules or Domains/Subdomains are updated, it triggers a change to the webserver configuration which updates the of handling robots.txt requests to the new way. This may change the robots.txt handling behavior of your Cloud.
If you rely on blocking robots from one or more hostnames, while allowing robots on others, particularly if you are not utilizing redirects to force a canonical domain, please be aware that this behavior is no longer controlled from the MODX Cloud dashboard, and previous preferences may not translate to the new configuration. When in doubt, visiting the different robots.txt URLs in your browser is a great way to test the behavior. Please read the article linked above to understand the changes you need to make.
If you have any questions, we'll be happy to provide further guidance. Just click the Help button in the MODX Cloud dashboard to open a support ticket.