Introduction
This article gives answers to some frequently asked questions about the Monsido crawler.
What is monsidobot?
Monsidobot is the name of Monsido’s web crawler. It is used to scan websites requested by clients as part of our offering.
I’m not a client. Why am I seeing requests from monsidobot?
This is happening as one or more of our clients has links to your website and we’re making requests to determine if the link(s) is/are still valid.
For non-client links the crawler will try to make a HEAD request in order to be as resource frugal as possible. Unfortunately, not all websites support HEAD, and so for a range of non 2XX response codes the crawler will try to determine if the link is actually in a non-functional state or if the website does not support HEAD requests.
Why is monsidobot making a request even though the link is disallowed in robots.txt?
The request is made because the purpose is to verify the status of the link. Unless you are a client Monsido has no interest in the content of Your website and will not save nor use it in any way outside the response code for verification of the status of the link.
Notes
Monsidobot generally respects crawl-delay in robots.txt as long as the value is between 0 and 60. 60 will be used for any higher value.
For any other questions or concerns please contact us at monsidobot@monsido.com.
For information about how to set up and configure Monsido scans, see the User Guide article:
Configure Domain Scans.
Additional Resources
For definitions and explanations of acronyms and abbreviations used in the Monsido User Guide, see:
For further assistance regarding Monsido, contact the Monsido support team at support@monsido.com or use the Monsido chat and help features inside the application.