How to monitor Cloudflare sites
Is Cloudflare blocking our crawlers, preventing you from monitoring? Here's what to do next.
Is Cloudflare blocking our crawlers, preventing you from monitoring? Here's what to do next.
Contents
Cloudflare can help you make your website faster, more secure, and more reliable by acting as a protective shield. There is just one catch: sometimes Cloudflare blocks something that you actually need, like the crawler of your monitoring tool. And a monitoring tool is essential if you want to stay in control of the technical health and availability of your website. Is Cloudflare preventing you from monitoring with Semonto? We’ll walk you through what to do next.
Semonto frequently pings your website to verify whether it is still reachable. By default, we perform this test every five minutes. Depending on your Semonto plan, you can increase or decrease the testing frequency to an interval ranging from one minute to several hours. In addition to reachability testing, we also crawl your website every few days to detect broken links or mixed content.
A first sign could be that you get a seemingly false notification from Semonto that your website is down. In other words: you get a notification from us, but you have no trouble opening the site. If that happens, go to the Reachability report in Semonto and have a look at the HTTP status code. Status codes HTTP 403 or HTTP 999 mean “Forbidden - access denied” and indicate that we were not allowed to monitor the URL. This does not automatically mean that we were blocked by Cloudflare (the access can also be restricted by other sources), but it can be an indication.
When we crawl a webpage, we include a user agent in the HTTP header of the request. A user agent is a string of text identifying the software used to access a website. In other words: by clearly identifying ourselves in your logs, we increase the chance that Cloudflare recognizes us and treats us as legitimate traffic. In plain words: user agents tell Cloudflare: ‘Hey, it’s us. We are here to monitor. We are not a threat. No need to block us.’
This is what our user agent looks like:
Mozilla/5.0 (compatible; Semonto/2.0; +https://semonto.com/monitoring; websiteMonitoring)We always monitor from a fixed list of IP addresses (if they do change, we will let you know). You can whitelist our IP addresses to decrease the chance of them being blocked by the Cloudflare firewall.
This is our list of IP addresses in multiple formats:
Cloudflare rules can be created to control the access to your website. Those can be based on the User Agent as explained above, or via the origin IP addresses. We created a CloudFlare WAF expression rule that can be used to directly whitelist Semonto's IP addresses.
Are you still experiencing issues after implementing our suggestions? Let us know! We are always open to brainstorming with you to further improve your experience with Semonto.
