NEW: Adjustable Crawling Speed

How Semonto crawls your website

One of the features of Semonto is detecting broken links and mixed content. Such links can harm the user experience of your website visitors, your SEO impact and more. To detect broken links or mixed content, Semonto uses a crawler that fully crawls, reads, and searches all pages of your website. We are doing this at a slow rate to ensure we don’t impact your website by performing too many requests at the same time. A high crawling speed could result in a slower website.

NEW: adjust the crawling speed

We received feature requests to allow users to control the crawling speed. While we already scan at a low rate, there are some rare cases in which you want to scan even slower. For example, when your website is handling a spike of traffic. Or when you simply want to reserve as many resources as possible for regular website visitors.

How it works

• In Semonto, go to your monitor.
• Go to the Broken Links (or Mixed Content) section.
• Hit the settings.
• Select a slower rate limit. This reduces the number of requests per second that Semonto can make to your server while scanning for broken links.

When it can be useful to adjust the crawling speed

Errors like HTTP 429 might indicate that Semonto is making too many requests. Adjusting the crawl speed in his setting could be a solution. If you are hosting your website on CloudFlare, you can also check the following guide: How to monitor Cloudflare sites

Bonus tip: You can also monitor your web servers’ load using our server health monitoring. This allows you to set a maximum load or disk space usage for your server and receive an alert when the threshold is crossed.

Let us know what you think!

We hope that this added setting is useful for you, too. If you have any suggestions for improvement or any other feature request, we would love to hear about it. We appreciate the feedback from our community and are committed to tailor Semonto to your needs.