Bad Bot, Semrush Bot

Semrush is an SEO tool that at one time in my online career I might have liked. Semrush as an SEO tool is interesting, they seemed to be able to show information that would be difficult to get from other places. It makes more sense how they get that data when you look at what the Semrush bot does.

Officially from the Semrush bot website the Semrush bot does the following:

Data collected by SemrushBot is used for:

  • the public backlink search engine index maintained as a dedicated tool called Backlink Analytics (webgraph of links)
  • the Site Audit tool, which analyzes on-page SEO, technical and usability issues
  • the Backlink Audit tool, which helps discover and clean up potentially dangerous backlinks of your profile
  • the Link Building tool, which helps you find prospects, reach out to them and monitor your newly acquired backlinks
  • the SEO Writing Assistant tool to check if URL is accessible
  • the On Page SEO Checker and SEO Content template tools reports
  • the Topic Research tool reports
  • the SplitSignal tool to create SEO A/B tests on your website
  • the ContentShake AI tool reports
  • the Plagiarism Checker tool, to verify whether the content being checked appears on your website

Let’s be clear, Semrush is crawling your website and then offering that data, your data, to their PAID subscribers so your COMPETITORS can use that data against you. Think about that for a second, allowing the Semrush bot will allow your competitors to better compete against you.

If that is not enough of a reason to ban the Semrush bot consider this. I checked the logs for one of my websites that gets a reasonable amount of traffic. The Semrush bot accounted for 10% of the requests to that website during the last seven days. 10%! That means not only would you be providing your data to Semrush for your competitors can use it against you, but 10% of your hosting bill is being spent providing it to Semrush! For comparison Google bot, a bot that actually will provide you with something by sending visitors to your website, accounted for only 1.5% of all requests to the same website. Bing during the same time was a little higher at 2% of the requests.

You can block the Semrush bot using the robots.txt file according to their website, and I have no evidence to say it will not work. I can tell you that the Semrush bot does not get the hint when presented with a 403 forbidden response since the website I got the statistics from with Semrush bot being 10% of all requests returned ONLY a 403 forbidden response to Semrush bot.

Categories: Semrush bot SEO