The importance of Blocking Bots

[Builder_ok]

Hello Builders,
Just wanted to understand how important in the very beginning point of a website to block popular bots that might be used for spying?

Also, how would it impact my ability to utilize these tools, e.g Ahrefs

 

[Ryuzaki]

Let's use Ahrefs as an example since you chose it. If you block their bots from crawling your site, you won't be able to use their all their tools that concern your site. That means their audit features, looking at your site structure, how all your internal links are set up, etc. This is the crap that your competitors don't really care about anyways and if they want it they'll get it in some way. You won't be able to block every spidering tool in existence, especially those where you can change the user-agent, etc.

What you won't be able to stop is their ability to crawl other sites that link to you (your backlink data). I'm assuming this is what 99% of people are concerned about. They'll crawl all the sites that link to you and list those links and expose those pages on your s ite in their database, etc.

It's kind of fruitless endeavor, which is why there's so many little spam services out there to spam your backlink profile with gobs of nonsense and then to get Ahrefs to crawl those and pick them up. Blocking bots to hide backlinks is so futile that you're better off obfuscating your links by spamming yourself, but then you render that data worthless for your own use too, and it's extremely useful data.

The bots you want to be concerned with are people trying to find open ports on your server, get into your XML-RPC, do MySQL injections, and all that crap. And you're better off trying to block IP ranges from Amazon AWS, but good luck finding and blocking every possible IP address that these people use. They'll just rotate through a zillion residential proxies.

I feel like it's all pretty useless to be honest and you're better off just hardening your security or hiring a hosting company that manages all of this and does so with a great track record.

 

[Darth]

FWIW if you have paid ahrefs sub and verify your domain with them, they will ignore any robots.txt.