Best life hack to get rid of most of Google’s bots from your website:
map $http_user_agent $block_user_agent {
default 0; # do not block by default
~*"(Nexus 5X)"* 1;
}
server {
# Block requests with matching User-Agent
if ($block_user_agent) {
return 403;
}
}
Don’t worry, nobody uses a Nexus 5X anymore.
For some reason in the past 1-2 years Google bots have stopped abiding to the gentlemen’s convention of using the+
http://www.google.com/bot.html string somewhere in their UA and they just dump you user agents like this where they pretend to be a 10-year-old Android device:
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; GoogleOther)
And, since apparently in some cases they also stopped abiding to the gentlemen’s agreement of respecting the robots.txt, this is probably a more explicit way of keeping them at bay.
