A brand new game
for Windows, HTML5, 3DS Homebrew, iPhone, Android TV and more,
28th October, 2018
Last night I rejigged my IP Banning settings a little, to catch any spambots in their tracks.
Between 6:20 and 6:26 this morning, BingBot decided to do it's daily update, and requested over 800 pages of content within that short amount of time.
Woah there, BingBot!!!
Over the course of those few minutes, it managed to get itself temporarily locked out from my server on multiple occasions, and each time it went "Oh, I broke something.. I'd better try that again with a different IP address", and then continued, repeatedly getting itself locked out, and repeatedly switching up IP addresses to continue its crawl.
This morning, I added a crawl-delay to the robots.txt file. I'm not 100% sure how well that will actually work.. Is it suggesting "crawl a bit slower, you idiot", or is it suggesting "crawl after x days" at which point it'll go "RAWR!!" and try mass-reading all the data once again..?
The stuff I'm reading doesn't seem to specify which of those two it is.
So, yeah, between inadvertently blocking Bing and successfully blocking another spambot, it's certainly been an eventful night.