A brand new game
for Windows, Linux, Mac, HTML5, 3DS Homebrew and more
every week,
unless exhausted!
About Games Music Pixelart Poetry Home Patreon Scores JNKPlat Petit Switch
Blog
Poor Bing!
  28th October, 2018
Last night I rejigged my IP Banning settings a little, to catch any spambots in their tracks.

Between 6:20 and 6:26 this morning, BingBot decided to do it's daily update, and requested over 800 pages of content within that short amount of time.
Woah there, BingBot!!!

-=-=-

Over the course of those few minutes, it managed to get itself temporarily locked out from my server on multiple occasions, and each time it went "Oh, I broke something.. I'd better try that again with a different IP address", and then continued, repeatedly getting itself locked out, and repeatedly switching up IP addresses to continue its crawl.

This morning, I added a crawl-delay to the robots.txt file. I'm not 100% sure how well that will actually work.. Is it suggesting "crawl a bit slower, you idiot", or is it suggesting "crawl after x days" at which point it'll go "RAWR!!" and try mass-reading all the data once again..?
The stuff I'm reading doesn't seem to specify which of those two it is.
hmmm..

So, yeah, between inadvertently blocking Bing and successfully blocking another spambot, it's certainly been an eventful night.

Views 28, Upvotes 1
Daily Blog
Site credits : Jayenkai, one crazy fool who has far too much time on his hands.
(c) Jayenkai 2017 and onwards. RSS feed
Blog - Poor Bing! - AGameAWeek