A few weeks ago, I was looking at my site statistics, and discovered I was suddenly vastly more popular than I had ever been. From my previous position of around 400-500 hits per day, I was getting somewhere in the region of 2000. Wow. Well, it’s nice to be popular. However, when I looked at my site statistics, it became apparent that about 55% of the traffic I was getting was from webcrawlers.

Now I’ve got nothing against Google, Yahoo, MSN etc crawling my site, because frankly I want to get nice high listings in their search engines, but I was suspicious that of the web crawlers, about 80% of them were unknown. Now, it could just be that my free stats package is failing to recognise some perfectly legitimate bots. That seemed perfectly plausible but I reckoned it was also entirely possible that some unscrupulous web crawler chappies had come across my site from somewhere and were frantically crawling all over it, presumably either to try and hijack my email form, give me lots of comments telling me where I could buy tramadol (I don’t want any, okay? I don’t even know what it is, to be honest), or be scouring my site for email addresses and or private things.

Now Google, Yahoo, MSN and most legitimate bots will honour the robots.txt exclusion protocol, so one way you can identify a bad bot is simply if it doesn’t honour this. In fact, the only legitimate organisation I’m aware of that has a bot which doesn’t honour the exclusion protocol is the Sitemorse testing bot, for which the theory is apparently that it is testing all content for accessibility, and it won’t allow you to exclude some sections (although not everyone is happy its behaviour, and indeed it prompted Dan Champion to write an article on blocking sitemorse and other unwanted robots). Some sites have even set up bot-traps to generate random links into excluded areas, which in turn generate more random links, so the good bots following the exclusion protocol don’t go in, and the bad bots go round in circles. But that’s still using up your bandwidth.

What I wanted was a method which would just cut out a look of this crap. And then I found the Bad Behaviour plug in for WordPress, which does just the job. It analyses the requests to see if they are likely to be a bad bot (are they breaking the exclusion protocol, are they claiming to be google when they aren’t etc) and returns just a simple screen telling them why their access has been denied — and what to do if they are legitimate. In the first week, this blocked around 90 bad bots visiting my site. Every week since, it’s blocked about 60, as it seems that many bad bots aren’t returning once they’ve been kicked out the first time.

Of course, it’s possible that I’m blocking some legitimate users too, since someone could have changed their user agent string to make it look like they are googlebot in order to get more content from a site (some sites serve additional content only to registered members, but also serve it to googlebot in order to get higher rankings; cloaking as googlebot therefore is naughty but would get you access to all the content). However, in these cases the error message "you are claiming to be a major search engine but do not appear to be" is sufficiently descriptive to let people know what the problem is.

On the whole, I’ve been getting about 15% less page hits since I’ve installed Bad Behaviour, and the only category showing a reduction is the unidentified bots — the actual browser based traffic seems to have increased slightly. So there you go, install bad behaviour, get rid of the crap and don’t put off your regular visitors.