What's a User Agent String and why should I care about it?

A user agent string is the string that is presented to the server by your browser when you request a web resource. Different user agent strings can therefore be used to identify whether you are using Internet Explorer, Mozilla Firefox, Opera, and also which version in most cases.

This should be of no concern to the web developer, because you should be looking to present your standards compliant site to every browser, rather than returning back to the bad old days of this site is best viewed with Internet Explorer.

However, people browsing your site may make up only a fraction of the traffic. The remainder is by automated robots that spider through a site. Some, like googlebot, honour the robots exclusion protocol, others may not – for example they may be spam harvesters (sometimes termed spambots), which seek to track down every unprotected email address on your site in order to flood them with spam. Alternatively, there may a legitimate organisation which do not honour the robots exclusion protocol (such as the SiteMorse testing spider) and you may wish to block that.

Well what's the IP Address thing, then?

An IP address, or Internet Protocol address, is a 32-bit numeric address made up of four numbers from 0 to 255 separated by full stops. You'll have seen them, they look something like 115.22.1.28. Companies which have their own internet connections tend to have a fixed IP address range, thus allowing them to be identified by that IP address. For example, over 1000 edits were made to Wikipedia articles from a range of addresses linked to the US House of Representatives and the Senate, leading to the blocking of some of these IP addresses; or you could read about how Dyslexic Duncan was exposed as an employee trying to promote a company product.

In other words, it's an alternative method of identifiying who is asking for your web resource. The question for you is, do you care who or what is looking? If it doesn't matter to you, then read no further. If it does matter to you, then read on – particularly if you want to block access using .NET. If you're more interested in Apache Servers, then I would instead suggest you read Dan Champion's article on blocking IP addresses with PHP and Apache.

Examples

On a per-page basis

On the page load event of each page you wish to apply the page to, check to see whether your search term is present in the user agent's name, or whether your IP address is found. Note the 'bar_explain' page does not exist, so don't go looking for it! To be honest, this isn't an ideal way to do it as it doesn't cater for multiple options – you might want to pinch some of the loop functionality from the following example if you intend to do it this way.

Dim userAgent As String
Dim userIP as string
userAgent = Request.UserAgent
userIP = Request.UserHostAddress.ToString
If userAgent.IndexOf("blahBlah") > -1 Then
  ' The user agent contains blahBlah - note that this is case sensitive.
  Response.Redirect("http://www.thepickards.co.uk/bar_explain.htm")
End If
If userIP.IndexOf("123.22.1.") > -1 Then
  ' The user agent contains 123.22.1
  Response.Redirect("http://www.thepickards.co.uk/bar_explain.htm")
End If


On a per-application basis

This is the way I would very much recommend doing it. It will affect the entire .NET application without you needing to add something to every page, it works for multiple blocks and it's easily customisable so you can change it from a blacklist to a whitelist and vice versa. The initial code for how to search the user agent string was taken from an example on webmasterworld, so thanks for that, but I've changed it somewhat and also included the IP address checking.

Basically, to use this method, add a global.asax file to your application and edit the Session_Start subroutine (or edit the existing one, if you already have the file). The code is commented so it should be quite straight-forward from there…


UserAgentsTest: 'if you don't want to test by user agents, remove this block
  'Define User Agents variables - set strings to lower case to make case INsensitive
  'Change this to list the user agent STARTING strings you wish to search for
  Dim strAgents As String = "badbot1,badbot2,spambot6a,purplebb"
  strAgents = strAgents.ToLower
  Dim ua As String = Request.UserAgent.ToLower
  Dim AgentsArray As Array = strAgents.Split(",")
  Dim AgentsSize As Integer = UBound(AgentsArray)
  Dim i As Integer = 0
  Dim UABanned As Boolean = False
  'Check user agents
  Do While (i < = AgentsSize)
    'Checks if user agent begins with something from your list
    If (Left(ua, Len(AgentsArray(i))) = AgentsArray(i)) Then
      UABanned = True
      Exit Do
    End If
    i = i + 1
  Loop
  If (UABanned = True) Then 'this is barred by means of user agent
    Response.Redirect("http://www.thepickards.co.uk/ban_explain.htm")
  End If
UserIPTest: 'if you don't want to test by IP addresses, remove this block
  'Define IP variables
  Dim strIPs As String = "10.22.1.,127.0.0."
  'If you want to allow/disallow RANGES, specify IP addresses ending with a "." i.e. 14.107.11.

  '(as 14.107.11 would match 14.107.110.xx range as well as the 14.107.11.xx range you were
  'seeking to find
  Dim j As Integer = 0
  Dim IParray As Array = strIPs.Split(",")
  Dim UserIP As String = Request.UserHostAddress.ToString
  Dim IPSize As Integer = UBound(IParray)
  Dim IPBanned As Boolean = True 'True for whitelist, False for blacklist
  'Check IP addresses
  Do While (j < = IPSize)
    'Checks if IP address begins with something from your list
    If (Left(UserIP, Len(IParray(j))) = IParray(j)) Then
      IPBanned = False 'False for whitelist, True for Blacklist
      Exit Do
    End If
    j = j + 1
  Loop
  If (IPBanned = True) Then 'this is barred by means of IP address
    Response.Redirect("http://www.thepickards.co.uk/ban_explain.htm")
  End If