Web stats and robots

2008-01-08 One-minute read

I was analyzing the web traffic of some of our most popular member sites to try to understand how to better handle high traffic and I found an interesting statistic. Here are the top four IP addresses in the web logs of one member’s web site (over a period of five days).

	IP ADDRESS        NUMBER OF LINES IN ACCESS LOG

	66.249.67.154     61729 
	74.6.8.113        47576 
	74.6.8.107        44039 
	83.231.136.9       6560 

What we’re seeing are three IP addresses that are dominating the site with between 7 and 10 times the number of hits than any other IP address. Who are these people?!? Well, the first one identifies itself as Google and the second two as Yahoo. In other words, they are robots feeding search engines. The fourth looks like a regular web browser (running Opera!!).