When software chews up website bandwidth for reasons other than people actually viewing content, the owners of those sites get a little upset. Many companies understand this, so web-crawlers are often respectful and don't abuse sites when snooping around for information Ė and they also make themselves well known. That's why a recent surge in traffic on some sites had piqued the curiosity of many webmasters. Apparently, a relatively new security tool owned by AVG has been identified as a huge bandwidth hog
and a generator of false web traffic statistics.
The software works pretty simply. When someone runs a search through a search engine, it automatically takes the results (perhaps the first 10 results) and visits them, scanning them for malware before the user has a chance to see them. The idea is that it can protect users from malware before they ever even have a chance to visit the site. A good idea on the surface, but it seems that the owners of affected websites aren't as happy with it. The bigger issue at hand is that the AVG software attempts to mask itself as a real user, which can complicate things if trying to filter it out. Bad for the bad guys, but also bad for the good guys.
This is an interesting look at robots on the web today. AVG's stance is that they do what they do in the name of security Ė and are unlikely to change the behavior of their software.