Bandwidth Pirates

Over the last few weeks, one of the e-commerce websites (online DVD sales) that we manage has been experiencing brief periods of unusually high traffic activity. Bandwidth usage for this site has been fairly consistent over the past year with gradual seasonal variations, but now we were seeing occasional spikes where bandwidth usage would surge quite significantly. Browsing through the website's Google Analytics reports didn't reveal anything unusual that would account for the bandwidth spikes. 

A few days ago, the occasional spikes suddenly became the norm. What used to be a handful of spikes per day, each lasting less than a few minutes, was now ocurring repeatedly. Our daily bandwidth consumption was now consistently 10 to 15 times over what we were averaging over the past year without any visible signs of increased visitors or sales. If this continued, we would certainly be facing a very large bandwidth bill at the end of the month.

Still unable to find anything unsual in our analytics reports, we decided to revisit the raw access logs for our http server. We immediately noticed the flood of requests for the large image files hosted on our server. Like most e-commerce websites, we provide product images in multiple sizes. There are small thumbnail images for product listing pages, medium sized images for detailed product pages, and high-resolution images for close-up views of the products. In general, the number of requests for the high-resolution images are much less than for the smaller ones; only after visitors browse through many products (20 thumbnail images per product listing page) do they zoom in on a product and look at the larger images. Based on what we were now seeing in the access logs, the requests for these high-resolution images were now outpacing the smaller ones.

Since instrumenting this website with Google Analytics, we have almost completely stopped looking at its raw http logs. This is a good reminder that although Google Analytics is very good at providing reports on where your visitors are coming from and what they do on your site, it doesn't give you the full picture. In this case, it didn't tell us when someone was hotlinking to our images and stealing our bandwidth.

Looking closely at the access logs, we found at least a dozen websites making requests for high-resolution product images on our server. In our case, these were all Chinese language websites offering (illegal) peer-to-peer movie downloads. I knew piracy was rampant in China, but now they're also resorting to bandwidth piracy? It's bad enough that piracy is affecting our online sales, but now we also have to pay for their bandwidth?

It was very interesting looking through the access logs and doing some simple queries to learn about the sites that were stealing our bandwidth. Most of the websites that were hotlinking to our product images didn't even use domain names. They would setup basic websites using a forum (discussion based) software, and host them at various low-cost hosting providers across the United States with just the IP addresses they were given.  That's how clandestine these operations were. A few that used domain names had very cryptic domain registration information.

In these forum websites, there would be thousands upon thousands of posts with links to  movie downloads using peer-to-peer networks such as eDonkey. These posts would contain hotlinks to the high-resolution (DVD cover) images from our site.

A quick search on Google will reveal many possible solutions to this problem of hotlinking and bandwidth theft. The solution we chose was to updated our .htaccess file (for those using the Apache web server) with the following:

This solution isn't perfect, as there are some minor drawbacks, but it is a very easy one to implement for those using Apache as their web server. Given how persistent these pirates are, I'm sure they'll find a way around this (temporary) roadblock. We'll probably need to revisit this problem again very soon.