Observations on Web Site Data

How compressible is web data? For a recent project of mine...
  • 135 GB of web pages downloaded.
  • 15 GB of actual data extracted.
  • 0.6 GB compressed data archives.
So, about a 25:1 compression ratio for text data.

Derived from the above:
  • 15/135 ~= 10%
So, approximately 90% of the web is crud.

Sturgeon's Law - ask for it by name!

No comments: