The Modular Company (TMC)

There are not many spots where you can substitute varnish for squid, if it weren't for the weird names open source coders give to their pet projects then the internet would be no different.

Squid is a proxy-cache that is normally intended to be the 'man in the middle' passing requests from a group of users behind a fire-wall to the internet at large. Another - less well known - use of squid is to be used as an HTTP accelerator. In the past squid was a lot faster than apache at serving static requests, but with apache taking advantage of multiple cpu cores and squid remaining single core the difference between squid and apache performance wise has all but disappeared. On one of my machines I actually found out that squid when tuned to the hilt was still outperformed by the original apache server.

Clearly this is not right, an accelerator should make things faster, not 'about as fast or slower' when compared with the original setup. After reading through some fora I found that the squid people do not seem to have any solid plans to make squid up-to-date, and in the open source world that means your package is as good as dead. Oh, and configuring squid is a real pain in the ass.

Enter 'varnish', which seems to have the http accelerator market all to itself, now that the squid folks have decided to let their codebase get behind the times.

Varnish was written from the ground up as an http accelerator, and it uses a novel approach to managing the on disk cache, it doesn't. Basically varnish has outsourced that job to the operating system, a single giant swap file is the only connection between varnish and the disk.

Elegant, and simple. And *FAST*. Varnish easily outperformed squid on the same setup, and took about 10 seconds to set up, the default configuration was good enough for my needs.

But if you have something more demanding it comes with a small built in language called "VCL" that let's you tune the guts of varnish. VCL is pretty quick because it gets compiled in, instead of 'parsed' and turned into data structures that have to be scanned on every request.

I'll continue testing varnish the next couple of days, the application I have in mind is pretty demanding, a content delivery network that stores about 100 M images and that is updated quite frequently, currently spread out over 6 machines.

If varnish delivers that will be the equivalent of doubling the server capacity instantly.