Death by static content
With the popularity of platforms like Ruby on Rails, people are building things faster, but applications often fall apart under heavy load.
You can help improve performance by caching dynamic content, reducing the number of database hits, etc. When you get hit really hard by Digg or Slashdot, measures like caching or reducing the number of database hits will only go so far.
For every page view, the browser requests not only the page itself, but every static asset you have in your page – stylesheets, javascript, images, etc. These additional queries can quickly add up until your server is saturated with requests and can’t keep up. Memory runs out, CPU is at 100%, money and traffic is lost.
An easy and effective optimization is to move static content to a secondary HTTP server. Images and stylesheets don’t need the flexibility or overhead of Apache. You just want to serve them fast.
But do it now, before you’re under heavy load. When you’re losing traffic, the last thing you want to be doing is scrambling around editing your templates and HTML to point all your static content to new URLs.
I’ve used both lighttpd and thttpd for serving static content. thttpd is lightning fast, but lighttpd is also very fast and is a bit easier to administer. Both did an excellent job of freeing Apache to deal with the heavy worker request.
Always keep the question in mind: are you getting the most out of your hardware? Inefficiencies can exist in any solution, regardless of how much it’s touted or how comfortable you are with a setup.