Surviving Slashdot

100 %
0 %
Information about Surviving Slashdot

Published on July 18, 2007

Author: johnbuckman



Technical presentation from 2005 about how to make your web site survive slashdot/digg style mega load.

Surviving Slashdot: How to plan for the deluge.

Magnatune’s architecture: linux/apache/mysql/php

What happened when Slashdot hit.

1. Thousands of Apache processes

2. Http Servers unresponsive

3. Http Audio Streaming not fast enough for real time

Analysis & Solutions

Why is Yahoo so “snappy” ?

By default, Web browsers ask for every image on every page with “If-Modified-Since”.

Ironically, lots of small graphics on a web page is the least efficient use of HTTP.

What if you could make browsers not ask for images that are unchanging?

Trick #1: use “Expires” HTTP header on static graphics: Expires: Thu, 15 Apr 2010 20:00:00 GMT

I did this by hacking the C source code to my HTTP server.

Is there a fast Apache way to add an Expires: header when returning GIF/JPEGs?

next problem...

In many cases, HTTP performance is constrained by the receiver, not the sender.

Large files take a long time to be downloaded by the requestor.

Each download uses up an Apache process.

Apache Bench shows great large-file performance.

But Apache Bench isn’t a useful metric with large files in the real world.

The problem is the “fork” based architecture of Apache.

Thread-based HTTP servers have the same problem.

Apache has an experimental “async” mode -- this may be a solution in the future.

Trick #2: Use a dedicated async- based HTTP server for large files.

next problem...

PHP is handy. But PHP is slow.

Apache Bench Speed tests: PHP: 9 pages/sec HTML: 250 pages/sec

Trick #3: Statically cache most PHP pages into HTML.

Use Apache “MultiViews” option.

URLs shouldn’t include .php or .html

ie: could be PHP or HTML, doesn’t matter.

Have a script regularly HTTP GET most PHP files, save as HTML and delete the PHP.

Yields a 25x speed increase.


Add a comment

Related pages

Surviving Slashdot - Wenn ein einfacher Webserver einfach ...

Surviving Slashdot Wenn ein einfacher Webserver einfach nicht mehr reicht Volker Tanger Senior Consultant System Security HiSolutions AG, Berlin
Read more

Surviving Slashdot: Wenn ein einfacher Web-Server nicht ...

Der Web-Server (-Shop, ...) ist down - oder kurz davor. Was kann man noch tun? Oder wie kann man sich auf so etwas vorbereiten? Es muss nicht immer ein ...
Read more

Sam Ruby: Surviving Slashdot

It’s just data. Surviving Slashdot Tue 19 Jul 2005 at 16:10. I was able to survive a slashdotting yesterday. The solution was simple: Redirect /wiki/pie ...
Read more

Surviving slashdot'ing with a small server

Introduction. The science and technology news service at, self-described "news for nerds," has become a widely read and extremely popular ...
Read more

Surviving the Cashless Cataclysm - Slashdot

MrSeb writes "There's been a lot of noise about Sweden becoming a cashless economy, and the potential repercussions that it might cause, most notably the ...
Read more

Surviving Outsourcing? - Slashdot

Search Slashdot. Login; or; Sign up; Topics: Devices; Build; Entertainment; Technology; Open Source
Read more

Surviving Slashdotting with a Small Server - Slashdot

S.BartFarst writes "Our little departmental server has been slashdotted twice in the last year and survived! Implementation of a two-headed redundant ...
Read more

Surviving Tornadoes - Slashdot

Search Slashdot. Login; or; Sign up; Topics: Devices; Build; Entertainment; Technology; Open Source
Read more

MySQL Lists: mysql: Re: surviving the Slashdot effect

Thread • surviving the Slashdot effect: Thomas D. Wilkinson: 12 Jul • Re: surviving the Slashdot effect: Michael R. Gile: 12 Jul • Re: surviving the ...
Read more

Erlang on Xen - at the heart of super-elastic clouds

Surviving "Slashdot effect"-----Finally, your shiny app gets noticed by media and a praise piece is published on a high-traffic website.
Read more