Updated: November 22, 2010
I have written about browser speed before. We have discussed browser speed as a manifestation of one or many misconfigured or sub-optimized components in an operating system, resulting in a flawed browsing experience. We tried to understand what the problems were and how we could gain back the lost value.
Today, we will talk about a different kind of browsing speed issues. The far end of the spectrum, the browser speed benchmarks. Supposedly, they ought to tell us something about our product of choice and make us proud and convinced in our decision. The tests have nothing to do with your setup; they have everything to do with what your browsers looks like inside. Which makes them totally, completely irrelevant. Let me explain.
Understanding the technology
Most people have no idea what browser speed benchmarks mean and how they affect their everyday use. But they do understand the words fast and fastest. They automatically translate the words into an image of a majestic supercar and they enjoy a moment of proverbial arousal. The reality is a little different.
Browser benchmarks are a kind of conceptual superiority contest between code developers, who try to best one another by writing the most optimal browser engine. It's about single versus multi-threaded processes, mutexing, sandboxing, and other geeky words.
Truth to be told, and few will tell you the truth, browser speed tests are as relevant to real life as the fuel consumption of Lamborghini LM002. They are a sort of a glimpse into what technology underneath the hood could do, in a perfect world without slow users or network congestion. Too bad the world is not perfect. My story goes as follows:
Your browser is NOT the bottleneck
The simple fact is, your browser is only one link in a long chain of devices and software that make the Internet work. Your browser is an endpoint to a supremely complex and convoluted multi-layered, multi-tiered grid.
When you type in an address of a website you would like to see, a whole of series of events happen before the web page is displayed and you're happy. Your request for a web page is translated into packets of data that are routed through your network stack, past all kinds of filtering tools, anti-virus software and firewalls, to your router, your modem, your ISP switch, and then another ten or so hops to the Web server that actually holds the page.
Every one of these elements could be performing at sub-optimal levels, causing a tiny delay in the quality of service, which you would eventually interpret as a browser experience.
The Web server you have just contacted might be loaded. The page you have requested might be big and contain third-party code hosted on yet other servers. It could be the Internet rushhour, and your generously over-committed provider is struggling. You might be downloading a movie, which could also affect the results. Wireless signal, multiple computers running and sharing the bandwidth resources at the same time, periodic software updates in the background, CPU load, all of these are some of many factors affecting the browsing. Now, combine everything and you get one big unknown. But most people have no idea. All they see and feel is the loading of requested web pages.
Let me give you a personal example
A few months ago, I upgraded my weakest, 1.5Mbit line to a 12Mbit line. Let me ask you a question. Do you think my browsing experience improved? The answer is NO! If there's any change, it's imperceptible. I do notice a significant, quite linear change in sustained uploads and downloads, and even the ping times have improved, which makes online gaming more fun, but the browser space remains untouched.
How come, you ask. The answer is simple: my browsing experience was already optimized to the max, independent of the bandwidth. Which is the first and foremost thing you need to do, before considering browser speed.
You need to optimize your machine so that you get the best performance available. You need to make sure that the bottleneck is located outside your scope of control. You want the bottleneck to reside in one of the many links beyond the perimeter of your home. For more details about slow browsers, take a look at this article.
But it comes down to something like this: If you have a 1MB line and you need to download 1MB worth of website data, text, images and all, the operation will not take less than 1 second, no matter what. Browsers can't do magic.
Benchmarking, what does it mean?
Assuming your computer setup is perfect and you don't have any lame security software crippling your machine, then you can start thinking about browser speed testing, although it's still one big cauldron of unknowns. How would you go about comparing browsers? Load a web page, right? Well, in a way, this is correct, but let's not forget that you have no control over what happens outside.
The only truly accurate test would be to start a Web server on your own machine and run tests against localhost. This way you would know there are no external factors impacting your network and carried over into the browser space, creating a false impression.
The Internet is an ever-changing storm. Even if you load a web page in different browsers within seconds from one another, you would still be running an experiment under different circumstances. The only way you could claim any sort of accuracy is to run hundreds of benchmarking tests against multiple websites, many times over during a day, during different days of the week, for several weeks. You would have to do this for different browsers, different operating systems, different hardware platforms, different Internet provides, network technologies, encryption, and so forth. No one does this.
Measuring browser response is not simple, either. What your eyes see is not what the software sees. You can create a perception of speed using visual transition effects inside the browser interface, while the actual rendering remains the same. A good example of perceived and true slowness is the opening of new tabs in Internet Explorer, compared to the competition. While Firefox, Chrome and the rest open tabs instantaneously, when not crippled by stupid anti-virus software and alike, Internet Explorer takes approx. half a second to open a tab. This does make a major difference. Even if content is later on loaded fast, the overall effect is ruined.
To this end, developers have come with all kinds of computational benchmarks, which stress the browser's rendering engine, most notably the Javascript component. You can definitely see a big difference in this kind of tests.
However, the results need to be taken with a pinch of salt, and maybe pepper, too. Raw Javascript benchmarking is all good and well, but it does not take into account the reality of the Web.
Many pages use relatively small amounts of code, which is executed within a few milliseconds, long before the user can notice anything. On the other hand, benchmark tests take seconds to execute and complete, many times more than a typical page load. This means that any clear advantage one browser may have over another is diminished. First, let's begin with the hype.
A useless example
This belongs in my first installment on browser slowness, but I could not refuse the temptation. Pretty much every would-be browser expert out there will include the so-called cold and hot starts into their Excel graphs to add depth to seemingly scientific testing of their browsers. A cold start is how quickly the browser interface opens the first time you launch the program after a machine reboot. A hot start is relaunching the browser the second or third time.
Let's not forget that you need to do the test a few hundred times on different operating systems and hardware platforms for your checks to have any validity, but even so, what does the cold start tell you? It's not a Lada Niva jeep trying to start in the Siberian winter. It's a browser, which depends on a hundred background processes contesting for the CPU queue, the scheduler, the priority, and a million other parameters that no one cares about when they run their useless tests.
The same with hot starts. Upon closing has the operating system freed all the pages from memory and kernel buffers? What about your memory cache and throughput? What about the shared memory? What is your disk doing at this moment? Is it spinning? How fast is it? And so forth. None of these questions are ever answered in any browser speed benchmark and they only get worse when you work with browser engine code.
What you end up is a bunch of autistic graphs with numbers, which are taken out of context. But everyone's an expert. And then you rely on experts to tell you what is considered fast, based on how quickly their eyes can perceive loading of web elements on their favorite websites in their favorite browsers on top of their custom-built desktops or bargain netbooks.
A less useless example
For the sake of argument, the Sunspider benchmark test may take 5 seconds to complete in browser A, 10 seconds in browser B. The five-second difference is significant. However, an average web page takes, say, one second to load, half of that dedicated to Javascript execution. This means that if browser A takes 0.5 seconds to perform an operation, browser B will take 1 second. Is half a second noticeable? Perhaps. But it is definitely not significant.
Now, what if two browsers differ by 40% (still a lot). The relative difference now becomes 0.2 seconds. That's about as far as you blink. This is approx. the human response threshold, so anything within that time frame looks the same to the user. And we're talking pure engine rendering speed, without any regard to all other web elements, the content, etc.
Now, if you compare modern browsers, you will see that overall differences, here and there, large in terms of statistics, but tiny in human terms. If you aggregate over several hours of browsing, you might claim some validity to benchmarks, but for humans, who appreciate their browsing in instantaneous quantums, the times are negligible.
More explanations ...
Furthermore, we must also pay attention to how the extra time in a slower browser affects the user. If the slowness manifests in some kind of a button at the bottom of the page being loaded last, no one will notice this. Users will be busy reading, watching images, absorbing the web page layout and elements. They will hardly see any particular element struggling to keep up.
Then, when you want to open a new page, what do you do? Well, if you're even semi-competent, you will middle-click the link of interest, which will open it in a second tab, while keeping focus of the current one. You will continue doing whatever it is that you're doing and eventually switch over to the next tab. Your mind won't even acknowledge the extra seconds it might have taken the other page to load. For all practical purposes, Web browsing is a seamless activity.
Lastly, many pages contains tons of sub-optimized code, code that does not comply with standards and even errors. If you run a validation check against your popular online resorts, you will be amazed by the amount of errors and warnings. Many websites are non-compliant. Let's not forget browser-specific hacks, most notably the IE6 hacks that are required to make the antiquated browser display the pages with any kind of fidelity to the original. All of these add a significant amount of noise that renders precision imprecise.
Benchmarks are good for the race track. You can pit Bugatti Veyron against McLaren F1 and see which one wins on Nurburgring. But can you tell which one will cross London faster, at 8 o'clock Monday morning?
It's the same with browsers. Sterile lab tests will tell you one story and it's a good story. But reality will twist the story into a muddle. Benchmarks will hold true if browsers differ by orders of magnitude, but for browsers with similar performance, within several tens of percents, the end result will be pretty much the same. The user won't know the difference.
Again, a personal example. Firefox, Chrome and Opera (when I used it), all give roughly the same results. Chrome feels faster and lighter, but my real life tests show no clear advantage. Memory usage differs from one machine to another, one operating system to another. Eventually, it comes down to taste and habit.
Still, I can't exactly not feel an urge to blowtorch someone whenever they mention the browser benchmarks. You can't stay impartial to the beautiful, eloquent and, above all, scientific claims like "X browser sucks" and "X browser is faster" uttered by people who might, on a good day, not get confused tying their shoelaces. But everyone has a PhD in Internetics.
Finally, why benchmarks are meaningless
If the above long and detailed sections do not satisfy the more skeptic among you, then brace yourselves. Benchmarks are meaningless because they are first and foremost a publicity stunt. They are used to create the illusion of perfection and take the spotlight away from things that really matter, like security, privacy, usability, modularity, compatibility.
Speed benchmarks make for lovely Matrix-like videos and their parodies, benchmarks make fanboys excited in forums, benchmarks make captivating titles in articles written by people who have no basic clue in Design of Experiment, but they don't make your life any faster.
Software is just one piece of the browsing experience. And we forget the user. The person who clicks. People think in seconds, if not longer. The milliseconds of difference you get from optimizing your rendering engine to sadistic perfection is lost on people who merely want to talk to their true friends on Facebook or read an article on this and that celebrity.
Most people have no idea what browsers are. Even today, the majority of Windows users run Internet Explorer, the default browser supplied with the their operating system. For the common user, the Internet is synonymous with Internet Explorer. A browser is something that feeds on leaves and tender shoots. Javascript is a weird kind of lettering used on the island of Java. Benchmark is probably a warning sign on a freshly painted park bench.
Geeks may adore their milliseconds, but it makes no difference whatsoever. Worst of all, the browsers may be perfect, but they are used by people who are, well, average. If there's anything worse than giving a perfect product to a simpleton, I can't think of what it might be. Except maybe watching reality TV, studying chemistry or dying in a spacecraft accident.
But it's a frenzy, and it has caught. One browser vendor decided to scream at the top of its lungs that it has the fastest product and the rest followed suit. No one dares stop and think. It's too risky. No one wants to appear old-fashioned in this would-be modern race.
Now, every website that wants to sound posh and techy writes long big words about Javascript and whatnot, using massively-repeated quotes and numbers to prove something that can't really be proven. It's the Hundredth Monkey Effect. Browser benchmarks have become the holy grail of browser benchmarking. It really makes no difference what the original intention was. Nothing matters. You must have the fastest browser, whatever this means. Finally ... if you happen to turn off Javascript in your browser, you make all these lovely benchmarks meaningless. It goes back to text, as it ought to be. Oh, that's cruel.
Dessert
But it hardly stops there. Enter HTML5 and browser codecs. If you thought browser benchmarks are silly, you should see what this is all about. But, that's a different story altogether. Another article, perhaps. And we will talk about tabs on top, too!
Conclusion
I hope you liked this article. But you probably did not. You are convinced, with the conviction of a Spartan facing the Persian hordes, that your browser is the pinnacle of technology and that it holds the answer to life, the universe and everything. But it does not. And Javascript, as a word, is overrated.
I bet you a florin that your computer setup has at least ten variables that can be tweaked before you will let your browser breathe at its natural speed. It's not the browser that is slow. Don't blame the browser. It's fine. Except maybe Internet Explorer. Version 9 seems reasonable, though.
Anyhow, Javascript benchmarking is all nice and well, but it has nothing to do with reality. For all practical purposes, you can have guinea pig powering your browser. Stop wasting your time worrying and caring about nonsense and focus on important things, like browser W3C compliance and stability. Now, you're talking business.
Cheers.