Firefox 3 ultimate feature: Performance
Along with the long list of new features and improvements we can find in Firefox 3 betas so far, there is a series of performance improvements coming to Firefox 3 Beta 4.
I’ve finally managed to run a set of tests to see how Firefox 3 performance compares to Firefox 2’s.
JavaScript
As you may know, JavaScript is progressively becoming more important as we, users come to expect web applications to behave more like their desktop equivalents. For web developers, the top tool for doing this is JavaScript, the language used to program the different web page elements and in some cases, query the server for small pieces of data.
For Firefox, it is even more critical since the whole interface (extensions included) runs on JavaScript: open a context menu, a menu, a tab or a window, and there’s JavaScript at work all around connecting the underlying pieces.
So you will be glad to know JavaScript performance in Firefox 3 has almost tripled compared to Firefox 2, doubled compared to Beta 3, and has become the fastest engine among all current browsers including Opera 9.5 beta and Safari 3.0.4 (also in beta). This is according to SunSpider, a JavaScript benchmark tool designed by WebKit (the project that makes Safari’s engine) to measure JavaScript performance.
It must be noted, that as any benchmark tool it is not perfect or free of criticism: an engine will most probably perform better in one test than another.
It is also said that an engine or application could be tweaked to perform better in a particular well known test. I think if the test is well designed and effectively represents what affects most users, then tweaking those particular factors only makes sense. What would be the point of fixing some obscure interfaces if it’s not what real world applications will benefit from?
I for one will be happy to learn about more and better JavaScript tests to benchmark Firefox 2 and 3.
Memory footprint
Firefox 2 has fame for consuming lots of memory and most annoyingly eating more of it when it is supposed to be sleeping.
So I designed a simple test with a set of 9 web pages on Windows XP SP2, with an empty profile and a blank page set as home page. Measures were taken with Windows Task Manager. The test covers a few simple web pages with simple browsing. Then more complex pages are added keeping the first pages opened. A couple of tabs are then closed to see how much memory is freed and finally the browser is left running for an hour to see if memory consumption remains flat.
Here are the steps in case you want to reproduce:
1. Visited The New York Times, clicked on Technology section, open a couple of articles
2. Visited Digg, clicked on Upcoming and clicked on pages 2 and 3
3. CNET News, clicked on each of the top three stories. Get back to the homepage afte reach
4. To cover international support, I visited the Russian and
5. Chinese Editions of Wikipedia. I then clicked on the first link in the left sidebar on each and then went back to the main page.
A. I took a first measure here.
6. Visited this YouTube video, view the video and one more from the right related videos sidebar.
7. On Gmail, I logged in, composed an email with 2 large images (about 2MB each), sent it to myself, and then opened the received email.
8. On Google Docs, I created a new text document, wrote a five word sentence, made it bold, centered it, added a small 20K GIF image and saved the document.
9. I visisted Netvibes, visit all the default tabs, change the page title and added a short item to the todo list
B. Second measure here.
10. Then closed Netvibes and You Tube and waited 20 seconds for Firefox to do its cleanup.
C. Third measure here.
12. The I left it opened for about one hour.
D. Last measure here.
Results:
| Measure | Firefox 2.0.0.12 | Firefox 3 Beta 3 |
| A. Regular web pages | 73 MB | 80 MB |
| B. Web 2.0 web sites | 149 MB | 135 MB |
| C. Close a couple of pages | 142 MB | 123 MB |
| D. After 1 hour with no activity | 181 MB | 123 MB |
As you can see, Firefox 3 shows better memory management as it freed more memory when a couple of tabs were closed and it kept memory usage flat when not in use.
This test is far from comprehensive: depending on your browsing habits you may have much more or less tabs opened at the same time. Also, memory piles up as you close and open new tabs and windows, with more and different plugins. So this is just a reference of what some very casual browsing takes on each version and most importantly shows the many memory management improvements Firefox 3 will introduce.
Startup time
My poor man’s approach to measure how long it takes for Firefox to launch, consisted of a batch file that ran the time command and then the tested browser set with a simple home page that prints the current time with one line of JavaScript code. By subtracting these times I got the loading time with roughly to the second precision I need.
I restarted my laptop before each test to get a clear cache and good cold start times. Then closed and restarted Firefox to see how much it benefits from the OS caching.
In summary I got the same results: 10 seconds for a cold start, 4 seconds for a restart. There may be some milliseconds there but that’s beyond what a regular user can perceive.
More
Starting today, Firefox nightly Windows builds have the benefit of profile guided optimization (PGO). With PGO, after Firefox is compiled (written code is made ready for the computer as binary code), the just generated binary is ran while another application monitors how it behaves and creates a profile of what parts of Firefox are more used and in what order. Then the code is compiled again but this time guided by the produced profile generating a new optimized build.
Ted Mielczarek, who led the effort to PGO-enable the Mozilla build process, says a 10% performance gain can already be seen in some of Mozilla’s performance metrics. Linux builds may get PGO after Beta 4 and hopefully Mac OS X builds wil also benefit from it before the final release.
Another improvement came a few weeks ago as told by Vladimir Vukicevic thanks to the discovery of an undocumented Mac OS X interface that will now help Firefox do some work instead of just waiting for its turn to draw, which translates to more efficiency.
Firefox 3 Beta 4 should be out within the next couple of weeks.
Subscribe RSS
Subscribe email
February 29th, 2008 at 12:22 pm
You missed a ‘k’ in my last name. :)
[Reply]
February 29th, 2008 at 12:33 pm
Looks like more and more performance tweaks are being made for each beta release!
I hope thunderbird 3 gets as much work put into it! Looking forward to seeing that.
[Reply]
February 29th, 2008 at 12:57 pm
@Ted, sorry about that. Fixed.
[Reply]
February 29th, 2008 at 1:29 pm
that graphic is really impressive but i have to say the real problem with firefox are extension and the imposibility to know wich one is sucking all the ram
i’ve just tested the beta3, opera do better job with the back and foward, but it feel good, i think its interpreting better the css
sorry for my bad english, have to say it xd
[Reply]
February 29th, 2008 at 1:36 pm
Is there any reason you didn’t include Safari (which claims to be the fastest mainstream browser) in your test?
[Reply]
February 29th, 2008 at 3:40 pm
@B, only reason is laziness. It has been corrected and the article updated with Safari metrics.
[Reply]
February 29th, 2008 at 4:03 pm
Cool, thanks - Safari 3 feels significantly faster to me than the other browsers’ officially released versions, but it seems the Safari folks will have some catching up to do when FF3 ships!
[Reply]
February 29th, 2008 at 5:23 pm
Really good work, thank’s a lot! ;-)
[Reply]
February 29th, 2008 at 9:38 pm
Awesome post Percy.
You do a great job of firing us up for Firefox 3. Perhaps a tagline?
Get fired up for Firefox 3?
I am really excited about Firefox 3 but what really makes me happy is that it all doesn’t stop at Firefox 3. We have future versions and improvements to look forward to. It’s like a cookie jar that doesn’t get empty.
[Reply]
March 1st, 2008 at 10:01 am
I have done the SunSpider test with Opera 9.50 beta buil 9815 and I got 5738.4 ms, link http://webkit.org/perf/sunspider-0.9/sunspider-results.html?%7B%223d-cube%22:%5B140,140,140,141,140%5D,%223d-morph%22:%5B203,187,188,172,187%5D,%223d-raytrace%22:%5B140,140,141,141,140%5D,%22access-binary-trees%22:%5B78,78,78,78,63%5D,%22access-fannkuch%22:%5B281,280,281,281,281%5D,%22access-nbody%22:%5B141,140,141,141,141%5D,%22access-nsieve%22:%5B156,156,156,141,172%5D,%22bitops-3bit-bits-in-byte%22:%5B63,47,47,46,46%5D,%22bitops-bits-in-byte%22:%5B78,78,78,78,94%5D,%22bitops-bitwise-and%22:%5B436,422,390,390,390%5D,%22bitops-nsieve-bits%22:%5B141,156,156,156,156%5D,%22controlflow-recursive%22:%5B47,47,47,46,46%5D,%22crypto-aes%22:%5B125,140,140,125,140%5D,%22crypto-sha1%22:%5B78,62,78,62,63%5D,%22date-format-tofte%22:%5B234,234,265,250,249%5D,%22date-format-xparb%22:%5B452,453,453,452,452%5D,%22math-cordic%22:%5B172,156,156,156,156%5D,%22math-partial-sums%22:%5B141,172,156,156,156%5D,%22math-spectral-norm%22:%5B78,63,78,63,78%5D,%22regexp-dna%22:%5B639,639,639,624,639%5D,%22string-base64%22:%5B203,203,187,219,219%5D,%22string-fasta%22:%5B390,328,328,359,312%5D,%22string-tagcloud%22:%5B296,265,265,250,265%5D,%22string-unpack-code%22:%5B1014,1014,967,983,983%5D,%22string-validate-input%22:%5B140,141,156,140,156%5D%7D
[Reply]
March 1st, 2008 at 1:43 pm
Excellent article Percy! I can appreciate the “poor man’s” way of testing as much as or even more than controlled lab testing. The “poor man’s” approach I feel more accurately reflects results that would be seen “in the field” rather than “in the lab”, the latter testing results rarely reflecting real use.
By the way, after my usual morning rounds about WWW which ended here at Mozilla Links and this article I decided to run the script test you described just “off the cuff” to see what kind of results I’d get with a random type test. Just so you know I have compatibility checking disabled for extensions and am running 4 so called “incompatible” extensions (up to FX3 beta 3) on the nightly builds (FX3 B4pre) and with three other tabs open (my last 3 sites visited in my usual morning rounds). The overall results were fairly close to your more controlled results for the latest nightly build:
6551.6ms +/- 0.7%
That initially shows potentially good consistency across user surfing habits. Now if only the developers will acknowledge that actual scripting performance of actual websites in some cases is worse than Firefox 2, driving CPU usage to 99% when viewed via the nightly builds when Firefox 2.0.0.x shows no such adverse behavior. Yes, some embedded scripting no version of Firefox has ever been able to handle when compared to say, IE 6/7 but I have seen specific examples of script driven “news tickers” that run across the top of a website that Firefox 2 never had a problem with, drive the nightly builds of Firefox 3 into nearly locking up the entire computer by driving CPU usage to near 100% (tested using single core machine, 1 GB memory on Windows XP SP2).
Besides that it looks like Firefox 3 is really shaping up to be an excellent browser.
[Reply]
March 1st, 2008 at 2:26 pm
Hey Percy - great review, as always. Just when I think it’s time for me to start writing a bit more about what we’re focusing on, I find that you’re already there and on the case!
(And yes, I know I owe you email :( You’ll be happy to know that with interest, I now owe you email and a bottle of scotch)
[Reply]
March 1st, 2008 at 2:47 pm
@Kirk M: If you’re seeing 100% CPU situations with sites that didn’t do it with Firefox 2, please please please file bugs. Make sure that when you do you use the “regression” keyword in the bug field. That will get people’s attention. Good subject lines help too - like “Firefox nightly 100% cpu on site that did not have problems with Firefox 2.” That will get people’s attention.
In any case, thanks for testing!
[Reply]
March 2nd, 2008 at 4:50 pm
@bordean, I ran Opera 9.5b build 9613 which I got here: http://www.opera.com/download/?ver=9.50b. If you provide a link for the latest beta I will gladly run the test and update if necessary. Did you ran a Firefox nightly? It may be that your PC is just much faster than mine.
[Reply]
March 3rd, 2008 at 12:48 am
This is the latest build of Opera http://snapshot.opera.com/windows/o950s_9815m.exe.
I don’t have Firefox 3 beta 4 and I didn’t see your computers specs to compare it with mine. I have a Pentium E2160 @ 3GHz, Asus P5N-E SLI, 2 Gb RAM @ 800 MHz and Window Vista
[Reply]
March 3rd, 2008 at 10:14 pm
@Christopher: A bug has been submitted via Martijn Wargers who picked up on my posts at MozillaZine Forums and my first “bug” attempt at Bugzilla and submitted my findings as a new bug (regression). As of this morning it appears that other developers have picked up on this as well and are apparently considering this as critical.
Ironic that despite years of testing software and all the wonderful…er…stuff that goes along with it, being new to Bugzilla or WordPress Trac for example still requires a bit of familiarizing before I can effectively utilize it. I’m just glad it could be reported and taken seriously this close to release.
[Reply]
March 4th, 2008 at 5:16 am
I just run the test using the 3rd of March nightly, and it’s getting even faster!: 3620.2ms
http://webkit.org/perf/sunspider-0.9/sunspider-results.html?%7B%223d-cube%22:%5B153,146,144,147,147%5D,%223d-morph%22:%5B142,136,143,140,141%5D,%223d-raytrace%22:%5B140,135,135,137,142%5D,%22access-binary-trees%22:%5B47,46,46,46,46%5D,%22access-fannkuch%22:%5B261,257,261,261,262%5D,%22access-nbody%22:%5B157,164,169,164,166%5D,%22access-nsieve%22:%5B66,66,67,66,66%5D,%22bitops-3bit-bits-in-byte%22:%5B69,74,68,67,67%5D,%22bitops-bits-in-byte%22:%5B111,109,109,109,109%5D,%22bitops-bitwise-and%22:%5B190,180,177,182,177%5D,%22bitops-nsieve-bits%22:%5B142,147,142,142,144%5D,%22controlflow-recursive%22:%5B43,41,42,41,41%5D,%22crypto-aes%22:%5B88,87,88,88,139%5D,%22crypto-md5%22:%5B90,90,90,91,90%5D,%22crypto-sha1%22:%5B101,99,100,101,100%5D,%22date-format-tofte%22:%5B187,189,191,192,189%5D,%22date-format-xparb%22:%5B102,100,100,102,100%5D,%22math-cordic%22:%5B199,198,193,197,201%5D,%22math-partial-sums%22:%5B211,225,212,221,223%5D,%22math-spectral-norm%22:%5B86,84,85,85,84%5D,%22regexp-dna%22:%5B237,229,244,245,239%5D,%22string-base64%22:%5B89,91,92,94,91%5D,%22string-fasta%22:%5B179,174,175,180,178%5D,%22string-tagcloud%22:%5B134,134,135,140,142%5D,%22string-unpack-code%22:%5B278,274,279,281,284%5D,%22string-validate-input%22:%5B109,108,112,111,110%5D%7D
[Reply]
March 4th, 2008 at 10:09 am
@Mike, thanks for your comments! ;)
@bordean, thanks for the link. I will update the statistics after rerunning the tests.
[Reply]
March 4th, 2008 at 12:57 pm
I’m impressed with the shown javascript speed improvements… and I’m glad you qualified your own statements towards the accuracy of benchmarks… only the real world can show how well they stand up…
As for memory issues though, I’m generally ok with the speed that firefox works at, but I have major issues with memory usage.
At work, I have a dumpy optiplex dell desktop, with 2GB of ram. I have 8 or so plugins for firefox… and this past weekend I left my browser up. When I came back to work two days later, it was eating every free ounce of ram my computer had, and windows was yelling at me about being out of free virtual memory. My task manger showed firefox running at 2GB of ram before I killed the process.
I swear by firefox, I love its extensibility. But having to kill it at a process level every day (sometimes more) is absurd. Frankly I don’t think I’d be using it if it didn’t have the restore previous session feature.
I’m really looking forward to FF3.
keep up the good work.
Thanks
[Reply]
March 4th, 2008 at 6:39 pm
what about Memory footprint comparison (Opera/Firefox,Safari…) ?!
[Reply]
March 10th, 2008 at 5:16 am
Wonderful!
I just tried FF3b3, and I experienced quite a big difference compared to FF2. FF3 was way faster, specially when just browsing around. Pages just came up quicker.
Nice work! I can’t wait for the final FF3-release! :)
[Reply]
March 12th, 2008 at 12:37 pm
Since you are comparing a beta to released product, how about comparing FireFox 3 Beta to IE8 and Webkit for fairness?
[Reply]
April 24th, 2008 at 11:57 am
Is there any reason you didn’t include Safari (which claims to be the fastest mainstream browser) in your test?
[Reply]
June 27th, 2008 at 10:38 am
Firefox is the worst version of the Firefox ever release. Look at your CPU usages that Firefox is causing after updated to Final version of Firefox. Try to stay away from this unstable product till they release a stable release.
For more info check http://tips.developersvoice.com/firefox-3-cpu-usages-problemfirefox-3-cpu-usages-problem.html
[Reply]
June 28th, 2008 at 9:30 am
Sudhir,
I found the same problem. Try this post for the fix, it works great:
http://www.greenmtngeek.com/2008/06/firefox-3-rc1-high-cpu-use-try-this-fix/
BTW, your link in your comment leads to a 404 page.
HTH
[Reply]
June 28th, 2008 at 9:47 am
Sudhir,
On second thought here I see you already found that solution (I found your article by searching your site). Not only did you find it but you listed the steps verbatim in your post without proper attribution. This is in rather bad form.
Your statement about Firefox CPU usage and lack of stability is completely unfounded. By reading your article that you to attempted to link to in your comment, you state that:
“The Firefox process CPU usage keeps varying between 2-3% and 6-7%.”
This amount of CPU usage is excellent for Firefox or any other piece of software during use so I don’t understand your complaint.
Proper attribution and research are required before you make any blanket statements about any piece of software.
[Reply]