Ruby Implementations Shootout: Ruby vs Yarv vs JRuby vs Gardens Point Ruby .NET vs Rubinius vs Cardinal

A newer shooutout is available.

Many brilliant developers are working on improving the current implementation of Ruby and on creating alternatives. I was curious about their current respective speeds, so I installed and ran some benchmarks for the most popular implementations. In this article, I’m sharing the results for the community to see.

Disclaimer

  • Don’t read too much into this and don’t draw any final conclusions. Each of these exciting projects have their own reason for being, as well as different pros and cons, which are not considered in this post. They each have a different level of stability and completeness. Furthermore, some of them haven’t been optimized for speed yet. Take this post for what it is: an interesting experiment;
  • The results may entirely change in the next 3, 6, 12 months… I’ll be back!
  • The scope of the benchmarks is limited because they can’t stress every single feature of each implementation. It’s just a sensible set of benchmarks that give us a general idea of where we are in terms of speed;
  • These tests were run on my machine, your mileage may vary;

Benchmark Environment

My tests were conducted on an AMD Athlon™ 64 3500+ processor, with 1 GB of RAM.

The tested Ruby implementations were:

The operating system that was used for all – but Ruby.NET – is Ubuntu 6.10 (for x86). Ruby.NET currently runs on Microsoft Windows only, therefore I’ve used Vista with the .NET Framework 2.0 and have also run Ruby 1.8.5-p12 on Windows as a means of having a more direct comparison with Ruby.NET.

Ruby 1.9, JRuby, Rubinius and Cardinal were all installed using their respective latest development versions from trunk.

Tests used

The 41 tests used to benchmark the various Ruby implementations can be found within the benchmark folder in the repository of Ruby 1.9. The following is a list of the tests with a direct link to the source code for each of them:

Results

The following table shows the execution time expressed in seconds for Ruby 1.8.5 on Linux, Ruby 1.8.5 on Windows, Ruby 1.9 (Yarv/Rite) on Linux, JRuby on Linux, Gardens Point Ruby.NET on Windows, Rubinius on Linux and finally Cardinal on Linux.



LEGEND:
  • A blue bold font indicates that the given Ruby implementation was faster than the current stable, mainstream one was (Ruby 1.8.5 on Linux);
  • The baby blue background indicates that the given Ruby implementation was the fastest of the lot for the given test;
  • ‘Error’ indicates an abnormal interruption of the program. ‘Too long’ instead, is an indication that the execution took longer than 15 minutes and was manually interrupted;
  • Average and Median values take in consideration only working tests (they exclude ‘Too long’ programs as well).

Below is a chart which shows the average and median values, visually:



You may also be interested in visualizing a direct comparison of how many times a given implementation was faster or slower than Ruby 1.8.5 on Linux:



Of course, the bold green values indicate a positive performance, so for example Cardinal was 4 times faster than Ruby 1.8.5 on Linux for the test vm1_swap, but it was also 18 times slower for so_matrix (therefore in red).

I won’t provide too many personal considerations but rather let you enjoy the numbers. Generally speaking though, Ruby on Windows was about 1.5 times slower than on Linux. Yarv (merged in the development version of Ruby) is clearly the fastest by a long shot. This is good news (there are hopes for a fast Ruby 2.0), and it is not an unexpected result.

Ruby.NET and JRuby had similar performances and were able to execute most of the tests. It is clear though that they will need to focus on improving their individual speeds in the coming future, in order to be ready for prime time.

Cardinal wasn’t able to complete most tests, and was extremely slow in some others. However on a few occasions, it also showed decent results (beating Ruby 1.8.5 in 3 tests). Rubinius was extremely slow too but correctly handled a larger amount of tests than Cardinal was able to (and it was significantly faster in executing so_sieve.rb).

I’d like to conclude by saying that all the people involved with these projects are doing an amazing job. And while some implementations show that they are in an early stage of development, it is in no way detrimental of the great effort and work done by their developers, nor attempts to predict their future success or failure. So once again, great job guys, all of this is nothing short of exciting!

UPDATE 02/21/07: Wow, it looks like this article received a lot of attention and naturally I’m glad it did. Slashdot linked to this and traffic sky rocketed, giving major exposure to all these projects.

Most importantly, I initially thought I’d run another batch of tests in 3 months time, but given the amount of feedback that I’ve received, I’ll carry out another test run fairly soon to incorporate many of the insightful suggestions and requests that were received.

By the way, Ruby 1.8.6 is out in preview, and some of you sent me emails asking to test it out. Running the test shows that it’s usually slightly faster than 1.8.5 and it seems to notably speeds up recursion based tests. The next test run will have details for Ruby 1.8.6 as well.

Get more stuff like this

Subscribe to my mailing list to receive similar updates about programming.

Thank you for subscribing. Please check your email to confirm your subscription.

Something went wrong.

59 Comments

  1. Lorenzo February 19, 2007
  2. Michael Silver February 19, 2007
  3. Asd February 19, 2007
  4. Tom February 19, 2007
  5. Giovanni Intini February 19, 2007
  6. nobody February 19, 2007
  7. Charles Oliver Nutter February 19, 2007
  8. Evan Phoenix February 19, 2007
  9. Antonio Cangiano February 19, 2007
  10. Antonio Cangiano February 19, 2007
  11. p24t February 19, 2007
  12. Ben February 19, 2007
  13. Drew February 19, 2007
  14. Antonio Cangiano February 19, 2007
  15. Grant Hutchins February 19, 2007
  16. Antonio Cangiano February 19, 2007
  17. Jason Trenouth February 19, 2007
  18. Jo Stockley February 19, 2007
  19. Jim V Tess February 19, 2007
  20. Miguel de Icaza February 19, 2007
  21. Kevin S. February 19, 2007
  22. mr foo bar February 19, 2007
  23. Antonio Cangiano February 19, 2007
  24. Matt Diephouse February 19, 2007
  25. Antonio Cangiano February 19, 2007
  26. Antonio Cangiano February 19, 2007
  27. Christopher C. Aycock February 19, 2007
  28. Michael Silver February 19, 2007
  29. megatux February 19, 2007
  30. http://kurtstephens.com February 19, 2007
  31. Matthew Elder February 19, 2007
  32. Sebastien Lorion February 19, 2007
  33. newbie ruby fanboy February 19, 2007
  34. Mike Schwab February 20, 2007
  35. Paul Oswald February 20, 2007
  36. Asd February 20, 2007
  37. rub'ish February 20, 2007
  38. mathie February 20, 2007
  39. TSO February 20, 2007
  40. Antonio Cangiano February 20, 2007
  41. YARV vs LuaJIT? February 20, 2007
  42. John Gough February 20, 2007
  43. Antonio Cangiano February 20, 2007
  44. Kannan Goundan February 20, 2007
  45. Steve February 20, 2007
  46. Andreas February 26, 2007
  47. Jabari Zakiya February 28, 2007
  48. Antonio Cangiano February 28, 2007
  49. Mohan March 19, 2007
  50. Mohan March 19, 2007
  51. M. Edward (Ed) Borasky March 20, 2007
  52. M. Edward (Ed) Borasky March 20, 2007
  53. Antonio Cangiano March 20, 2007
  54. myself March 27, 2007
  55. Dan Kubb April 20, 2007
  56. Alan Brown April 28, 2007
  57. John Smith May 5, 2007
  58. Beck ham May 6, 2007
  59. Antonio Cangiano May 6, 2007