Very frequently, you read benchmark claims for various languages. Those claims have lots of problems. First, they usually are a “Hello, world!” program, and not complicated business logic. Second, they usually compare optimized code in X vs. poorly written Y.
I decided to write a “Video Poker Trainer” program. In video poker gambling, you are dealt 5 cards. You choose what cards to hold, and then draw new cards. Based on your final hand, you receive a payout.
Some video poker games have a payout close to 100%, but the strategy is complicated. The casino advantage comes from the fact that many people make suboptimal plays. Therefore, it would be useful to have a program that helps you practice.
My program solves the video poker hand by checking all the possibilities. There are 2^5=32 possibilities for what you hold, and then 47_choose_n possibilities for draws, where n is the number of cards you draw. I wasn’t intending to do this as a benchmark test, but when the performance of my first version was unacceptable, I wound up rewriting it in several languages.
All of this code is running on my desktop PC.
I then tried Chrome (V8). There, it took 1.6 seconds! That’s a substantial difference compared to Firefox!
However, 1.6 seconds is still unacceptable for a game. The browser window would freeze while the calculation ran. I could sprinkle SetTimeout during the calculation, but that would make it take even longer. Also, if it’s a browser game, I can’t assume that the client is using Chrome.
I could have tried HVVM (Facebook’s version of optimized PHP), but didn’t.
My final try was C/C++. I expected that C would clobber everything, including Chrome/V8. Again, it was the exact same algorithm, changing the PHP syntax to C/C++ syntax.
I was surprised that the C/C++ version took 1.6 seconds, the same as Chrome/V8! I was pretty shocked by that.
I probably could go back and tune the code, but to be fair, then I’d have to do it in all 3 languages. To make the comparison perfectly accurate, it should be the exact same algorithm in each language.
It wouldn’t be a proper experiment if I got the answer I expected. I thought that the C version would clobber everything, including the Chrome/V8 version. Instead, C and Chrome came out tied.
Oops, just as I finished this post, I realized I made a mistake. I was using gcc WITH THE DEFAULT COMPILE OPTIONS! I tried again with -O3 full optimization flags! Now, the C version takes 0.6 seconds, coming out ahead of V8, like it should. My confidence in C is restored. (For completeness, -O1 took 0.71 seconds, -O2 0.61 seconds, -O3 0.60 seconds.)
Another improvement, “gcc -O3 -ffast-math” took 0.57 seconds! (I don’t mind an error in the 7th decimal place, since this is a game, so -ffast-math is a reasonable compiler optimization.)
The benchmark comparison was on my desktop. Moving to my Linode (using g++ instead of gcc), the unoptimized version took 1.3 seconds per hand. The -O3 version took 0.36 seconds per hand! (Actually, because the Linode is shared, the time it takes to run depends on how much the other users are using the server. I got a range of 0.36 seconds per hand to 0.5 seconds per hand.)
My initial conclusion, as I was composing this post, was that Chrome/V8 and C had equal performance. However, I was making a big mistake! I wasn’t turning on optimization settings in gcc! That shows how a poorly conducted benchmark test can lead to false conclusions. I almost made that mistake myself! (For Chrome/V8, I was using the default settings. I don’t know anything about extra optimization settings in Chrome/V8.)
The game still isn’t finished. I just wrote the code that does the calculations. I’ll put it up here when I”m done. It looks like only way to get acceptable performance is to shell_exec call the C program that does the calculation, and then serve that to the client. If I’m just calling an external C program, it doesn’t matter what language I use on the server, but I’m going to stick with PHP because that’s what I like the best.
It’s also educational to write the same algorithm in multiple languages. By the time I got around to the C/C++ version, I was making sure it was proper reusable code, so I could use the same function to solve multiple video poker variants. I plan for my actual program to work on 9/6 and Double Joker Wild. (Double Joker Wild and 9/6 are two video poker variants. 9/6 has a house edge of only 0.6% played perfectly, and Double Joker Wild has a house edge of 0.09% played perfectly, making a practice program useful. Adding wild cards means I need a completely different hand evaluation function, which is why I had to make that a function pointer.)