Note: This post was originally part of my LinuxWorld blog; now migrated here after my resignation.Fudging tests to make your company or product look better is not new, but there are certain companies and organizations which seem to have a better reputation for it than others. The latest Get the Facts document is a perfect example of how to bend the rules.

First, some good things about the tests:

• They used the same hardware
• They had a standard set of tests
• They used a standard set of clients – individual – rather than some testing mechanism saturating a handful of machines
• They provide full details of how they did everything

Unfortunately, this last item is exactly what highlights the problems with the comparisons. Here are the main points of failure in their testing methodology:

• They use an old version of Redhat Advanced Server and Redhat Professional. We’re talking old kernel code here, something which (especially the new threading model) is going to make a significant difference. By comparison, they are using Windows Server 2003 RC2, not a final release, to be fair, but with a completely different set of features compared to the Linux OS they used; there’s not a huge difference, functionality and performance wise, between RC2 and the current release. There’s a heck of a difference between RHEL 2.1 and the current release.
• They use different SSL standards on the two boxes. For Windows they use one of the simplest, and least processor intensive options. On the Linux box, they use a more complex and more CPU intensive algorithm.
• They compare CGI performance on Apache against ISAPI performance on IIS. They also compare CGI on both, but comparing ISAPI to CGI is unfair. It’s not that Apache doesn’t have the functionality available to host an equivalent, such as mod_perl, mod_python or even PHP. Better would have been to tested the PHP ISAPI in IIS against the PHP module in Apache.
• They made some interesting tweaks. Many of the Windows and Linux configuration tweaks are pretty much identical. Some of them, though, are different but the casual observer would think they were identical. For example, under Windows they say ‘Set HKLMSystemCurrentControlSetControlFileSystemNtfsDisableLastAccess to 1.’ and within Linux/Apache they say ‘Disabled Access Logging’. For those not familiar with Windows, the registry tweak they mention disables last access time updates on the filesystem. What they did on Linux was disabled Web server logging within Apache. What they didn’t disable filesystem access time updating (which they could have done with a simple noaccess option when mounting the fs). Access time updating is a known performance hog and one which most Linux performance tuners would disable merely out of habit.

As always, what is sad is that many people will read this document and choose Windows without a second thought. They won’t even check the facts – not even a simple reconciliation of version numbers. Windows Server 2003 – even with SP1 – is still referred to as such, but Linux, even commercial versions, are constantly evolving and improving and there’s a big difference between RHEL 2.1 and RHEL 3.0.