03 December 2007 - 2.0-beta2 |
This document aims mainly to compare the results obtained with the popular JMeter tool from apache with those obtained with soapUI and to show and discuss some of the differences in the results obtained with the 2 tools. If you think/know we are doing something wrong in our tests or conclusions, don't hesitate to contact us and we will try to sort things out :-)
A feature comparison will between the two tools will not be done in any detail, generally one could say that JMeter is much better at load-testing in general (ie HTTP, JDBC, JMS, etc, etc) while soapUI is better at load-testing web services specifically. Most things you can do with soapUI can probably be done in JMeter, but since JMeter has a more "generic" approach to almost everything, it won't always be as "intuitive" as it is (or at least should be) in soapUI. On the other hand, many things that can be done in JMeter are not possible in soapUI (for example distributed load testing).
In general one can say that Load-testing web services is extremely difficult since so many factors can affect the measured results; network latency, host-load, hardware, jvms, etc... Both soapUI and JMeter will be a "victim" to this, although soapUI has slightly more options to control exactly what and how results are to be measured (in the TestCase options dialog).
The setup for the comparison is a simple request to a web service running locally under an improved version of JBoss 4.0.3SP1 and running a slightly modified sample Web Service from the jbossws 4.0.3 samples. A simple request will be run 5000 times under 4 different loads; 1 thread, 5 threads, 10 threads and 25 threads all using the simple strategy with no delay, the TestCase Options are configured to include only request write times, and will be run with and without closing connections after each request.
This is in JMeter is configured equivalently using one ThreadGroup with no ramp-up and varying the loop count and number of threads accordingly; 1/5000, 5/1000, 10/500 and 25/200. The "WebService(SOAP) Request" sampler is used and configured as shown below. An "Aggregate Report" listener is configured for collecting the results.
My laptop is running WinXP SP2 on a Pentium M 1.8 with 1Gb of RAM. JRE 1.5.0_06 is used for running JBoss, JMeter and soapUI.
The table shows the results, for each test the average and TPS times are shown, (*) tests are run without closing connections
Threads | jmeter | soapUI | soapUI (*) | soapUI cmdline | soapUI cmdline (*) |
---|---|---|---|---|---|
1 | 8 ms, 105 TPS | 6.78 ms, 147 TPS | 10.7 ms, 94 TPS | 5.75 ms, 174 TPS | 10 ms, 99 TPS |
5 | 43 ms, 110 TPS | 38.7 ms, 128 TPS | 23.7 ms, 211 TPS | 30.4 ms, 164 TPS | 24 ms, 210 TPS |
10 | 86 ms, 112 TPS | 82 ms, 122 TPS | 46.5 ms, 215 TPS | 61 ms, 164 TPS | 38 ms, 262 TPS |
25 | 214 ms, 114 TPS | 204 ms, 123 TPS | 124 ms, 202 TPS | 159 ms, 157 TPS | 95 ms, 263 TPS |
This can be plotted as follows:
As can be seen in the graph and numbers, soapUI consistently gives a higher TPS value than JMeter, except when reusing connections and running only one thread. This may be a bit surprising, evidently it is faster for HttpClient and the JVM to create new connections than to pool existing ones.
The biggest difference between the two tools can be seen when switching to connection-reuse and running from the command-line. This is expected, connection reuse should have a big impact when running many threads, and running from the command-line uses no UI resources at all.
One could argue that closing connections after each request is a more "realistic" scenario for web services that serve a large number of different clients, for example an online weather service. Web services running internally in a SOA may be more likely to serve a smaller number of clients, so these could be tested with connection reuse enabled.
Looking through the JMeter sources, it seems as if JMeter uses HTTP 1.0 and does not maintain any
internal connection pool for Web Service requests. Also, it uses System.currentTimeMillis()
for measuring
elapsed time instead of the System.nanoTime()
feature available in Java 1.5 and used by soapUI.
Since the above results indicate an improved result when reusing connections, I ran a LoadTest in soapUI using the "Threads" strategy increasing the number of threads from 1 to 25 over a period of 250 seconds, both with and without connection reuse. The graphs below show the results:
First without connection reuse (click on image for full-size):
This confirms the previous results; TPS does not vary much when increasing the number of threads, the average request time "takes" the entire increased load.
And then with connection reuse (click on image for full-size):
The results here also confirmed those above; when reusing connections the TPS value increases steadily up to about 6 threads and then lies still. Also, one can see that the average request time starts increasing steadily first when the TPS has hit its peak, the average then "takes" all the increased load just as when not reusing connections at all.
The values obtained with soapUI are almost consistently "higher", if this is better is a matter of discussion. One clear advantage of soapUI seems to be the possibility to reuse connections, which also makes a big difference when running many threads (ie simulating many clients).
Based on the above results my recommendation would therefore be the following:
If you want to run the tests yourself, the following files are available in comparison-files.zip:
File | Description |
---|---|
comparison-soapui-project.xml | soapUI project file containing all tests run in the comparison |
soapui-compare.jmx | jmeter project file containing all tests run in the comparison |
ws4ee-samples-server-ejb.jar | precompiled jar file containing tested Web Service, just drop this in the JBoss 4.0.3 deploy folder and the above tests should run as configured (you may need to change the endpoint) |
You can download JBoss 4.0.3SP1 from here, but you can also use the 4.0.4RC2 release using the new JBossWS stack which gives slightly different results than those shown above (not shown here since this comparison is not meant to compare jboss versions)