This section provides some comparisons of Simple's performance against two popular servers, Apache and Resin. This does not provide a benchmark such a SPECweb, but serves to provide an overview of the performance characteristics of Simple. Typically when a site is heavily loaded content is served at a slower rate. The goal of a performance comparison is to determine what kind of service a server is capable of by comparing it with a known quantity, like Apache. The results shown here illustrate the concurrency capability of Simple under stress, and its ability to keep response times per connection high during such stress.

The comparison between servers was done with Autobench to retrieve a static 7 kilobyte file over the LAN connecting it to the server. Both Simple and Resin were run using the Java HotSpot/1.4 interpreter and the server OS used was Linux/2.4.19. Although the tests performed provide some details as to the performance of the servers involved, the benchmark was conducted in a static environment which tends to favor servers like Apache, and does not accurately reflect the activity of clients.

Apache Comparison

The first comparison done was with Apache/1.3.26. The two primary measurements done were response rate per connection (throughput) and the error rate. The response rate statistics performed by autobench measure the throughput by sampling several connections and calculating an average of the samples obtained. This technique is a poor indicator of how well the server is performing under the applied load. This is because the measurement does not account for the concurrency capabilities of the server.

The measurements below show the output from tests done by autobench on Apache. This shows that although the response rate average is high the number of response messages received is low. Clearly if the response rate samples are taken from a few active connections the overall performance of the server is not represented in the results.

      Reply rate [replies/s]: min 100.3 avg 127.2 max 145.0 stddev 19.2 (5 samples)
      Reply time [ms]: response 520.2 transfer 460.4
      Reply size [B]: header 269.0 content 7043.0 footer 0.0 (total 7312.0)
      Reply status: 1xx=0 2xx=3241 3xx=0 4xx=0 5xx=0

The same test performed on Simple produced very different results. While the duration of the test on Apache was shorter it resulted in much lower concurrency, as the number of response messages was low. The output below illustrates that the concurrency rates for Simple were much higher. Simple produced up to 3 times the number of response messages and the response rate for each connection was higher than Apache. The results below show the output from autobench for the same test as that done with Apache.

      Reply rate [replies/s]: min 80.4 avg 134.2 max 153.4 stddev 16.5 (15 samples)
      Reply time [ms]: response 1479.6 transfer 2214.3
      Reply size [B]: header 173.0 content 7043.0 footer 0.0 (total 7216.0)
      Reply status: 1xx=0 2xx=10763 3xx=0 4xx=0 5xx=0

This occurs because Simple spreads effort evenly across each connection. By distributing effort across each connection evenly autobench does not cause a connection timeout on idle connections. The timeout value set for the tests was 10 seconds, which is roughly the time you would expect a client to wait for some sort of response. Below is a comparison of the response rates per connection generated by autobench.

Figure 1   The response rate comparison with Apache

The graph in Figure 1 shows that on a per connection basis Simple performs slightly better. However as shown above this response rate was distributed across more connections, as Apache caused a large portion of timeouts per connection. The graph below verifies the behavior of Apache under heavy load and illustrates the capability of Simple to handle many connections. This concurrency capability can be significantly increased as demonstrated by the Resin comparison.

Figure 2   The error rate comparison with Apache

Resin Comparison

The second comparison done was with Resin/2.1.6. Resin was chosen for the second comparison because it is a fast Java based Web server. Because the Java platform has a performance overhead a comparison with a server on an equal footing would provide some details on the performance of the design. As with Apache the performance of Resin is slightly poorer than Simple as load is applied. As with the comparisons with Apache, Simple demonstrates much better capabilities for concurrency.

Figure 3   The response rate comparison with Resin

From the tests performed on Resin it is also apparent that it does not deal with concurrency effectively. The error rate for Resin was much higher than Simple and significantly higher than Apache for the same load. The graph in Figure 4 shows a lower error rate because the wait period for Simple was set to 2 seconds when the default used for the Apache comparison was 1.

Figure 4   The error rate comparison with Resin

Tomcat Comparison

The third comparison done was with Tomcat/4.1 and Simple/2.5.2. This comparative measurement unlike the previous two is a measurement of the application servers ability to serve dynamic content. Here, the test was performed on a Servlet packaged with Tomcat, the HelloWorldExample Servlet. Performing the measurement on this, the most basic of Servlets, ensures the results reflect the performance of the server without much interference from the time taken to process the Servlet, or Service in the case of Simple.

This comparison was done with Autobench to retrieve the contents of the HelloWorldExample Servlet from Tomcat and a direct port for Simple. Both Simple and Tomcat were run using the Java HotSpot/1.4 interpreter and the server OS used was Solaris 8 on a dual 1002 MHz processor with 2048 MB of RAM. The benchmark was performed several times to ensure that the comparison made was fair and unbiased. For those skeptics the autobench results and configuration files for all four tests can be downloaded from here. The results have been saved in a format compatible with Microsoft Excel.

The first set of results illustrated in Figure 5 demonstrates the response rate comparison of the two application servers. Here the response rate of Tomcat is slightly higher when the stress on the server is low, however as the stress increases Tomcat continues to degrade its response times and as can be seen from Figure 7 this also coincides with a large number of dropped requests. Simple throughout the increase in stress continues to sustain a consistent response rate and maintains a higher throughput.

Figure 5   The response rate comparison with Tomcat

The next set of results illustrate how both servers cope under stress and provides a more accurate picture of performance. Unlike the response rate which, as explained earlier, only indicates the time taken for requests to complete, the throughput indicates the volume of content delivered over a peroid of time. For example, if the server dropped a large percentage of requests then the response rate will only indicate the time taken to respond a small number of successful requests. So the response rate is clearly not a useful measurement unless it is paired with the measurement of errors. The throughput combines the response rate with the error rate for a much clearer picture of what is happening. Below the graph of throughput indicates how Tomcat deteriorates significantly as stress is increased.

Figure 6   The throughput comparison with Tomcat

From the tests performed on Tomcat it is also apparent that it does not deal with concurrency effectively. The error rate for Tomcat was much higher than Simple and increased uniformly with the stress applied. The graph in Figure 7 demonstrates that Simple has a significantly better tolerance for stress than Tomcat.

Figure 7   The error rate comparison with Tomcat