Benchmarking PHP Frameworks

There are many reasons for choosing one framework over another, personally, my primary concern is making the development cycle as painless as possible but I can't really ignore some of the real world implications of using frameworks - that application performance is greatly effected.  There are valid reasons for this, frameworks sort out the DB connection, url handing and a raft of other bits and bobs so a performance hit is expected - we need all that stuff anyway, right? Thing is, these components can be developed in many ways and for many purposes so as a little experiment I thought I'd benchmark the major frameworks using there 'out-the-box' configuration and a basic page.

The Page

The page simply pulls out all the records in a particular table and displays the results, the page can be viewed here.

This simple page will test the framework initialisation, routing handler, the ORM and output rendering.

The Frameworks

For the experiment I chose Cake (1.1.2), Zend (1.6.2 min) and PHPWAX (0.8) - as a raw benchmark I knocked up the example page using bare bones PHP.  I also tired to setup Symphony but it just didn't happen on the day - I'll get back to it at some point tho and update the results.  The code for all these frameworks with the app included can be downloaded here (3.5mb zip).

The Test

I used ApacheBench to generate the results and tested using a 2Ghz Core Duo Mac with 1 GB RAM.  The web server was Apache 2.0.59 running PHP 5.2.6.

The Results

At generating 1000 requests with a max of 5 concurrent requests the results were...

Bare Bones PHP
handling 754.72 requests per second

--------------------
The Frameworks
--------------------
1. PHPWAX
handing 28.30 requests per second

2. CAKE PHP
handling 26.57 requests per second

3. Zend
handling 21.27 requests per second

Conclusion

Firstly I was taken aback in the difference between bare bones PHP and the Frameworks - over 25x the performance. This demonstrates the amount of processing a Framework believes is necessary for even the simplest of requests.

Having a personal interest in WAX I was happy to see it come up trumps. In reality the relative closeness of Cake and even Zend (to a degree) suggests that migrating to any of these Frameworks based purely on performance would probably negated by the learning curve associated with development changes.  These tests have been fairly narrow but I've posted the results as a starting point, more will follow as I expand the test cases and framework selection.

ApacheBench Output

john:~/Sites/stats frogteacher$ ab -n 1000 -c 5 http://benchmark_basic.localhost/
This is ApacheBench, Version 1.3d <$Revision: 1.73 INSERT:CONTENT:ENDgt; apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking benchmark_basic.localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests
Server Software:        Apache/2.0.59                                      
Server Hostname:        benchmark_basic.localhost
Server Port:            80

Document Path:          /
Document Length:        1898 bytes

Concurrency Level:      5
Time taken for tests:   1.325 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      2091000 bytes
HTML transferred:       1898000 bytes
Requests per second:    754.72 [#/sec] (mean)
Time per request:       6.62 [ms] (mean)
Time per request:       1.33 [ms] (mean, across all concurrent requests)
Transfer rate:          1578.11 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.0      0     1
Processing:     1     6    8.7      4    99
Waiting:        0     6    8.7      4    99
Total:          1     6    8.7      4    99
ERROR: The median and mean for the initial connection time are more than twice the standard
       deviation apart. These results are NOT reliable.

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      6
  75%      6
  80%      7
  90%     10
  95%     15
  98%     33
  99%     48
 100%     99 (last request)




john:~/Sites/stats frogteacher$ ab -n 1000 -c 5 http://benchmark_zend.localhost/
This is ApacheBench, Version 1.3d <$Revision: 1.73 INSERT:CONTENT:ENDgt; apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking benchmark_zend.localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests
Server Software:        Apache/2.0.59                                      
Server Hostname:        benchmark_zend.localhost
Server Port:            80

Document Path:          /
Document Length:        1855 bytes

Concurrency Level:      5
Time taken for tests:   47.025 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      2048000 bytes
HTML transferred:       1855000 bytes
Requests per second:    21.27 [#/sec] (mean)
Time per request:       235.12 [ms] (mean)
Time per request:       47.03 [ms] (mean, across all concurrent requests)
Transfer rate:          43.55 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.0      0     1
Processing:    74   234  108.2    224   725
Waiting:       74   234  108.2    224   725
Total:         74   234  108.2    224   725
ERROR: The median and mean for the initial connection time are more than twice the standard
       deviation apart. These results are NOT reliable.

Percentage of the requests served within a certain time (ms)
  50%    224
  66%    260
  75%    286
  80%    317
  90%    379
  95%    432
  98%    523
  99%    564
 100%    725 (last request)




john:~/Sites/stats frogteacher$ ab -n 1000 -c 5 http://benchmark_cake.localhost/
This is ApacheBench, Version 1.3d <$Revision: 1.73 INSERT:CONTENT:ENDgt; apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking benchmark_cake.localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests
Server Software:        Apache/2.0.59                                      
Server Hostname:        benchmark_cake.localhost
Server Port:            80

Document Path:          /
Document Length:        1845 bytes

Concurrency Level:      5
Time taken for tests:   37.631 seconds
Complete requests:      1000
Failed requests:        83
   (Connect: 0, Length: 83, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      2156913 bytes
HTML transferred:       1844913 bytes
Requests per second:    26.57 [#/sec] (mean)
Time per request:       188.15 [ms] (mean)
Time per request:       37.63 [ms] (mean, across all concurrent requests)
Transfer rate:          57.32 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.0      0     1
Processing:    62   187   95.2    175   619
Waiting:       62   187   95.2    175   618
Total:         62   187   95.2    175   619
ERROR: The median and mean for the initial connection time are more than twice the standard
       deviation apart. These results are NOT reliable.

Percentage of the requests served within a certain time (ms)
  50%    175
  66%    209
  75%    236
  80%    264
  90%    321
  95%    361
  98%    436
  99%    479
 100%    619 (last request)



john:~/Sites/stats frogteacher$ ab -n 1000 -c 5 http://benchmark_wax.localhost/
This is ApacheBench, Version 1.3d <$Revision: 1.73 INSERT:CONTENT:ENDgt; apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking benchmark_wax.localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests
Server Software:        Apache/2.0.59                                      
Server Hostname:        benchmark_wax.localhost
Server Port:            80

Document Path:          /
Document Length:        1827 bytes

Concurrency Level:      5
Time taken for tests:   35.339 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      2156000 bytes
HTML transferred:       1827000 bytes
Requests per second:    28.30 [#/sec] (mean)
Time per request:       176.70 [ms] (mean)
Time per request:       35.34 [ms] (mean, across all concurrent requests)
Transfer rate:          61.01 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.0      0     0
Processing:    55   176  141.9    132   855
Waiting:       55   176  142.0    132   855
Total:         55   176  141.9    132   855

Percentage of the requests served within a certain time (ms)
  50%    132
  66%    182
  75%    233
  80%    266
  90%    381
  95%    467
  98%    611
  99%    709
 100%    855 (last request)

More from this section