Our production environment takes roughly 7 secs to fully render, of which it doesn't start rending until 3.4 secs. So rather than attempting all the great information listed here we built an identical Staging environment with the same exact setup as production. I will include our AB stats along with all the configuration information in hope that some of you might be able to point me in the right direction.
Any help would be greatly appreciated.
Server Layout (All RH 5.6 servers) & All running on Fibre SAN
DIRTY DMZ
Proxy - Resides on a DMZ on the least secure VLAN of our FW.
* Varnish & Pound - Varnish for http and Pound for SSL relaying to our web heads
* 2 cores
* 2GB RAM
SOLR - Purely for Search indexes
* Apache Solr
* 2 cores
* 2GB RAM
SAFE DMZ
Web Head
* Pressflow w/APC and Memcache enabled. I also installed the CacheRouter module, however, I cannot tell if it is working properly.
* Apache 2.2.3, PHP5.2.17
* 4 cores
* 2GB RAM
DB Server
* MySQL 5.0.77
* 4 cores
* 4GB RAM
AB Test from Web Head:
ab -A -n 2000 -c 10 http://staging.xxxxxx.org/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking staging.xxxxxxx.org (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Finished 2000 requests
Server Software: Apache
Server Hostname: staging.xxxxxxxx.org
Server Port: 80
Document Path: /
Document Length: 51227 bytes
Concurrency Level: 10
Time taken for tests: 767.664779 seconds
Complete requests: 2000
Failed requests: 0
Write errors: 0
Total transferred: 103328000 bytes
HTML transferred: 102454000 bytes
Requests per second: 2.61 [#/sec] (mean)
Time per request: 3838.324 [ms] (mean)
Time per request: 383.832 [ms] (mean, across all concurrent requests)
Transfer rate: 131.45 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.5 0 43
Processing: 1997 3832 4263.8 2972 51331
Waiting: 1938 3757 4245.8 2910 51250
Total: 1997 3833 4263.8 2972 51331
Percentage of the requests served within a certain time (ms)
50% 2972
66% 3163
75% 3322
80% 3480
90% 4769
95% 6220
98% 17109
99% 29688
100% 51331 (longest request)
****** TOP - While AB was running *******
top - 15:15:55 up 12 days, 6:19, 2 users, load average: 14.78, 10.14, 4.97
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26897 apache 16 0 171m 86m 6504 R 27.9 4.2 0:41.02 httpd
26900 apache 17 0 166m 80m 6336 R 40.8 4.0 0:58.27 httpd
26902 apache 18 0 171m 79m 5792 R 6.6 3.9 0:56.88 httpd
26866 apache 16 0 163m 77m 6504 S 0.0 3.8 0:48.62 httpd
26869 apache 16 0 162m 75m 5784 S 0.0 3.7 0:46.56 httpd
26870 apache 16 0 166m 72m 5972 S 0.0 3.6 0:47.56 httpd
26875 apache 15 0 156m 71m 6360 S 0.0 3.5 0:43.09 httpd
26893 apache 18 0 155m 70m 6608 R 21.2 3.5 0:55.39 httpd
26888 apache 17 0 154m 70m 6340 R 36.7 3.5 0:45.24 httpd
26868 apache 16 0 155m 70m 5872 S 0.0 3.5 0:46.81 httpd
26901 apache 16 0 154m 69m 5784 S 0.0 3.4 0:49.71 httpd
26892 apache 16 0 155m 68m 6360 S 0.0 3.4 0:43.59 httpd
26876 apache 18 0 142m 63m 6868 R 18.7 3.1 1:04.32 httpd
26874 apache 15 0 161m 61m 5780 S 0.0 3.0 0:43.95 httpd
26882 apache 16 0 137m 57m 6452 S 0.0 2.8 0:55.58 httpd
26898 apache 16 0 165m 54m 6360 S 3.5 2.7 0:40.80 httpd
26903 apache 16 0 162m 50m 5668 R 44.3 2.5 0:51.27 httpd
26889 apache 16 0 155m 49m 6572 S 0.0 2.5 0:51.53 httpd
26880 apache 17 0 125m 49m 6448 R 12.7 2.4 1:02.80 httpd
26904 apache 17 0 126m 48m 5788 R 46.2 2.4 0:55.72 httpd
26891 apache 16 0 157m 48m 6476 S 0.0 2.4 0:51.71 httpd
26886 apache 17 0 119m 45m 6432 R 16.1 2.2 0:45.71 httpd
26883 apache 15 0 120m 43m 6616 S 0.0 2.2 0:45.97 httpd
26879 apache 16 0 119m 43m 7168 S 0.0 2.2 0:46.45 httpd
26887 apache 16 0 119m 42m 6568 S 0.0 2.1 0:49.38 httpd
26867 apache 16 0 115m 40m 5908 S 0.0 2.0 0:48.33 httpd
26873 apache 15 0 110m 36m 7376 S 0.0 1.8 0:49.43 httpd
26890 apache 18 0 112m 36m 6344 R 0.0 1.8 1:00.11 httpd
26896 apache 15 0 110m 36m 7284 S 0.0 1.8 0:49.61 httpd
26899 apache 16 0 111m 36m 6520 S 0.0 1.8 0:48.72 httpd
26878 apache 16 0 110m 36m 6596 S 0.0 1.8 0:50.96 httpd
26877 apache 15 0 111m 35m 6436 S 0.0 1.8 0:51.67 httpd
26884 apache 16 0 111m 35m 6424 S 0.0 1.8 0:52.34 httpd
26871 apache 16 0 109m 35m 5736 S 0.0 1.7 0:40.85 httpd
26885 apache 16 0 109m 34m 5936 S 0.0 1.7 0:47.35 httpd
26894 apache 16 0 109m 34m 5904 S 8.9 1.7 0:49.96 httpd
26895 apache 15 0 108m 34m 6672 S 0.0 1.7 0:53.07 httpd
26881 apache 15 0 108m 34m 6596 S 0.0 1.7 1:03.08 httpd
26872 apache 16 0 108m 34m 6592 S 0.0 1.7 0:45.10 httpd
26905 apache 16 0 108m 33m 6340 S 0.0 1.7 0:50.93 httpd
26864 root 18 0 75856 8252 4996 S 0.0 0.4 0:00.49 httpd
CONFIGURATIONS
httpd.conf - prefork:
StartServers 40
MinSpareServers 40
MaxSpareServers 40
ServerLimit 40
MaxClients 40
MaxRequestsPerChild 50000
httpd.conf - worker:
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
Settings.php:
$conf['cache_inc'] = './sites/all/modules/contrib/cacherouter/cacherouter.inc';
$conf['cacherouter'] = array(
'default' => array(
'engine' => 'db',
'servers' => array(
'xxx.xxx.xxx.xxx:11211' => 'default',
'xxx.xxx.xxx.xxx:11212' => 'page',
'xxx.xxx.xxx.xxx:11213' => 'views',
'xxx.xxx.xxx.xxx:11214' => 'other',
),
'shared' => TRUE,
'prefix' => '',
'path' => 'sites/default/files/filecache',
'static' => FALSE,
'fast_cache' => TRUE,
),
);
PHP.ini
extension=apc.so
apc.enabled = 1
apc.shm_size = 48M
apc.rfc1867 = 1
; apc.stat = 0
/etc/php.d/memcache.ini
; Enable memcache extension module
extension=memcache.so
; Options for the memcache module
; Whether to transparently failover to other servers on errors
;memcache.allow_failover=1
; Defines how many servers to try when setting and getting data.
;memcache.max_failover_attempts=20
; Data will be transferred in chunks of this size
;memcache.chunk_size=8192
; The default TCP port number to use when connecting to the memcached server
;memcache.default_port=11211
; Hash function {crc32, fnv}
;memcache.hash_function=crc32
; Hash strategy {standard, consistent}
;memcache.hash_strategy=standard
; Options to use the memcache session handler
; Use memcache as a session handler
;session.save_handler=memcache
; Defines a comma separated of server urls to use for session storage
;session.save_path="tcp://localhost:11211?persistent=1&weight=1&timeout=1&retry_interval=15"
/etc/httpd/conf.d/pressflow.conf
ServerName staging.xxxxxxxxx.org
DocumentRoot /var/www/html/current/
AllowOverride All
AuthType Basic
AuthName Pressflow
AuthUserFile /etc/httpd/conf.d/pressflow.htpasswd
Require valid-user
Deny From All
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/javascript application/x-javascript text/javascript application/xml application/rss+xml application/rss application/rss+atom
DeflateCompressionLevel 8
</IfModule>
<IfModule mod_expires.c>
ExpiresActive On
ExpiresDefault "access plus 2 hours"
ExpiresByType image/png "access plus 12 hours"
</IfModule>
Header unset ETag
# Set special caching rules for PDF's
<FilesMatch "\.(pdf|PDF)$">
Header set Cache-Control "max-age=10, public, must-revalidate"
Header unset Last-Modified
</FilesMatch>
Include /etc/httpd/conf.d/pressflow.redirects
CustomLog /var/log/httpd/staging.xxxxxxxx.org-access_log combined
ErrorLog /var/log/httpd/staging.xxxxxxxx.org-error_log
FileETag none
php_value sendmail_from "noreply@xxxxxxxxxx.org"
php_value error_reporting 22519
php_value date.timezone America/xxxxxxxxxxxxx
php_value error_log /var/www/html/staging.xxxxxxxxxxxxx.org-php_log
I also have stats from the proxy, however, I figured to start on the Pressflow server first. Any ideas?