Since having attended DrupalCon Chicago (and really well before) we have tried to absorb as much as we could from the advice of the performance community and implement it.
We haven't yet implemented Varnish/memcache/apc..., but we would like to take this (varnish) as our next step. Memory is a bit of a concern for us, we have a handful of services all running on one 8GB machine with a fair amount of daily traffic. I aplogize for the loads of data ahead, but i figured better to write too much rather than not enough.
We are primarily concerned with,
1) Are we asking for too much out of one box? Is it ram, cpu, something else? What is our bottleneck?
2) If we aren't asking too much, what are realistic targets we can acheive in terms of page load time, and what might be safe resource allocation to varnish (and everything really) when we turn it on?
3) If we are asking too much, what would be the best use of additional hardware?
Our environment
Traffic Metrics
Average Per Day (pageviews) 48,786
Average Per Visit (views per visitor) 1.4
99.9% traffic is anonymous (may be growing soon, but always overwhelmingly anon)
Quick breakdown of services all on one box
Apache
Solr
MySQL
Soon to be varnish?
Server Stats
2xE5[34]*; 8GB; 4 drive bays; 80GB+80GB+80GB+80GB disks; Ubuntu 8.04 (LTS); CP: none; DT: 1000GB of Data Transfer;
chris@hefty:~/w/files$ sudo cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
stepping : 6
cpu MHz : 2333.406
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 4669.56
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
stepping : 6
cpu MHz : 2333.406
cache size : 6144 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 4666.88
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
stepping : 6
cpu MHz : 2333.406
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 4666.87
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
stepping : 6
cpu MHz : 2333.406
cache size : 6144 KB
physical id : 1
siblings : 4
core id : 1
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 4666.88
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
stepping : 6
cpu MHz : 2333.406
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 4699.07
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
stepping : 6
cpu MHz : 2333.406
cache size : 6144 KB
physical id : 1
siblings : 4
core id : 2
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 4666.88
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
stepping : 6
cpu MHz : 2333.406
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 4666.86
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
stepping : 6
cpu MHz : 2333.406
cache size : 6144 KB
physical id : 1
siblings : 4
core id : 3
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips : 4666.88
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
chris@hefty:~/w/files$ sudo cat /proc/meminfo
MemTotal: 8190564 kB
MemFree: 2412312 kB
Buffers: 569200 kB
Cached: 3065432 kB
SwapCached: 340012 kB
Active: 3712860 kB
Inactive: 1595000 kB
SwapTotal: 2048276 kB
SwapFree: 1531252 kB
Dirty: 4984 kB
Writeback: 324 kB
AnonPages: 1670104 kB
Mapped: 66756 kB
Slab: 401712 kB
SReclaimable: 326596 kB
SUnreclaim: 75116 kB
PageTables: 23068 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 6143556 kB
Committed_AS: 2525548 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 25220 kB
VmallocChunk: 34359712907 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Web server
Apache/2.2.8 (Ubuntu)
PHP 5.2.4-2ubuntu5.14
KeepAlive Off (recently switched... seemed to be hurting us on)
Database server
MySQL 5.0.51a (is this too old?)
Storage Engine: InnoDB (recent switch... though 2 of ~130 tables are still MyISAM)
From drupal admin/reports/status/sql
SQL
Command counters
Variable Value Description
Variable Value Description
Com_select 8 The number of SELECT-statements.
Com_insert 0 The number of INSERT-statements.
Com_update 0 The number of UPDATE-statements.
Com_delete 0 The number of DELETE-statements.
Com_lock_tables 0 The number of table locks.
Com_unlock_tables 0 The number of table unlocks.
Query performance
Variable Value Description
Variable Value Description
Select_full_join 0 The number of joins without an index; should be zero.
Select_range_check 0 The number of joins without keys that check for key usage after each row; should be zero.
Sort_scan 0 The number of sorts done without using an index; should be zero.
Table_locks_immediate 166401438 The number of times a lock could be acquired immediately.
Table_locks_waited 2529 The number of times the server had to wait for a lock.
Query cache information
The MySQL query cache can improve performance of your site by storing the result of queries. Then, if an identical query is received later, the MySQL server retrieves the result from the query cache rather than parsing and executing the statement again.
Variable Value Description
Variable Value Description
Qcache_queries_in_cache 4396 The number of queries in the query cache.
Qcache_hits 272403531 The number of times MySQL found previous results in the cache.
Qcache_inserts 102608966 The number of times MySQL added a query to the cache (misses).
Qcache_lowmem_prunes 66772564 The number of times MySQL had to remove queries from the cache because it ran out of memory. Ideally should be zero.
From Drupal contrib module DB Tuner
Queries
Uptime in seconds: 880393
Uptime: 10d 4h 33m 13s
Questions: 415406057
% slow queries: 0.00580203383987
slow query rate: 0.000569399942713 per day
Long query time: 1
Slow query logging: ON
% reads: 79.6215234635
% writes: 20.3784765365
qps: 471.841617323
reads per sec: 0.0781389632499 per day
writes per sec: 0.0199990273975 per day
Queries: 471.841617323 per second
Connections: 2 Million
Bytes sent: 3751 Billion
Bytes received: 97 Billion
versions
Supported Version: 5
Release Series: 5.0
version less then 5.1, upgrade!
(substr("version",0,3) ne "5.1")
(substr("5.0.51a-3ubuntu5.8-log",0,3) '5.0' !== "5.1")
Minor Version: 51
Distribution: (Ubuntu)
Distribution: (Ubuntu)
MySQL Architecture: x86_64
Query cache
Query cache efficiency (%): 71.5765302197
% query cache used: 26.983165741
The query cache is not being fully utilized.
(Qcache_free_memory / query_cache_size * 100 <80)
(4527024 / 16777216 * 100 26.983165741<80)
Query cache low memory prunes: 75.9171688098 per second
Increase query_cache_size -- there are too many low memory prunes.
(&hr_bytime(Qcache_lowmem_prunes/Uptime_since_flush_status) =~ /second|minute/)
(dbtuner_hr_bytime(66836944/880393) dbtuner_stristr('75.9171688098 per second', array('second', 'minute')))
Query cache size: 16.0 Mb
Query cache min result size: 1.0 Mb
The max size of the result set in the query cache is the default of 1 Mb. Changing this (usually by increasing) may increase efficiency.
(&hr_bytes(query_cache_limit) eq "1.0 Mb")
(dbtuner_hr_bytes(1048576) '1.0 Mb' === "1.0 Mb")
Sorts
Total sorts: 10036528
% sorts that cause temporary tables: 0.14190166161
rate of sorts that cause temporary tables: 58.2367192833 per hour
sort_buffer_size: 2.0 Mb
read_rnd_buffer_size: 256.0 Kb
Sort rows: 12444.3930256 per second
There are lots of rows being sorted. Consider using indexes in more queries to avoid sorting too often.
(&hr_bytime(Sort_rows/Uptime_since_flush_status) =~ /second|minute/)
(dbtuner_hr_bytime(10955956509/880393) dbtuner_stristr('12444.3930256 per second', array('second', 'minute')))
Joins,scans
rate of joins without indexes: 2.17890873735 per second
There are too many joins without indexes -- this means that joins are doing full table scans.
(&hr_bytime((Select_range_check + Select_scan + Select_full_join)/Uptime_since_flush_status) =~ /second|minute/)
(dbtuner_hr_bytime((0 + 1801031 + 117265)/880393) dbtuner_stristr('2.17890873735 per second', array('second', 'minute')))
rate of reading first index entry: 4.73679595363 per second
The rate of reading the first index entry is high; this usually indicates frequent full index scans.
(&hr_bytime(Handler_read_first/Uptime_since_flush_status) =~ /second|minute/)
(dbtuner_hr_bytime(4170242/880393) dbtuner_stristr('4.73679595363 per second', array('second', 'minute')))
rate of reading fixed position: 519.470768168 per second
The rate of reading data from a fixed position is high; this indicates many queries need to sort results and/or do a full table scan, including join queries that do not use indexes.
(&hr_bytime(Handler_read_rnd/Uptime_since_flush_status) =~ /second|minute/)
(dbtuner_hr_bytime(457338428/880393) dbtuner_stristr('519.470768168 per second', array('second', 'minute')))
rate of reading next table row: 20625.7471254 per second
The rate of reading the next table row is high; this indicates many queries are doing full table scans.
(&hr_bytime(Handler_read_rnd_next/Uptime_since_flush_status) =~ /second|minute/)
(dbtuner_hr_bytime(18158763389/880393) dbtuner_stristr('20625.7471254 per second', array('second', 'minute')))
temp tables
tmp_table_size-max_heap_table_size: 16777216
tmp_table_size and max_heap_table_size are not the same.
(tmp_table_size-max_heap_table_size !=0)
(33554432-16777216 16777216!=0)
tmp_table_size: 32.0 Mb
max_heap_table_size: 16.0 Mb
% temp disk tables: 33.6876432074
Too many temporary tables are being written to disk. Increase max_heap_table_size and tmp_table_size.
(Created_tmp_disk_tables / (Created_tmp_tables + Created_tmp_disk_tables) * 100 >25)
(4037397 / (7947404 + 4037397) * 100 33.6876432074>25)
temp disk rate: 4.58590311372 per second
Too many temporary tables are being written to disk. Increase max_heap_table_size and tmp_table_size.
(&hr_bytime(Created_tmp_disk_tables/Uptime_since_flush_status) =~ /second|minute/)
(dbtuner_hr_bytime(4037397/880393) dbtuner_stristr('4.58590311372 per second', array('second', 'minute')))
temp table rate: 9.02710948406 per second
Too many intermediate temporary tables are being created; consider increasing sort_buffer_size (sorting), read_rnd_buffer_size (random read buffer, ie, post-sort), read_buffer_size (sequential scan).
(&hr_bytime(Created_tmp_tables/Uptime_since_flush_status) =~ /second|minute/)
(dbtuner_hr_bytime(7947404/880393) dbtuner_stristr('9.02710948406 per second', array('second', 'minute')))
MyISAM index cache
MyISAM key buffer size: 16.0 Mb
max % MyISAM key buffer ever used: 81.7626953125
MyISAM key buffer (index cache) % used is low. You may need to decrease the size of key_buffer_size, re-examine your tables to see if indexes have been removed, or examine queries and expectations about what indexes are being used.
((Key_blocks_used)key_cache_block_size/key_buffer_size * 100 <95)
((13396)1024/16777216 * 100 81.7626953125<95)
% MyISAM key buffer used: 19.9279785156
MyISAM key buffer (index cache) % used is low. You may need to decrease the size of key_buffer_size, re-examine your tables to see if indexes have been removed, or examine queries and expectations about what indexes are being used.
((1-Key_blocks_unusedkey_cache_block_size/key_buffer_size) * 100 <95)
((1-131191024/16777216) * 100 19.9279785156<95)
% index reads from memory: 96.0059189303
other caches
table open cache size (5.1+): table_open_cache
Size of the table cache
(table_open_cache >-1)
(table_open_cache table_open_cache>-1)
rate of table open: 8.8613959902 per second
The rate of opening tables is high, increase table_open_cache to avoid this.
(&hr_bytime(Opened_tables/Uptime_since_flush_status) =~ /second|minute/)
(dbtuner_hr_bytime(7801511/880393) dbtuner_stristr('8.8613959902 per second', array('second', 'minute')))
% open files: 0.68359375
rate of open files: 0.686965934532 per day
Immediate table locks %: 99.9984796862
Table lock wait rate: 10.3535580133 per hour
thread cache: 8
Total threads created: 16083
thread cache hit rate %: 0.00642780835435
Threads that are slow to launch: 4
There are too many threads that are slow to launch
(Slow_launch_threads >0)
(4 4>0)
Slow launch time: 2
Connections
% connections used: 22.6666666667
Max connections used: 102
Max connections limit: 450
% aborted connections: 3.99664761198E-5
rate of aborted connections: 0.0981379906474 per day
% aborted clients: 0.013348803024
rate of aborted clients: 1.36575370318 per hour
InnoDB
Is InnoDB enabled?: YES
% innoDB log size: 62.5
InnoDB log file size is not an appropriate size, in relation to the InnoDB buffer pool. Consider changing either\ninnodb_log_file_size or innodb_buffer_pool_size
(innodb_log_file_size / innodb_buffer_pool_size * 100 >=0)
(5242880 / 8388608 * 100 62.5>=0)
other
MyISAM concurrent inserts: 1
INSERT DELAYED USAGE
Delayed_errors 0
Delayed_insert_threads 0
Delayed_writes 0
Not_flushed_delayed_rows
Drupal environment
Pressflow 6.20
Modules
148 enabled including 23 custom
CDN 6.x-2.1 (origin pull, two different IPs and subdomains...cookie free... all pointing to the same box)
Boost 6.x-1.18 (I've read dev is better)
Solr 6.x-1.2
GTMetrix Analysis:
Homepage on average about 2MB and 200 resources (yes I know that's a lot... we have a lot of media).
I have used image reduction software (jpegoptim) to help with size
Our scores are roughly 83 and 60 for Google Page Speed and YSlow respectively
Thanks so much for your advice!
- Chris