I am in the process of testing a new cluster. For this cluster we have 4 nodes with each configuration equal:
2 x Xeon E5620 prcoessors
32 GB RAM
160 GB SSD for Proxmox VE
3 x 4 TB WD Black WD4003FZEX disks for CEPH
2 x Intel Gigabit NIC's, 1 main IP and 1 for the storage network
I have created the CEPH cluster and configured all nodes to be a monitor. Each disk is added as an OSD which totals to 12 OSD's. The ceph pool has a size of 3 and pg_num is 512.
I created 1 KVM to benchmark and I think the performance is poor:
Ran the benchmarks with:
And:
Inbetween the write and the read I did:
On all hosts and the guest, as per instructions I found on other topics.
After this I tried different options in ceph.conf, I added:
I added this code under [osd]
Unfortunately this helped a little with read:
And:
Main issue is: I feel that the KVM server is sloppy, not good performing at all.
Questions:
1. Is this performance expected for the current config?
2. If 1=no, what could be the problem? What can we do to get better performance except adding more OSD's?
3. If 1=yes, what would be the expected performance if we expand this cluster to 8 nodes with the same config? Double the performance now (which is still bad write performance!) or more?
2 x Xeon E5620 prcoessors
32 GB RAM
160 GB SSD for Proxmox VE
3 x 4 TB WD Black WD4003FZEX disks for CEPH
2 x Intel Gigabit NIC's, 1 main IP and 1 for the storage network
I have created the CEPH cluster and configured all nodes to be a monitor. Each disk is added as an OSD which totals to 12 OSD's. The ceph pool has a size of 3 and pg_num is 512.
I created 1 KVM to benchmark and I think the performance is poor:
Code:
Run status group 0 (all jobs):
WRITE: io=5120.0MB, aggrb=15889KB/s, minb=15889KB/s, maxb=15889KB/s, mint=329962msec, maxt=329962msec
Run status group 0 (all jobs):
READ: io=5120.0MB, aggrb=47242KB/s, minb=47242KB/s, maxb=47242KB/s, mint=110977msec, maxt=110977msecRan the benchmarks with:
Code:
fio --max-jobs=1 --numjobs=1 --readwrite=write --blocksize=4M --size=5G --direct=1 --name=fiojobCode:
fio --max-jobs=1 --numjobs=1 --readwrite=read --blocksize=4M --size=5G --direct=1 --name=fiojobCode:
echo 3 > /proc/sys/vm/drop_cachesAfter this I tried different options in ceph.conf, I added:
Code:
osd mkfs options xfs = "-f -i size=2048"
osd mount options xfs = "rw,noatime,logbsize=256k,logbufs=8,inode64,al$
osd op threads = 8
osd max backfills = 1
osd recovery max active = 1
filestore max sync interval = 100
filestore min sync interval = 50
filestore queue max ops = 10000
filestore queue max bytes = 536870912
filestore queue committing max ops = 2000
filestore queue committing max bytes = 536870912Unfortunately this helped a little with read:
Code:
Run status group 0 (all jobs):
WRITE: io=5120.0MB, aggrb=15548KB/s, minb=15548KB/s, maxb=15548KB/s, mint=337206msec, maxt=337206msecCode:
Run status group 0 (all jobs):
READ: io=5120.0MB, aggrb=51013KB/s, minb=51013KB/s, maxb=51013KB/s, mint=102775msec, maxt=102775msecMain issue is: I feel that the KVM server is sloppy, not good performing at all.
Questions:
1. Is this performance expected for the current config?
2. If 1=no, what could be the problem? What can we do to get better performance except adding more OSD's?
3. If 1=yes, what would be the expected performance if we expand this cluster to 8 nodes with the same config? Double the performance now (which is still bad write performance!) or more?