Hi, what about aio support in vzdump? In my tests it shows significantly performance gain!
ISCSI MULTIPATH (2x1Gbit round robin) TO IBM STORWIZE 3700:
WITHOUT AIO:
fio --filename=/dev/mapper/36005076300808090c000000000000007 --direct=1 --rw=read --runtime=30 --group_reporting --name=file1
file1: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
2.0.8
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [11120K/0K /s] [2780 /0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=926539
read : io=333580KB, bw=11119KB/s, iops=2779 , runt= 30001msec
clat (usec): min=252 , max=40184 , avg=355.94, stdev=231.80
lat (usec): min=252 , max=40184 , avg=356.27, stdev=231.80
clat percentiles (usec):
| 1.00th=[ 298], 5.00th=[ 306], 10.00th=[ 310], 20.00th=[ 314],
| 30.00th=[ 322], 40.00th=[ 334], 50.00th=[ 350], 60.00th=[ 358],
| 70.00th=[ 362], 80.00th=[ 370], 90.00th=[ 398], 95.00th=[ 418],
| 99.00th=[ 516], 99.50th=[ 588], 99.90th=[ 1224], 99.95th=[ 4016],
| 99.99th=[ 9664]
bw (KB/s) : min= 9056, max=12024, per=100.00%, avg=11121.17, stdev=419.93
lat (usec) : 500=98.55%, 750=1.24%, 1000=0.10%
lat (msec) : 2=0.05%, 4=0.02%, 10=0.04%, 20=0.01%, 50=0.01%
cpu : usr=1.33%, sys=6.26%, ctx=83473, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=83395/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=333580KB, aggrb=11118KB/s, minb=11118KB/s, maxb=11118KB/s, mint=30001msec, maxt=30001msec
WITH AIO:
fio --filename=/dev/mapper/36005076300808090c000000000000007 --direct=1 --rw=read --ioengine=libaio --iodepth=512 --runtime=30 --group_reporting --name=file1
file1: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=512
2.0.8
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [127.7M/0K /s] [32.7K/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=928506
read : io=3907.6MB, bw=133336KB/s, iops=33333 , runt= 30009msec
slat (usec): min=2 , max=3674 , avg=16.32, stdev=37.64
clat (usec): min=326 , max=79864 , avg=15337.34, stdev=10716.91
lat (usec): min=338 , max=79869 , avg=15354.26, stdev=10716.71
clat percentiles (usec):
| 1.00th=[ 1896], 5.00th=[ 3024], 10.00th=[ 3888], 20.00th=[ 5600],
| 30.00th=[ 7648], 40.00th=[10048], 50.00th=[12864], 60.00th=[15936],
| 70.00th=[19584], 80.00th=[24448], 90.00th=[30336], 95.00th=[35072],
| 99.00th=[47360], 99.50th=[52480], 99.90th=[62720], 99.95th=[66048],
| 99.99th=[75264]
bw (KB/s) : min=99392, max=159409, per=99.97%, avg=133297.73, stdev=14002.66
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.04%
lat (msec) : 2=1.19%, 4=9.49%, 10=28.99%, 20=31.21%, 50=28.38%
lat (msec) : 100=0.69%
cpu : usr=10.61%, sys=52.96%, ctx=114430, majf=0, minf=534
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=1000318/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=3907.6MB, aggrb=133335KB/s, minb=133335KB/s, maxb=133335KB/s, mint=30009msec, maxt=30009msec
ON LOCAL RAW DEVICE SDA (RAID10):
WITHOUT AIO:
fio --filename=/dev/sda --direct=1 --rw=read --runtime=30 --group_reporting --name=file1
file1: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
2.0.8
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [89232K/0K /s] [22.4K/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=929244
read : io=2686.9MB, bw=91709KB/s, iops=22927 , runt= 30001msec
clat (usec): min=22 , max=40262 , avg=41.23, stdev=108.31
lat (usec): min=22 , max=40262 , avg=41.49, stdev=108.31
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 32], 20.00th=[ 35],
| 30.00th=[ 39], 40.00th=[ 40], 50.00th=[ 41], 60.00th=[ 41],
| 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 47], 95.00th=[ 52],
| 99.00th=[ 67], 99.50th=[ 82], 99.90th=[ 173], 99.95th=[ 362],
| 99.99th=[ 924]
bw (KB/s) : min=73688, max=122848, per=100.00%, avg=91765.54, stdev=9699.91
lat (usec) : 50=92.92%, 100=6.77%, 250=0.23%, 500=0.05%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=10.66%, sys=37.56%, ctx=688124, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=687840/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=2686.9MB, aggrb=91708KB/s, minb=91708KB/s, maxb=91708KB/s, mint=30001msec, maxt=30001msec
Disk stats (read/write):
sda: ios=686223/1146, merge=0/1307, ticks=24117/260, in_queue=24124, util=76.49%
WITH AIO:
fio --filename=/dev/sda --direct=1 --rw=read --ioengine=libaio --iodepth=512 --runtime=30 --group_reporting --name=file1
file1: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=512
2.0.8
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [263.8M/0K /s] [67.4K/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=931177
read : io=7358.1MB, bw=251178KB/s, iops=62794 , runt= 30001msec
slat (usec): min=3 , max=42877 , avg=13.66, stdev=173.28
clat (usec): min=83 , max=64249 , avg=8135.70, stdev=4290.50
lat (usec): min=106 , max=64263 , avg=8149.63, stdev=4295.37
clat percentiles (usec):
| 1.00th=[ 4768], 5.00th=[ 5472], 10.00th=[ 5984], 20.00th=[ 6304],
| 30.00th=[ 6368], 40.00th=[ 6432], 50.00th=[ 6624], 60.00th=[ 7072],
| 70.00th=[ 8160], 80.00th=[ 9280], 90.00th=[10560], 95.00th=[13632],
| 99.00th=[28544], 99.50th=[35584], 99.90th=[47360], 99.95th=[51456],
| 99.99th=[52992]
bw (KB/s) : min=127720, max=335384, per=100.00%, avg=251593.14, stdev=62331.85
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=85.62%, 20=11.47%, 50=2.82%
lat (msec) : 100=0.06%
cpu : usr=13.61%, sys=71.61%, ctx=3574, majf=0, minf=535
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=1883898/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=7358.1MB, aggrb=251178KB/s, minb=251178KB/s, maxb=251178KB/s, mint=30001msec, maxt=30001msec
Disk stats (read/write):
sda: ios=1879372/1015, merge=4/1041, ticks=1103389/165, in_queue=1102570, util=99.42%
ISCSI MULTIPATH (2x1Gbit round robin) TO IBM STORWIZE 3700:
WITHOUT AIO:
fio --filename=/dev/mapper/36005076300808090c000000000000007 --direct=1 --rw=read --runtime=30 --group_reporting --name=file1
file1: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
2.0.8
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [11120K/0K /s] [2780 /0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=926539
read : io=333580KB, bw=11119KB/s, iops=2779 , runt= 30001msec
clat (usec): min=252 , max=40184 , avg=355.94, stdev=231.80
lat (usec): min=252 , max=40184 , avg=356.27, stdev=231.80
clat percentiles (usec):
| 1.00th=[ 298], 5.00th=[ 306], 10.00th=[ 310], 20.00th=[ 314],
| 30.00th=[ 322], 40.00th=[ 334], 50.00th=[ 350], 60.00th=[ 358],
| 70.00th=[ 362], 80.00th=[ 370], 90.00th=[ 398], 95.00th=[ 418],
| 99.00th=[ 516], 99.50th=[ 588], 99.90th=[ 1224], 99.95th=[ 4016],
| 99.99th=[ 9664]
bw (KB/s) : min= 9056, max=12024, per=100.00%, avg=11121.17, stdev=419.93
lat (usec) : 500=98.55%, 750=1.24%, 1000=0.10%
lat (msec) : 2=0.05%, 4=0.02%, 10=0.04%, 20=0.01%, 50=0.01%
cpu : usr=1.33%, sys=6.26%, ctx=83473, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=83395/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=333580KB, aggrb=11118KB/s, minb=11118KB/s, maxb=11118KB/s, mint=30001msec, maxt=30001msec
WITH AIO:
fio --filename=/dev/mapper/36005076300808090c000000000000007 --direct=1 --rw=read --ioengine=libaio --iodepth=512 --runtime=30 --group_reporting --name=file1
file1: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=512
2.0.8
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [127.7M/0K /s] [32.7K/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=928506
read : io=3907.6MB, bw=133336KB/s, iops=33333 , runt= 30009msec
slat (usec): min=2 , max=3674 , avg=16.32, stdev=37.64
clat (usec): min=326 , max=79864 , avg=15337.34, stdev=10716.91
lat (usec): min=338 , max=79869 , avg=15354.26, stdev=10716.71
clat percentiles (usec):
| 1.00th=[ 1896], 5.00th=[ 3024], 10.00th=[ 3888], 20.00th=[ 5600],
| 30.00th=[ 7648], 40.00th=[10048], 50.00th=[12864], 60.00th=[15936],
| 70.00th=[19584], 80.00th=[24448], 90.00th=[30336], 95.00th=[35072],
| 99.00th=[47360], 99.50th=[52480], 99.90th=[62720], 99.95th=[66048],
| 99.99th=[75264]
bw (KB/s) : min=99392, max=159409, per=99.97%, avg=133297.73, stdev=14002.66
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.04%
lat (msec) : 2=1.19%, 4=9.49%, 10=28.99%, 20=31.21%, 50=28.38%
lat (msec) : 100=0.69%
cpu : usr=10.61%, sys=52.96%, ctx=114430, majf=0, minf=534
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=1000318/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=3907.6MB, aggrb=133335KB/s, minb=133335KB/s, maxb=133335KB/s, mint=30009msec, maxt=30009msec
ON LOCAL RAW DEVICE SDA (RAID10):
WITHOUT AIO:
fio --filename=/dev/sda --direct=1 --rw=read --runtime=30 --group_reporting --name=file1
file1: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
2.0.8
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [89232K/0K /s] [22.4K/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=929244
read : io=2686.9MB, bw=91709KB/s, iops=22927 , runt= 30001msec
clat (usec): min=22 , max=40262 , avg=41.23, stdev=108.31
lat (usec): min=22 , max=40262 , avg=41.49, stdev=108.31
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 32], 20.00th=[ 35],
| 30.00th=[ 39], 40.00th=[ 40], 50.00th=[ 41], 60.00th=[ 41],
| 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 47], 95.00th=[ 52],
| 99.00th=[ 67], 99.50th=[ 82], 99.90th=[ 173], 99.95th=[ 362],
| 99.99th=[ 924]
bw (KB/s) : min=73688, max=122848, per=100.00%, avg=91765.54, stdev=9699.91
lat (usec) : 50=92.92%, 100=6.77%, 250=0.23%, 500=0.05%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=10.66%, sys=37.56%, ctx=688124, majf=0, minf=27
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=687840/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=2686.9MB, aggrb=91708KB/s, minb=91708KB/s, maxb=91708KB/s, mint=30001msec, maxt=30001msec
Disk stats (read/write):
sda: ios=686223/1146, merge=0/1307, ticks=24117/260, in_queue=24124, util=76.49%
WITH AIO:
fio --filename=/dev/sda --direct=1 --rw=read --ioengine=libaio --iodepth=512 --runtime=30 --group_reporting --name=file1
file1: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=512
2.0.8
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [263.8M/0K /s] [67.4K/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=931177
read : io=7358.1MB, bw=251178KB/s, iops=62794 , runt= 30001msec
slat (usec): min=3 , max=42877 , avg=13.66, stdev=173.28
clat (usec): min=83 , max=64249 , avg=8135.70, stdev=4290.50
lat (usec): min=106 , max=64263 , avg=8149.63, stdev=4295.37
clat percentiles (usec):
| 1.00th=[ 4768], 5.00th=[ 5472], 10.00th=[ 5984], 20.00th=[ 6304],
| 30.00th=[ 6368], 40.00th=[ 6432], 50.00th=[ 6624], 60.00th=[ 7072],
| 70.00th=[ 8160], 80.00th=[ 9280], 90.00th=[10560], 95.00th=[13632],
| 99.00th=[28544], 99.50th=[35584], 99.90th=[47360], 99.95th=[51456],
| 99.99th=[52992]
bw (KB/s) : min=127720, max=335384, per=100.00%, avg=251593.14, stdev=62331.85
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=85.62%, 20=11.47%, 50=2.82%
lat (msec) : 100=0.06%
cpu : usr=13.61%, sys=71.61%, ctx=3574, majf=0, minf=535
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=1883898/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=7358.1MB, aggrb=251178KB/s, minb=251178KB/s, maxb=251178KB/s, mint=30001msec, maxt=30001msec
Disk stats (read/write):
sda: ios=1879372/1015, merge=4/1041, ticks=1103389/165, in_queue=1102570, util=99.42%