Disk performance on guest incredibly low.

classic Classic list List threaded Threaded
9 messages Options
frm
Reply | Threaded
Open this post in threaded view
|

Disk performance on guest incredibly low.

frm
My disk performance of Xen guest is very slow. My latest configuration:

At guest, I have:

`    dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.6421 s, 68.6 MB/s
`

At Dom0 I have:

`    dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
    1024+0 records in
    1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.81855 s, 281 MB/s
`

The dom0 is about 5 times faster than the guest.

Data about the hypervisor
`    Xen version 4.11.1
Boot parameters:
    options=loglvl=all noreboot dom0_mem=2048M max=2048M dom0_max_vcpus=4
dom0_vcpus_pin iommu=verbose ucode=scan flask=disabled conring_size=2097152
    kernel=vmlinuz-4.18.0-0.bpo.1-amd64 root=/dev/mapper/nuc1-root ro
console=tty quiet splash vt.handoff=7
    ramdisk=initrd.img-4.18.0-0.bpo.1-amd64
`

config file guest:

`    bootloader = '/usr/local/bin/pygrub'
    extra="console=hvc0"
    vcpus       = '4'
    memory      = '8192'
    root        = '/dev/xvda1 ro'
    disk        = [
                  'phy:/dev/vm/compute-root,xvda1,w'
              ]
    type      = 'pvh'
`

Startup parameters guest:

`    root=/dev/xvda1 ro elevator=noop xen_blkfront.max_indirect_segments=256
root=/dev/xvda1 ro console=hvc0
`


If I have tried also hvm and pv virtualisation modes; not improvement.

Any ideas?


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Disk performance on guest incredibly low.

andy smith-10
Hi,

On Sun, Dec 16, 2018 at 05:42:43PM +0100, [hidden email] wrote:

> At guest, I have:
>
> `    dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
>     1024+0 records in
>     1024+0 records out
>     1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.6421 s, 68.6 MB/s
> `
>
> At Dom0 I have:
>
> `    dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
>     1024+0 records in
>     1024+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.81855 s, 281 MB/s
> `

No ideas, sorry, but just for comparison:

guest:
$ dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.51228 s, 427 MB/s

dom0:
$ dd if=/dev/zero of=/var/tmp/a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.61745 s, 410 MB/s

PV guest, xen 4.10.3.

So yes it seems you are seeing something quite abnormal.

Cheers,
Andy

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Disk performance on guest incredibly low.

Håkon Alstadheim
In reply to this post by frm

Den 16.12.2018 17:42, skrev [hidden email]:

> My disk performance of Xen guest is very slow. My latest configuration:
>
> At guest, I have:
>
> `    dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
>      1024+0 records in
>      1024+0 records out
>      1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.6421 s, 68.6 MB/s
> `
>
> At Dom0 I have:
>
> `    dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
>      1024+0 records in
>      1024+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.81855 s, 281 MB/s
> `
>
> The dom0 is about 5 times faster than the guest.
>
Worse here, 15 to 20 x slower (I did a few runs just now, numbers vary
by quite a lot, but the magnitude difference is consistent). Same md
partition mounted on dom0 and as domu root. Md constituents are SAS
spinning rust. In the past I've tried looking at tunables for the domu
disks, to no effect. I would love a "domu disk tuning for dummies" .
Throughput speed is tolerable usually, but when a rebuild is running on
the disk-array, the domus are all but unusable interactively. I've been
poking at this issue for a couple of years, not getting anywhere. The
poking has been totally devoid of proper notes and a strategy, but over
time I have ditched LVM and tried various schedulers, triple-checked
alignment, used both xfs and ext4, all without ever seeing a definitive
break-through. I had bcache with an m2 ssd on top of LVM for a while
until my m2 wore out. That helped paper over the issue. (bcache on top
of virtual devices is not recommended, btw. If done, make sure you use
no-discard on the disk-configuration for the vm)

Xentop:

xentop - 20:06:04   Xen 4.11.1
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%) VCPUS
   Domain-0 -----r      33821    8.1    4995856    7.5 8
  steam.hvm --b---         51    0.3    7332136   11.0 6

Guest steam:

# df -hT  /tmp
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  162G    25G   87% /
# cd /tmp
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 60,1407 s, 17,9 MB/s

Dom0:

  # df -hT ./
Filesystem     Type  Size  Used Avail Use% Mounted on

/dev/md2p8     ext4  196G  163G   24G  88% /mnt/tull

# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.37238 s, 200 MB/s



_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Disk performance on guest incredibly low.

Roalt Zijlstra | webpower
Hey Håkon,

One little tip on disk IO in Virtual hosts, is changing the FS scheduler from cfq to noop. In our Xen PV configs we add this to the xen.cfg files at the end:

extra="clocksource=tsc elevator=noop"

Especially the "elevator=noop" parameter is forcing all FS-es to the noop scheduler. In my experience that gave our Xenservers a pretty nice boost.
Red Hat does recoomend this for all Virtual servers (see the link below). To test this you don't need to reboot at all.

For example from the link below there is a code snippet about getting and setting the scheduler for /dev/hda. (Replace hda with sdb or any other block device)

# cat /sys/block/hda/queue/scheduler
noop anticipatory deadline [cfq]

# echo 'noop' > /sys/block/hda/queue/scheduler
# cat /sys/block/hda/queue/scheduler
[noop] anticipatory deadline cfq


Best regards,
 
 Roalt Zijlstra
  Teamleader Infra & Deliverability
   
 [hidden email]
 +31 342 423 262
 roalt.zijlstra
 https://www.webpower-group.com
 
 
Facebook Twitter Linkedin
Barcelona | Barneveld | Beijing | Chengdu | Guangzhou
Hamburg | Shanghai | Shenzhen | Stockholm
 


Op zo 16 dec. 2018 om 20:18 schreef Håkon Alstadheim <[hidden email]>:

Den 16.12.2018 17:42, skrev [hidden email]:
> My disk performance of Xen guest is very slow. My latest configuration:
>
> At guest, I have:
>
> `    dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
>      1024+0 records in
>      1024+0 records out
>      1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.6421 s, 68.6 MB/s
> `
>
> At Dom0 I have:
>
> `    dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
>      1024+0 records in
>      1024+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.81855 s, 281 MB/s
> `
>
> The dom0 is about 5 times faster than the guest.
>
Worse here, 15 to 20 x slower (I did a few runs just now, numbers vary
by quite a lot, but the magnitude difference is consistent). Same md
partition mounted on dom0 and as domu root. Md constituents are SAS
spinning rust. In the past I've tried looking at tunables for the domu
disks, to no effect. I would love a "domu disk tuning for dummies" .
Throughput speed is tolerable usually, but when a rebuild is running on
the disk-array, the domus are all but unusable interactively. I've been
poking at this issue for a couple of years, not getting anywhere. The
poking has been totally devoid of proper notes and a strategy, but over
time I have ditched LVM and tried various schedulers, triple-checked
alignment, used both xfs and ext4, all without ever seeing a definitive
break-through. I had bcache with an m2 ssd on top of LVM for a while
until my m2 wore out. That helped paper over the issue. (bcache on top
of virtual devices is not recommended, btw. If done, make sure you use
no-discard on the disk-configuration for the vm)

Xentop:

xentop - 20:06:04   Xen 4.11.1
      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%) VCPUS
   Domain-0 -----r      33821    8.1    4995856    7.5 8
  steam.hvm --b---         51    0.3    7332136   11.0 6

Guest steam:

# df -hT  /tmp
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  162G    25G   87% /
# cd /tmp
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 60,1407 s, 17,9 MB/s

Dom0:

  # df -hT ./
Filesystem     Type  Size  Used Avail Use% Mounted on

/dev/md2p8     ext4  196G  163G   24G  88% /mnt/tull

# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.37238 s, 200 MB/s



_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Disk performance on guest incredibly low.

Håkon Alstadheim

Den 17.12.2018 10:32, skrev Roalt Zijlstra | webpower:

> Hey Håkon,
>
> One little tip on disk IO in Virtual hosts, is changing the FS
> scheduler from cfq to noop. In our Xen PV configs we add this to the
> xen.cfg files at the end:
>
> extra="clocksource=tsc elevator=noop"
>
> Especially the "elevator=noop" parameter is forcing all FS-es to the
> noop scheduler. In my experience that gave our Xenservers a pretty
> nice boost.
> Red Hat does recoomend this for all Virtual servers (see the link
> below). To test this you don't need to reboot at all.
>
> For example from the link below there is a code snippet about getting
> and setting the scheduler for /dev/hda. (Replace hda with sdb or any
> other block device)
>
> # cat /sys/block/hda/queue/scheduler
> noop anticipatory deadline [cfq]
>
> # echo 'noop' > /sys/block/hda/queue/scheduler
> # cat /sys/block/hda/queue/scheduler
> [noop] anticipatory deadline cfq
>
> More info: https://access.redhat.com/solutions/5427

Yes, found that resource some time ago. Tried again now, no
break-through. I've got 'none' available as scheduler, rather than noop,
but they should be equal. No matter what I do (in dom0 and/or domu) I
get at least 10 x higher speed in the dom0. :-/ .

Example:

###

###In dom0, md-raid on drives f j g h i k. echo-and-cat does the obvious.

### (I usually run with mq-deadline as scheduler. seems to give
marginally better performance)

# for f in f j g h i k ; do echo-and-cat
/sys/block/sd${f}/queue/scheduler;done
/sys/block/sdf/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdj/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdg/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdh/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdi/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdk/queue/scheduler
[none] mq-deadline kyber bfq
# mount /dev/disk/by-label/SAS-STEAM /mnt/tull
# cd /mnt/tull/tmp
# df -hT ./
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/md2p8     ext4  196G  164G   23G  88% /mnt/tull
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.56845 s, 235 MB/s
0:root@gentoo tmp # dd if=/dev/zero of=a_file bs=1M count=1024
conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.52557 s, 143 MB/s
###

### Booting domu with root and /tmp on SAS-STEAM, ssh into domu

###

# cd /tmp
# df -hT ./
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  163G    24G   88% /
# cat /sys/block/xvdb/
alignment_offset   device/            holders/ power/            
ro                 subsystem/
bdi/               discard_alignment  inflight queue/            
size               trace/
capability         ext_range          integrity/ range             
slaves/            uevent
dev                hidden             mq/ removable          stat
# cat /sys/block/xvdb/queue/scheduler
[none] mq-deadline
# cd /tmp
# df -hT ./
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  163G    24G   88% /
# cat /sys/block/xvdb/
alignment_offset   device/            holders/ power/            
ro                 subsystem/
bdi/               discard_alignment  inflight queue/            
size               trace/
capability         ext_range          integrity/ range             
slaves/            uevent
dev                hidden             mq/ removable          stat
# cat /sys/block/xvdb/queue/scheduler
[none] mq-deadline
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 64,2402 s, 16,7 MB/s
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 59,7517 s, 18,0 MB/s
#





_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Disk performance on guest incredibly low.

Roalt Zijlstra | webpower
Hi,

You are definitely running newer kernels then I do.. But I tested that with a setting in the grub.cfg I can enable the multi-queue block mode by adding 'scsi_mod.use_blk_mq=1'.
I would think that if you use 'scsi_mod.use_blk_mq=0' you can disable the multi-queue block mode.
In general the multi-queue block modes should be better, but it is worth a try in combination with Xen to test the single queue block mode.

Best regards,
 
 Roalt Zijlstra
  Teamleader Infra & Deliverability
   
 [hidden email]
 +31 342 423 262
 roalt.zijlstra
 https://www.webpower-group.com
 
 
Facebook Twitter Linkedin
Barcelona | Barneveld | Beijing | Chengdu | Guangzhou
Hamburg | Shanghai | Shenzhen | Stockholm
 


Op ma 17 dec. 2018 om 19:54 schreef Håkon Alstadheim <[hidden email]>:

Den 17.12.2018 10:32, skrev Roalt Zijlstra | webpower:
> Hey Håkon,
>
> One little tip on disk IO in Virtual hosts, is changing the FS
> scheduler from cfq to noop. In our Xen PV configs we add this to the
> xen.cfg files at the end:
>
> extra="clocksource=tsc elevator=noop"
>
> Especially the "elevator=noop" parameter is forcing all FS-es to the
> noop scheduler. In my experience that gave our Xenservers a pretty
> nice boost.
> Red Hat does recoomend this for all Virtual servers (see the link
> below). To test this you don't need to reboot at all.
>
> For example from the link below there is a code snippet about getting
> and setting the scheduler for /dev/hda. (Replace hda with sdb or any
> other block device)
>
> # cat /sys/block/hda/queue/scheduler
> noop anticipatory deadline [cfq]
>
> # echo 'noop' > /sys/block/hda/queue/scheduler
> # cat /sys/block/hda/queue/scheduler
> [noop] anticipatory deadline cfq
>
> More info: https://access.redhat.com/solutions/5427

Yes, found that resource some time ago. Tried again now, no
break-through. I've got 'none' available as scheduler, rather than noop,
but they should be equal. No matter what I do (in dom0 and/or domu) I
get at least 10 x higher speed in the dom0. :-/ .

Example:

###

###In dom0, md-raid on drives f j g h i k. echo-and-cat does the obvious.

### (I usually run with mq-deadline as scheduler. seems to give
marginally better performance)

# for f in f j g h i k ; do echo-and-cat
/sys/block/sd${f}/queue/scheduler;done
/sys/block/sdf/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdj/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdg/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdh/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdi/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdk/queue/scheduler
[none] mq-deadline kyber bfq
# mount /dev/disk/by-label/SAS-STEAM /mnt/tull
# cd /mnt/tull/tmp
# df -hT ./
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/md2p8     ext4  196G  164G   23G  88% /mnt/tull
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.56845 s, 235 MB/s
0:root@gentoo tmp # dd if=/dev/zero of=a_file bs=1M count=1024
conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.52557 s, 143 MB/s
###

### Booting domu with root and /tmp on SAS-STEAM, ssh into domu

###

# cd /tmp
# df -hT ./
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  163G    24G   88% /
# cat /sys/block/xvdb/
alignment_offset   device/            holders/ power/            
ro                 subsystem/
bdi/               discard_alignment  inflight queue/            
size               trace/
capability         ext_range          integrity/ range             
slaves/            uevent
dev                hidden             mq/ removable          stat
# cat /sys/block/xvdb/queue/scheduler
[none] mq-deadline
# cd /tmp
# df -hT ./
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  163G    24G   88% /
# cat /sys/block/xvdb/
alignment_offset   device/            holders/ power/            
ro                 subsystem/
bdi/               discard_alignment  inflight queue/            
size               trace/
capability         ext_range          integrity/ range             
slaves/            uevent
dev                hidden             mq/ removable          stat
# cat /sys/block/xvdb/queue/scheduler
[none] mq-deadline
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 64,2402 s, 16,7 MB/s
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 59,7517 s, 18,0 MB/s
#





_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Disk performance on guest incredibly low.

jaap-2

Hi,

 

I just run newer kernels because I hoped it would improve the IO performance, which apparently is not the case.

Which kernel versions are you running? Then I can try those, to see if that solves the problem.

 

Best,

 

-- Jaap

 

Van: Xen-users <[hidden email]> Namens Roalt Zijlstra | webpower
Verzonden: Tuesday, 18 December 2018 16:12
Aan: Håkon Alstadheim <[hidden email]>
CC: [hidden email]
Onderwerp: Re: [Xen-users] Disk performance on guest incredibly low.

 

Hi,

 

You are definitely running newer kernels then I do.. But I tested that with a setting in the grub.cfg I can enable the multi-queue block mode by adding 'scsi_mod.use_blk_mq=1'.

I would think that if you use 'scsi_mod.use_blk_mq=0' you can disable the multi-queue block mode.

In general the multi-queue block modes should be better, but it is worth a try in combination with Xen to test the single queue block mode.

 

Best regards,

 

 

 

Roalt Zijlstra

 

 

Teamleader Infra & Deliverability

 

 

 

 

[hidden email]

 

+31 342 423 262

 

roalt.zijlstra

 

https://www.webpower-group.com

 

 

 

Facebook

 

Twitter

 

Linkedin

 

Barcelona | Barneveld | Beijing | Chengdu | Guangzhou
Hamburg | Shanghai | Shenzhen | Stockholm

 

 

 

 

Op ma 17 dec. 2018 om 19:54 schreef Håkon Alstadheim <[hidden email]>:


Den 17.12.2018 10:32, skrev Roalt Zijlstra | webpower:


> Hey Håkon,
>
> One little tip on disk IO in Virtual hosts, is changing the FS
> scheduler from cfq to noop. In our Xen PV configs we add this to the
> xen.cfg files at the end:
>
> extra="clocksource=tsc elevator=noop"
>
> Especially the "elevator=noop" parameter is forcing all FS-es to the
> noop scheduler. In my experience that gave our Xenservers a pretty
> nice boost.
> Red Hat does recoomend this for all Virtual servers (see the link
> below). To test this you don't need to reboot at all.
>
> For example from the link below there is a code snippet about getting
> and setting the scheduler for /dev/hda. (Replace hda with sdb or any
> other block device)
>
> # cat /sys/block/hda/queue/scheduler
> noop anticipatory deadline [cfq]
>
> # echo 'noop' > /sys/block/hda/queue/scheduler
> # cat /sys/block/hda/queue/scheduler
> [noop] anticipatory deadline cfq
>
> More info: https://access.redhat.com/solutions/5427

Yes, found that resource some time ago. Tried again now, no
break-through. I've got 'none' available as scheduler, rather than noop,
but they should be equal. No matter what I do (in dom0 and/or domu) I
get at least 10 x higher speed in the dom0. :-/ .

Example:

###

###In dom0, md-raid on drives f j g h i k. echo-and-cat does the obvious.

### (I usually run with mq-deadline as scheduler. seems to give
marginally better performance)

# for f in f j g h i k ; do echo-and-cat
/sys/block/sd${f}/queue/scheduler;done
/sys/block/sdf/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdj/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdg/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdh/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdi/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdk/queue/scheduler
[none] mq-deadline kyber bfq
# mount /dev/disk/by-label/SAS-STEAM /mnt/tull
# cd /mnt/tull/tmp
# df -hT ./
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/md2p8     ext4  196G  164G   23G  88% /mnt/tull
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.56845 s, 235 MB/s
0:root@gentoo tmp # dd if=/dev/zero of=a_file bs=1M count=1024
conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.52557 s, 143 MB/s
###

### Booting domu with root and /tmp on SAS-STEAM, ssh into domu

###

# cd /tmp
# df -hT ./
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  163G    24G   88% /
# cat /sys/block/xvdb/
alignment_offset   device/            holders/ power/            
ro                 subsystem/
bdi/               discard_alignment  inflight queue/            
size               trace/
capability         ext_range          integrity/ range             
slaves/            uevent
dev                hidden             mq/ removable          stat
# cat /sys/block/xvdb/queue/scheduler
[none] mq-deadline
# cd /tmp
# df -hT ./
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  163G    24G   88% /
# cat /sys/block/xvdb/
alignment_offset   device/            holders/ power/            
ro                 subsystem/
bdi/               discard_alignment  inflight queue/            
size               trace/
capability         ext_range          integrity/ range             
slaves/            uevent
dev                hidden             mq/ removable          stat
# cat /sys/block/xvdb/queue/scheduler
[none] mq-deadline
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 64,2402 s, 16,7 MB/s
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 59,7517 s, 18,0 MB/s
#




_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Disk performance on guest incredibly low.

Roalt Zijlstra | webpower
Hi Jaap,

We mostly use the stock 4.9 kernels for Stretch. We currently do have two new Dell PE R740 servers which require at least 4.11 kernel and they now run on a 4.16 kernel of backports.
But I think you need to disable the Multi queue block mode at boot time of Dom0  and DomU using the kernel boot setting:
scsi_mod.use_blk_mq=0

I am not sure (have not tested anything about that) if multi-queue block mode propagates from Dom0 to DomU automatically.
 
Best regards,
 
 Roalt Zijlstra
  Teamleader Infra & Deliverability
   
 [hidden email]
 +31 342 423 262
 roalt.zijlstra
 https://www.webpower-group.com
 
 
Facebook Twitter Linkedin
Barcelona | Barneveld | Beijing | Chengdu | Guangzhou
Hamburg | Shanghai | Shenzhen | Stockholm
 


Op di 18 dec. 2018 om 16:43 schreef Jaap Gordijn <[hidden email]>:

Hi,

 

I just run newer kernels because I hoped it would improve the IO performance, which apparently is not the case.

Which kernel versions are you running? Then I can try those, to see if that solves the problem.

 

Best,

 

-- Jaap

 

Van: Xen-users <[hidden email]> Namens Roalt Zijlstra | webpower
Verzonden: Tuesday, 18 December 2018 16:12
Aan: Håkon Alstadheim <[hidden email]>
CC: [hidden email]
Onderwerp: Re: [Xen-users] Disk performance on guest incredibly low.

 

Hi,

 

You are definitely running newer kernels then I do.. But I tested that with a setting in the grub.cfg I can enable the multi-queue block mode by adding 'scsi_mod.use_blk_mq=1'.

I would think that if you use 'scsi_mod.use_blk_mq=0' you can disable the multi-queue block mode.

In general the multi-queue block modes should be better, but it is worth a try in combination with Xen to test the single queue block mode.

 

Best regards,

 

 

 

Roalt Zijlstra

 

 

Teamleader Infra & Deliverability

 

 

 

 

[hidden email]

 

+31 342 423 262

 

roalt.zijlstra

 

https://www.webpower-group.com

 

 

 

Facebook

 

Twitter

 

Linkedin

 

Barcelona | Barneveld | Beijing | Chengdu | Guangzhou
Hamburg | Shanghai | Shenzhen | Stockholm

 

 

 

 

Op ma 17 dec. 2018 om 19:54 schreef Håkon Alstadheim <[hidden email]>:


Den 17.12.2018 10:32, skrev Roalt Zijlstra | webpower:


> Hey Håkon,
>
> One little tip on disk IO in Virtual hosts, is changing the FS
> scheduler from cfq to noop. In our Xen PV configs we add this to the
> xen.cfg files at the end:
>
> extra="clocksource=tsc elevator=noop"
>
> Especially the "elevator=noop" parameter is forcing all FS-es to the
> noop scheduler. In my experience that gave our Xenservers a pretty
> nice boost.
> Red Hat does recoomend this for all Virtual servers (see the link
> below). To test this you don't need to reboot at all.
>
> For example from the link below there is a code snippet about getting
> and setting the scheduler for /dev/hda. (Replace hda with sdb or any
> other block device)
>
> # cat /sys/block/hda/queue/scheduler
> noop anticipatory deadline [cfq]
>
> # echo 'noop' > /sys/block/hda/queue/scheduler
> # cat /sys/block/hda/queue/scheduler
> [noop] anticipatory deadline cfq
>
> More info: https://access.redhat.com/solutions/5427

Yes, found that resource some time ago. Tried again now, no
break-through. I've got 'none' available as scheduler, rather than noop,
but they should be equal. No matter what I do (in dom0 and/or domu) I
get at least 10 x higher speed in the dom0. :-/ .

Example:

###

###In dom0, md-raid on drives f j g h i k. echo-and-cat does the obvious.

### (I usually run with mq-deadline as scheduler. seems to give
marginally better performance)

# for f in f j g h i k ; do echo-and-cat
/sys/block/sd${f}/queue/scheduler;done
/sys/block/sdf/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdj/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdg/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdh/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdi/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdk/queue/scheduler
[none] mq-deadline kyber bfq
# mount /dev/disk/by-label/SAS-STEAM /mnt/tull
# cd /mnt/tull/tmp
# df -hT ./
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/md2p8     ext4  196G  164G   23G  88% /mnt/tull
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.56845 s, 235 MB/s
0:root@gentoo tmp # dd if=/dev/zero of=a_file bs=1M count=1024
conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.52557 s, 143 MB/s
###

### Booting domu with root and /tmp on SAS-STEAM, ssh into domu

###

# cd /tmp
# df -hT ./
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  163G    24G   88% /
# cat /sys/block/xvdb/
alignment_offset   device/            holders/ power/            
ro                 subsystem/
bdi/               discard_alignment  inflight queue/            
size               trace/
capability         ext_range          integrity/ range             
slaves/            uevent
dev                hidden             mq/ removable          stat
# cat /sys/block/xvdb/queue/scheduler
[none] mq-deadline
# cd /tmp
# df -hT ./
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  163G    24G   88% /
# cat /sys/block/xvdb/
alignment_offset   device/            holders/ power/            
ro                 subsystem/
bdi/               discard_alignment  inflight queue/            
size               trace/
capability         ext_range          integrity/ range             
slaves/            uevent
dev                hidden             mq/ removable          stat
# cat /sys/block/xvdb/queue/scheduler
[none] mq-deadline
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 64,2402 s, 16,7 MB/s
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 59,7517 s, 18,0 MB/s
#




_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Disk performance on guest incredibly low.

jaap-2
In reply to this post by Roalt Zijlstra | webpower

I used another disk benchmark tests:

dd if=/dev/zero of=a_file bs=1G count=1 oflag=dsync
dd if=/dev/zero of=a_file bs=512 count=1000 oflag=dsync

 

Then the results between dom0 and domU are much closer related.

 

Van: Xen-users <[hidden email]> Namens Roalt Zijlstra | webpower
Verzonden: Tuesday, 18 December 2018 16:12
Aan: Håkon Alstadheim <[hidden email]>
CC: [hidden email]
Onderwerp: Re: [Xen-users] Disk performance on guest incredibly low.

 

Hi,

 

You are definitely running newer kernels then I do.. But I tested that with a setting in the grub.cfg I can enable the multi-queue block mode by adding 'scsi_mod.use_blk_mq=1'.

I would think that if you use 'scsi_mod.use_blk_mq=0' you can disable the multi-queue block mode.

In general the multi-queue block modes should be better, but it is worth a try in combination with Xen to test the single queue block mode.

 

Best regards,

 

 

 

Roalt Zijlstra

 

 

Teamleader Infra & Deliverability

 

 

 

 

[hidden email]

 

+31 342 423 262

 

roalt.zijlstra

 

https://www.webpower-group.com

 

 

 

Facebook

 

Twitter

 

Linkedin

 

Barcelona | Barneveld | Beijing | Chengdu | Guangzhou
Hamburg | Shanghai | Shenzhen | Stockholm

 

 

 

 

Op ma 17 dec. 2018 om 19:54 schreef Håkon Alstadheim <[hidden email]>:


Den 17.12.2018 10:32, skrev Roalt Zijlstra | webpower:


> Hey Håkon,
>
> One little tip on disk IO in Virtual hosts, is changing the FS
> scheduler from cfq to noop. In our Xen PV configs we add this to the
> xen.cfg files at the end:
>
> extra="clocksource=tsc elevator=noop"
>
> Especially the "elevator=noop" parameter is forcing all FS-es to the
> noop scheduler. In my experience that gave our Xenservers a pretty
> nice boost.
> Red Hat does recoomend this for all Virtual servers (see the link
> below). To test this you don't need to reboot at all.
>
> For example from the link below there is a code snippet about getting
> and setting the scheduler for /dev/hda. (Replace hda with sdb or any
> other block device)
>
> # cat /sys/block/hda/queue/scheduler
> noop anticipatory deadline [cfq]
>
> # echo 'noop' > /sys/block/hda/queue/scheduler
> # cat /sys/block/hda/queue/scheduler
> [noop] anticipatory deadline cfq
>
> More info: https://access.redhat.com/solutions/5427

Yes, found that resource some time ago. Tried again now, no
break-through. I've got 'none' available as scheduler, rather than noop,
but they should be equal. No matter what I do (in dom0 and/or domu) I
get at least 10 x higher speed in the dom0. :-/ .

Example:

###

###In dom0, md-raid on drives f j g h i k. echo-and-cat does the obvious.

### (I usually run with mq-deadline as scheduler. seems to give
marginally better performance)

# for f in f j g h i k ; do echo-and-cat
/sys/block/sd${f}/queue/scheduler;done
/sys/block/sdf/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdj/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdg/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdh/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdi/queue/scheduler
[none] mq-deadline kyber bfq
/sys/block/sdk/queue/scheduler
[none] mq-deadline kyber bfq
# mount /dev/disk/by-label/SAS-STEAM /mnt/tull
# cd /mnt/tull/tmp
# df -hT ./
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/md2p8     ext4  196G  164G   23G  88% /mnt/tull
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.56845 s, 235 MB/s
0:root@gentoo tmp # dd if=/dev/zero of=a_file bs=1M count=1024
conv=fsync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.52557 s, 143 MB/s
###

### Booting domu with root and /tmp on SAS-STEAM, ssh into domu

###

# cd /tmp
# df -hT ./
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  163G    24G   88% /
# cat /sys/block/xvdb/
alignment_offset   device/            holders/ power/            
ro                 subsystem/
bdi/               discard_alignment  inflight queue/            
size               trace/
capability         ext_range          integrity/ range             
slaves/            uevent
dev                hidden             mq/ removable          stat
# cat /sys/block/xvdb/queue/scheduler
[none] mq-deadline
# cd /tmp
# df -hT ./
Filsystem      Type Størrelse Brukt Tilgj. Bruk% Montert på
/dev/xvdb      ext4      196G  163G    24G   88% /
# cat /sys/block/xvdb/
alignment_offset   device/            holders/ power/            
ro                 subsystem/
bdi/               discard_alignment  inflight queue/            
size               trace/
capability         ext_range          integrity/ range             
slaves/            uevent
dev                hidden             mq/ removable          stat
# cat /sys/block/xvdb/queue/scheduler
[none] mq-deadline
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 64,2402 s, 16,7 MB/s
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync oflag=direct
1024+0 oppføringer inn
1024+0 oppføringer ut
1073741824 byte (1,1 GB, 1,0 GiB) kopiert, 59,7517 s, 18,0 MB/s
#




_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users