File-based domU - Slow storage write since xen 4.8

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

File-based domU - Slow storage write since xen 4.8

Benoit Depail
Hi,

At my company, we are using file-based domU for our hosting infrastructures.

We have no issues on Xen 4.4 (Debian Jessie). However we are concerned
by this aging platform and are testing Xen 4.8 (Debian Stretch).

The problem is that our storage writing speed are about the third of
what they used to be.

On xen 4.4, the domU and the dom0 can easily reach a steady 100MB/s when
writing to the image file.

On xen 4.8, the domU is stuck at about 30MB/s whereas the dom0 can still
reach about 100MB/s.

The domU is the same in both case, I just switch the dom0 OS.

I doubt that the cache have anything to do with it since the image file
is hosted on a NFS share, and I can see the network traffic reach
800Mb/s when everything is fine.

I could also reproduce this behaviour on a stock alpine linux based
dom0, running xen 4.8. So it seems our debian installation is not in cause.

There are no changes in reading speed.


What could explain this behaviour ?

--
Benoit Depail
Senior Infrastructures Architect
NBS System



_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users

signature.asc (1023 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

WebDawg


On Thu, Jul 13, 2017 at 12:42 PM, Benoit Depail <[hidden email]> wrote:
Hi,

At my company, we are using file-based domU for our hosting infrastructures.

We have no issues on Xen 4.4 (Debian Jessie). However we are concerned
by this aging platform and are testing Xen 4.8 (Debian Stretch).

The problem is that our storage writing speed are about the third of
what they used to be.

On xen 4.4, the domU and the dom0 can easily reach a steady 100MB/s when
writing to the image file.

On xen 4.8, the domU is stuck at about 30MB/s whereas the dom0 can still
reach about 100MB/s.

The domU is the same in both case, I just switch the dom0 OS.

I doubt that the cache have anything to do with it since the image file
is hosted on a NFS share, and I can see the network traffic reach
800Mb/s when everything is fine.

I could also reproduce this behaviour on a stock alpine linux based
dom0, running xen 4.8. So it seems our debian installation is not in cause.

There are no changes in reading speed.


What could explain this behaviour ?

--
Benoit Depail
Senior Infrastructures Architect
NBS System





Now this is very interesting.   Is it the same hardware, switches, nics and all?

I might start with looking at NFS.  The new kernel might require some different settings to match what the old did automatically.  I do not know what has changed in NFS.

You are saying you tested NFS speed on a dom0?  I know dom0 is more like a domU then most think but what tests do you get out of that?

I would start with direct NFS speeds and move forward through the storage stack.  I would do an iperf test too.


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Benoit Depail
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi, thanks for your answer.

On 07/13/17 19:20, WebDawg wrote:
> Now this is very interesting.   Is it the same hardware, switches,
> nics and all?

It is the same physical machine, I just switched the OS version.

> I might start with looking at NFS.  The new kernel might require
> some different settings to match what the old did automatically.  I
> do not know what has changed in NFS.
>
> You are saying you tested NFS speed on a dom0?  I know dom0 is more
> like a domU then most think but what tests do you get out of that?

I realized I did not explain our setup in detail.

This is the domU configuration (Using pv) :

kernel      = '/data/boot/linux-4.9.0/vmlinuz-4.9.0-0.bpo.2-amd64'
ramdisk     = '/data/boot/linux-4.9.0/initrd.img-4.9.0-0.bpo.2-amd64'
vcpus       = '1'
memory      = '1000'
root    = '/dev/xvda'
disk        = [
                'file:/media/datastores/1/d-anb-nab2_root.img,xvda,w',
                'file:/media/datastores/1/d-anb-nab2_data.img,xvdb,w',
                ]
name        = 'd-anb-nab2'
vif         = [
                'ip=10.0.2.1,mac=00:16:3E:00:02:01, bridge=br2,
vifname=d-anb-nab2.e0',
                'ip=10.0.1.1,mac=00:16:3E:00:01:01, bridge=br3,
vifname=d-anb-nab2.e1'
                ]
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

/media/datastores/1/ is a NFS share from a NetApp appliance, mounted
on the dom0. So our domU do not use NFS directly, only the dom0 will
use NFS.

As per this configuration, the raw image file is setup as a loop
device before being exposed as /dev/xvd* to the DomU.


When the server is running Debian Jessie and Xen4.4 (the domU is
always the same, running Debian Jessie and backported kernel 4.9.0) :

dom0# xl info
host                   : ikea
release                : 3.16.0-4-amd64
version                : #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)
machine                : x86_64
nr_cpus                : 24
max_cpu_id             : 23
nr_nodes               : 4
cores_per_socket       : 12
threads_per_core       : 1
cpu_mhz                : 2200
hw_caps                :
178bf3ff:efd3fbff:00000000:00001300:00802001:00000000:000837ff:00000000
virt_caps              : hvm
total_memory           : 49147
free_memory            : 44580
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 4
xen_extra              : .1
xen_version            : 4.4.1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : dom0_mem=2G,max:2G dom0_max_vcpus=2
dom0_vcpus_pin
cc_compiler            : gcc (Debian 4.9.2-10) 4.9.2
cc_compile_by          : carnil
cc_compile_domain      : debian.org
cc_compile_date        : Wed Dec  7 06:03:38 UTC 2016
xend_config_format     : 4

dom0# dd if=/dev/zero of=/media/datastores/1/d-anb-nab2_data.img bs=4M
count=$((5*1024/4))
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 48.6082 s, 110 MB/s

domU# dd if=/dev/zero of=/dev/xvdb bs=4M count=$((5*1024/4))
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 47.7129 s, 113 MB/s

When I reboot the server under Debian Stretch and xen4.8

dom0# xl info
host                   : ikea
release                : 4.9.0-3-amd64
version                : #1 SMP Debian 4.9.30-2 (2017-06-12)
machine                : x86_64
nr_cpus                : 24
max_cpu_id             : 23
nr_nodes               : 4
cores_per_socket       : 12
threads_per_core       : 1
cpu_mhz                : 2200
hw_caps                :
178bf3ff:80802001:efd3fbff:000837ff:00000000:00000000:00000000:00000100
virt_caps              : hvm
total_memory           : 49147
free_memory            : 45574
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 8
xen_extra              : .1
xen_version            : 4.8.1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : dom0_mem=2G,max:2G dom0_max_vcpus=2
dom0_vcpus_pin
cc_compiler            : gcc (Debian 6.3.0-16) 6.3.0 20170425
cc_compile_by          : ian.jackson
cc_compile_domain      : eu.citrix.com
cc_compile_date        : Tue May  2 14:06:04 UTC 2017
build_id               : 0b619fa14fca0e6ca76f2a8b52eba64d60aa37de
xend_config_format     : 4

dom0# dd if=/dev/zero of=/media/datastores/1/d-anb-nab2_data.img bs=4M
count=$((5*1024/4))
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 48.6297 s, 110 MB/s

domU# dd if=/dev/zero of=/dev/xvdb bs=4M count=$((5*1024/4))
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 170.778 s, 31.4 MB/s


However, when the domU use a physical device, such as a LVM logical
volume, there are no issues and the results are consistent between
xen4.4 and 4.8.


So the million dollar question is : What happens (most probably in
blkback/blkfront afaict) when the backend is a loop device, that
doesn't happen when it's a LVM logical volume ?



- --
Benoit Depail
Senior Infrastructures Architect
NBS System
+33 158 565 611
-----BEGIN PGP SIGNATURE-----

iQKxBAEBCgCbFiEEUVI4Zy9ojyLG8rQSok/WASnIOwQFAllsfytfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDUx
NTIzODY3MkY2ODhGMjJDNkYyQjQxMkEyNEZENjAxMjlDODNCMDQdHGJlbm9pdC5k
ZXBhaWxAbmJzLXN5c3RlbS5jb20ACgkQok/WASnIOwTjXRAAuieF4wLKoPEilADD
opOK68GjnSCUp2LuovrtFebfw7Wps016g+eMsNy20oOLCxjSzRVFDNqhkoCc4mVB
iLm55byV4LchafbLBHTshkYDwyuzqFP6cl/9uR1n/mzaH6EVhTiGvwOwpJosH5IM
WGRRmXNdC4IdKGrAFfoBgC+8Ymnsr2vf85BIR6gc5ValiaDLmWPQyYZafxSVjV64
caQmnC0sK0cgKCutLWRia/xO1/2ZsrTJq3T1w8enh1SehKBWlzX0bJWW6MabNUni
ovTgC3TyqNSu3OVPdSs6zo7F2YtXK4dXZ6v6HYht85WY8hb0vwYkorLR/b5WSiR2
MFyH11KOgHN0vWA7ZsH1wMh/TtwgsRK78pW4DGVRwo23ySeVM8oV++x+96q53pdC
wq63nJ0BAX04Kh4gayDkwBlflOxJUxaLB9RouqD8EWHLjt/BQ1jLKN1LgigVhV7e
grXpbn0B8u4UHhYh9hVdp1YgmGeko117ZTx1CAtbxL9Vgng0uE70SAwRDxkxinOZ
VXv9tIRs1IB8aDFSNWj7YuERe3gSq6JXUJL9mlAimkrI8hQzHA8iV1QohyP0K33u
dFMJe2zcJFXjVq3+z/1mKCdYdipJdd4ew54oLiPOSzSIzU13gWeiRdK/rWIWwqRE
srIh8U6CBzB5q2KpE2A3poejvco=
=2pNr
-----END PGP SIGNATURE-----

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Benoit Depail
I forgot to include some tests results for when I test the loop device
on the dom0:

Xen4.4:

dom0# dd if=/dev/zero of=/dev/loop1 bs=4M count=$((5*1024/4))
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 48.574 s, 111 MB/s


Xen4.8:

dom0# dd if=/dev/zero of=/dev/loop1 bs=4M count=$((5*1024/4))
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 45.589 s, 118 MB/s

On 07/17/17 11:11, Benoit Depail wrote:

> Hi, thanks for your answer.
>
> On 07/13/17 19:20, WebDawg wrote:
>> Now this is very interesting.   Is it the same hardware, switches,
>> nics and all?
>
> It is the same physical machine, I just switched the OS version.
>
>> I might start with looking at NFS.  The new kernel might require
>> some different settings to match what the old did automatically.  I
>> do not know what has changed in NFS.
>
>> You are saying you tested NFS speed on a dom0?  I know dom0 is more
>> like a domU then most think but what tests do you get out of that?
>
> I realized I did not explain our setup in detail.
>
> This is the domU configuration (Using pv) :
>
> kernel      = '/data/boot/linux-4.9.0/vmlinuz-4.9.0-0.bpo.2-amd64'
> ramdisk     = '/data/boot/linux-4.9.0/initrd.img-4.9.0-0.bpo.2-amd64'
> vcpus       = '1'
> memory      = '1000'
> root    = '/dev/xvda'
> disk        = [
>                 'file:/media/datastores/1/d-anb-nab2_root.img,xvda,w',
>                 'file:/media/datastores/1/d-anb-nab2_data.img,xvdb,w',
>                 ]
> name        = 'd-anb-nab2'
> vif         = [
>                 'ip=10.0.2.1,mac=00:16:3E:00:02:01, bridge=br2,
> vifname=d-anb-nab2.e0',
>                 'ip=10.0.1.1,mac=00:16:3E:00:01:01, bridge=br3,
> vifname=d-anb-nab2.e1'
>                 ]
> on_poweroff = 'destroy'
> on_reboot   = 'restart'
> on_crash    = 'restart'
>
> /media/datastores/1/ is a NFS share from a NetApp appliance, mounted
> on the dom0. So our domU do not use NFS directly, only the dom0 will
> use NFS.
>
> As per this configuration, the raw image file is setup as a loop
> device before being exposed as /dev/xvd* to the DomU.
>
>
> When the server is running Debian Jessie and Xen4.4 (the domU is
> always the same, running Debian Jessie and backported kernel 4.9.0) :
>
> dom0# xl info
> host                   : ikea
> release                : 3.16.0-4-amd64
> version                : #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)
> machine                : x86_64
> nr_cpus                : 24
> max_cpu_id             : 23
> nr_nodes               : 4
> cores_per_socket       : 12
> threads_per_core       : 1
> cpu_mhz                : 2200
> hw_caps                :
> 178bf3ff:efd3fbff:00000000:00001300:00802001:00000000:000837ff:00000000
> virt_caps              : hvm
> total_memory           : 49147
> free_memory            : 44580
> sharing_freed_memory   : 0
> sharing_used_memory    : 0
> outstanding_claims     : 0
> free_cpus              : 0
> xen_major              : 4
> xen_minor              : 4
> xen_extra              : .1
> xen_version            : 4.4.1
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          :
> xen_commandline        : dom0_mem=2G,max:2G dom0_max_vcpus=2
> dom0_vcpus_pin
> cc_compiler            : gcc (Debian 4.9.2-10) 4.9.2
> cc_compile_by          : carnil
> cc_compile_domain      : debian.org
> cc_compile_date        : Wed Dec  7 06:03:38 UTC 2016
> xend_config_format     : 4
>
> dom0# dd if=/dev/zero of=/media/datastores/1/d-anb-nab2_data.img bs=4M
> count=$((5*1024/4))
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 48.6082 s, 110 MB/s
>
> domU# dd if=/dev/zero of=/dev/xvdb bs=4M count=$((5*1024/4))
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 47.7129 s, 113 MB/s
>
> When I reboot the server under Debian Stretch and xen4.8
>
> dom0# xl info
> host                   : ikea
> release                : 4.9.0-3-amd64
> version                : #1 SMP Debian 4.9.30-2 (2017-06-12)
> machine                : x86_64
> nr_cpus                : 24
> max_cpu_id             : 23
> nr_nodes               : 4
> cores_per_socket       : 12
> threads_per_core       : 1
> cpu_mhz                : 2200
> hw_caps                :
> 178bf3ff:80802001:efd3fbff:000837ff:00000000:00000000:00000000:00000100
> virt_caps              : hvm
> total_memory           : 49147
> free_memory            : 45574
> sharing_freed_memory   : 0
> sharing_used_memory    : 0
> outstanding_claims     : 0
> free_cpus              : 0
> xen_major              : 4
> xen_minor              : 8
> xen_extra              : .1
> xen_version            : 4.8.1
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          :
> xen_commandline        : dom0_mem=2G,max:2G dom0_max_vcpus=2
> dom0_vcpus_pin
> cc_compiler            : gcc (Debian 6.3.0-16) 6.3.0 20170425
> cc_compile_by          : ian.jackson
> cc_compile_domain      : eu.citrix.com
> cc_compile_date        : Tue May  2 14:06:04 UTC 2017
> build_id               : 0b619fa14fca0e6ca76f2a8b52eba64d60aa37de
> xend_config_format     : 4
>
> dom0# dd if=/dev/zero of=/media/datastores/1/d-anb-nab2_data.img bs=4M
> count=$((5*1024/4))
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 48.6297 s, 110 MB/s
>
> domU# dd if=/dev/zero of=/dev/xvdb bs=4M count=$((5*1024/4))
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 170.778 s, 31.4 MB/s
>
>
> However, when the domU use a physical device, such as a LVM logical
> volume, there are no issues and the results are consistent between
> xen4.4 and 4.8.
>
>
> So the million dollar question is : What happens (most probably in
> blkback/blkfront afaict) when the backend is a loop device, that
> doesn't happen when it's a LVM logical volume ?
>
>
>
>
> _______________________________________________
> Xen-users mailing list
> [hidden email]
> https://lists.xen.org/xen-users
>
--
Benoit Depail
Senior Infrastructures Architect
NBS System
+33 158 565 611


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users

signature.asc (1023 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Roger Pau Monné-3
In reply to this post by Benoit Depail
On Mon, Jul 17, 2017 at 11:11:20AM +0200, Benoit Depail wrote:
> When the server is running Debian Jessie and Xen4.4 (the domU is
> always the same, running Debian Jessie and backported kernel 4.9.0) :
>
> dom0# xl info
> host                   : ikea
> release                : 3.16.0-4-amd64

So in this case Dom0 is using kernel 3.16, while ...

[...]
> When I reboot the server under Debian Stretch and xen4.8
>
> dom0# xl info
> host                   : ikea
> release                : 4.9.0-3-amd64

... here you are using kernel 4.9.

In order to try to figure out the problem, can you try Xen 4.9 with
Linux kernel 3.16?

Thanks, Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Benoit Depail
On 07/17/17 14:49, Roger Pau Monné wrote:
> On Mon, Jul 17, 2017 at 11:11:20AM +0200, Benoit Depail wrote:
> In order to try to figure out the problem, can you try Xen 4.9 with
> Linux kernel 3.16?
>

Good call !

So, with this configuration :

dom0# xl info
host                   : ikea
release                : 3.16.0-4-amd64
version                : #1 SMP Debian 3.16.43-2 (2017-04-30)
machine                : x86_64
nr_cpus                : 24
max_cpu_id             : 23
nr_nodes               : 4
cores_per_socket       : 12
threads_per_core       : 1
cpu_mhz                : 2200
hw_caps                :
178bf3ff:80802001:efd3fbff:000837ff:00000000:00000000:00000000:00000100
virt_caps              : hvm
total_memory           : 49147
free_memory            : 45572
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 8
xen_extra              : .1
xen_version            : 4.8.1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : dom0_mem=2G,max:2G dom0_max_vcpus=2 dom0_vcpus_pin
cc_compiler            : gcc (Debian 6.3.0-16) 6.3.0 20170425
cc_compile_by          : ian.jackson
cc_compile_domain      : eu.citrix.com
cc_compile_date        : Tue May  2 14:06:04 UTC 2017
build_id               : 0b619fa14fca0e6ca76f2a8b52eba64d60aa37de
xend_config_format     : 4

The writing performances are the same in the dom0 and in the domU.

In the mean time I found that the problem is not NFS specific so I'll
stick to raw image files on the local disks, to keep things simple.

I'll see if I can roughly pinpoint the kernel version that trigger this
behaviour.

Thanks _a lot_ for your help. This is a good enough work around for us,
so we can start extensive testing to get xen 4.8 in production.

Best regards,

--
Benoit Depail
Senior Infrastructures Architect
NBS System


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users

signature.asc (1023 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Roger Pau Monné-3
On Mon, Jul 17, 2017 at 05:40:55PM +0200, Benoit Depail wrote:

> On 07/17/17 14:49, Roger Pau Monné wrote:
> > On Mon, Jul 17, 2017 at 11:11:20AM +0200, Benoit Depail wrote:
> > In order to try to figure out the problem, can you try Xen 4.9 with
> > Linux kernel 3.16?
> >
>
> Good call !
>
> So, with this configuration :
>
> dom0# xl info
> host                   : ikea
> release                : 3.16.0-4-amd64
> version                : #1 SMP Debian 3.16.43-2 (2017-04-30)
> machine                : x86_64
> nr_cpus                : 24
> max_cpu_id             : 23
> nr_nodes               : 4
> cores_per_socket       : 12
> threads_per_core       : 1
> cpu_mhz                : 2200
> hw_caps                :
> 178bf3ff:80802001:efd3fbff:000837ff:00000000:00000000:00000000:00000100
> virt_caps              : hvm
> total_memory           : 49147
> free_memory            : 45572
> sharing_freed_memory   : 0
> sharing_used_memory    : 0
> outstanding_claims     : 0
> free_cpus              : 0
> xen_major              : 4
> xen_minor              : 8
> xen_extra              : .1
> xen_version            : 4.8.1
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          :
> xen_commandline        : dom0_mem=2G,max:2G dom0_max_vcpus=2 dom0_vcpus_pin
> cc_compiler            : gcc (Debian 6.3.0-16) 6.3.0 20170425
> cc_compile_by          : ian.jackson
> cc_compile_domain      : eu.citrix.com
> cc_compile_date        : Tue May  2 14:06:04 UTC 2017
> build_id               : 0b619fa14fca0e6ca76f2a8b52eba64d60aa37de
> xend_config_format     : 4
>
> The writing performances are the same in the dom0 and in the domU.
>
> In the mean time I found that the problem is not NFS specific so I'll
> stick to raw image files on the local disks, to keep things simple.
>
> I'll see if I can roughly pinpoint the kernel version that trigger this
> behaviour.
>
> Thanks _a lot_ for your help. This is a good enough work around for us,
> so we can start extensive testing to get xen 4.8 in production.

There are not a huge amount of changes to blkback, so it would be
helpful if you could pinpoint the change that introduced this
regression for you, or if you can get a kernel version that introduced
the regression that would also be helpful.

Thanks, Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Benoit Depail
Hello,

On 07/17/17 18:46, Roger Pau Monné wrote:
> There are not a huge amount of changes to blkback, so it would be
> helpful if you could pinpoint the change that introduced this
> regression for you, or if you can get a kernel version that introduced
> the regression that would also be helpful.

I used git bisect on the linux-stable source tree to build (a lot of)
tests kernels, and was able to find this commit as the first one
introducing the regression :

d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
commit d2081cfe624b5decaaf68088ca256ed1b140672c
Author: Keith Busch <[hidden email]>
Date:   Tue Jan 12 15:08:39 2016 -0700

    block: split bios to max possible length

In term of kernel version, the first one showing bad performances in my
case is 4.4.2 (with, obviously, 4.4.1 working as expected).

Interestingly, this commit is an improvement of
d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
but do not show any performance issue in our case.

I can also confirm that this issue is still present in the latest
unstable kernel (we tested 4.13-rc1).

I think I have reached a point where more competent people should
investigate, but I'll be glad to help if needed.


Best regards,

--
Benoit Depail
Senior Infrastructures Architect
NBS System


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users

signature.asc (1023 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Roger Pau Monné-3
I'm afraid I'm also not that knowledgeable, so I'm adding the author
and the Linux block ml.

On Wed, Jul 19, 2017 at 04:25:02PM +0200, Benoit Depail wrote:

> Hello,
>
> On 07/17/17 18:46, Roger Pau Monné wrote:
> > There are not a huge amount of changes to blkback, so it would be
> > helpful if you could pinpoint the change that introduced this
> > regression for you, or if you can get a kernel version that introduced
> > the regression that would also be helpful.
>
> I used git bisect on the linux-stable source tree to build (a lot of)
> tests kernels, and was able to find this commit as the first one
> introducing the regression :
>
> d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
> commit d2081cfe624b5decaaf68088ca256ed1b140672c
> Author: Keith Busch <[hidden email]>
> Date:   Tue Jan 12 15:08:39 2016 -0700
>
>     block: split bios to max possible length
>
> In term of kernel version, the first one showing bad performances in my
> case is 4.4.2 (with, obviously, 4.4.1 working as expected).
>
> Interestingly, this commit is an improvement of
> d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
> but do not show any performance issue in our case.
>
> I can also confirm that this issue is still present in the latest
> unstable kernel (we tested 4.13-rc1).
>
> I think I have reached a point where more competent people should
> investigate, but I'll be glad to help if needed.

For the sake of the people that I've added, would you mind describing
the issues you are seeing and the tests that you performed?

Thanks, Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Benoit Depail
On 07/20/17 10:52, Roger Pau Monné wrote:
> For the sake of the people that I've added, would you mind describing
> the issues you are seeing and the tests that you performed?
>
> Thanks, Roger.
>

Sure.

The main issue we are seeing is degraded write performance on storage
devices of Xen PV DomUs, about half (or even a third on our production
setup where NFS is involved) of what we used to have.

Our test setup is as follow (we left the NFS out as it has nothing to do
with our problem):

On a physical server running debian stretch as a Xen 4.8 hypervisor, we
create a PV domU, also running debian stretch. The storage of this domU
resides on the server's local disk as raw image files, which will be
setup as loop-devices before being exposed to the domU as /dev/xvd{a,b}.

the domU configuration (relevant part):

disk        = [
   'file:/mnt/domu_root.img,xvda,w',
   'file:/mnt/d-anb-nab2.img,xvdb,w'
]


When the domU is running, a loop-device is created:

/dev/loop1: [65027]:12 (/mnt/d-anb-nab2.img)

We perform a simple dd to test the sequential writing speed:

##### Dom0 with a kernel < 4.4.2 (here, 4.1.42, we had similar results
with 3.16.0)

# dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 47.5052 s, 113 MB/s

# dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 46.8227 s, 115 MB/s

On the domU:
# dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 46.7245 s, 115 MB/s


##### Dom0 with a kernel >= 4.4.2, or a custom 4.4.1 including commit
d2081cfe624b5decaaf68088ca256ed1b140672c

On the dom0:
# dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 46.3234 s, 116 MB/s

# dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 44.948 s, 119 MB/s

On the domU:
# dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 102.943 s, 52.2 MB/s


For completeness sake, I'll put my findings below:

> I used git bisect on the linux-stable source tree to build (a lot of)
> tests kernels, and was able to find this commit as the first one
> introducing the regression :
>
> d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
> commit d2081cfe624b5decaaf68088ca256ed1b140672c
> Author: Keith Busch <[hidden email]>
> Date:   Tue Jan 12 15:08:39 2016 -0700
>
>     block: split bios to max possible length
>
> In term of kernel version, the first one showing bad performances in my
> case is 4.4.2 (with, obviously, 4.4.1 working as expected).
>
> Interestingly, this commit is an improvement of
> d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
> but do not show any performance issue in our case.
>
> I can also confirm that this issue is still present in the latest
> unstable kernel (we tested 4.13-rc1).

Thanks,

--
Benoit Depail
Senior Infrastructures Architect
NBS System


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users

signature.asc (1023 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Keith Busch
On Thu, Jul 20, 2017 at 05:12:33PM +0200, Benoit Depail wrote:
>
> The main issue we are seeing is degraded write performance on storage
> devices of Xen PV DomUs, about half (or even a third on our production
> setup where NFS is involved) of what we used to have.

Read performance is unchanged?

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Benoit Depail
On 07/20/17 17:33, Keith Busch wrote:
> On Thu, Jul 20, 2017 at 05:12:33PM +0200, Benoit Depail wrote:
>>
>> The main issue we are seeing is degraded write performance on storage
>> devices of Xen PV DomUs, about half (or even a third on our production
>> setup where NFS is involved) of what we used to have.
>
> Read performance is unchanged?
>

Read performance is unaffected indeed.

--
Benoit Depail
Senior Infrastructures Architect
NBS System


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users

signature.asc (1023 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Keith Busch
In reply to this post by Benoit Depail
On Thu, Jul 20, 2017 at 05:12:33PM +0200, Benoit Depail wrote:

> ##### Dom0 with a kernel >= 4.4.2, or a custom 4.4.1 including commit
> d2081cfe624b5decaaf68088ca256ed1b140672c
>
> On the dom0:
> # dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 46.3234 s, 116 MB/s
>
> # dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 44.948 s, 119 MB/s
>
> On the domU:
> # dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 102.943 s, 52.2 MB/s
>
>
> For completeness sake, I'll put my findings below:
>
> > I used git bisect on the linux-stable source tree to build (a lot of)
> > tests kernels, and was able to find this commit as the first one
> > introducing the regression :
> >
> > d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
> > commit d2081cfe624b5decaaf68088ca256ed1b140672c
> > Author: Keith Busch <[hidden email]>
> > Date:   Tue Jan 12 15:08:39 2016 -0700
> >
> >     block: split bios to max possible length
> >
> > In term of kernel version, the first one showing bad performances in my
> > case is 4.4.2 (with, obviously, 4.4.1 working as expected).
> >
> > Interestingly, this commit is an improvement of
> > d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
> > but do not show any performance issue in our case.
> >
> > I can also confirm that this issue is still present in the latest
> > unstable kernel (we tested 4.13-rc1).

I admit I don't have a good working knowledge of xen block. I've spent a
few minutes looking over the code, and the best I can tell is that this
patch will build requests up to 88 blocks ((11 segments * 4k) / 512)
that would have been fewer blocks without this patch. The resulting bio
vectors may also have offsets that weren't there before.

I'm not sure what the significance is. Perhaps it's causing the number
of grants to exceed BLKIF_MAX_SEGMENTS_PER_REQUEST, which appears to take
a less optimal path in the code, but I'm not sure why that would only
harm writes.

As a test, could you throttle the xvdb queue's max_sectors_kb? If I
followed xen-blkfront correctly, the default should have it set to 44.
Try setting it to 40.

  echo 40 > /sys/block/xvdb/queue/max_sectors_kb


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Benoit Depail
On 07/20/17 19:36, Keith Busch wrote:

> On Thu, Jul 20, 2017 at 05:12:33PM +0200, Benoit Depail wrote:
>> ##### Dom0 with a kernel >= 4.4.2, or a custom 4.4.1 including commit
>> d2081cfe624b5decaaf68088ca256ed1b140672c
>>
>> On the dom0:
>> # dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
>> 1280+0 records in
>> 1280+0 records out
>> 5368709120 bytes (5.4 GB) copied, 46.3234 s, 116 MB/s
>>
>> # dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
>> 1280+0 records in
>> 1280+0 records out
>> 5368709120 bytes (5.4 GB) copied, 44.948 s, 119 MB/s
>>
>> On the domU:
>> # dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
>> 1280+0 records in
>> 1280+0 records out
>> 5368709120 bytes (5.4 GB) copied, 102.943 s, 52.2 MB/s
>>
>>
>> For completeness sake, I'll put my findings below:
>>
>>> I used git bisect on the linux-stable source tree to build (a lot of)
>>> tests kernels, and was able to find this commit as the first one
>>> introducing the regression :
>>>
>>> d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
>>> commit d2081cfe624b5decaaf68088ca256ed1b140672c
>>> Author: Keith Busch <[hidden email]>
>>> Date:   Tue Jan 12 15:08:39 2016 -0700
>>>
>>>     block: split bios to max possible length
>>>
>>> In term of kernel version, the first one showing bad performances in my
>>> case is 4.4.2 (with, obviously, 4.4.1 working as expected).
>>>
>>> Interestingly, this commit is an improvement of
>>> d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
>>> but do not show any performance issue in our case.
>>>
>>> I can also confirm that this issue is still present in the latest
>>> unstable kernel (we tested 4.13-rc1).
>
> I admit I don't have a good working knowledge of xen block. I've spent a
> few minutes looking over the code, and the best I can tell is that this
> patch will build requests up to 88 blocks ((11 segments * 4k) / 512)
> that would have been fewer blocks without this patch. The resulting bio
> vectors may also have offsets that weren't there before.
>
> I'm not sure what the significance is. Perhaps it's causing the number
> of grants to exceed BLKIF_MAX_SEGMENTS_PER_REQUEST, which appears to take
> a less optimal path in the code, but I'm not sure why that would only
> harm writes.
>
> As a test, could you throttle the xvdb queue's max_sectors_kb? If I
> followed xen-blkfront correctly, the default should have it set to 44.
> Try setting it to 40.
>
>   echo 40 > /sys/block/xvdb/queue/max_sectors_kb
>

The default value on my domU is 128.

I ran a couple of tests with different values, starting from 40 and up
to 128, clearing the cache between each tests.

The only value that showed the issue is 128. Even setting max_sectors_kb
to 127 is enough to get normal behaviour.

Best regards,

--
Benoit Depail
Senior Infrastructures Architect
NBS System

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Keith Busch
On Fri, Jul 21, 2017 at 12:19:39PM +0200, Benoit Depail wrote:

> On 07/20/17 19:36, Keith Busch wrote:
> >
> > As a test, could you throttle the xvdb queue's max_sectors_kb? If I
> > followed xen-blkfront correctly, the default should have it set to 44.
> > Try setting it to 40.
> >
> >   echo 40 > /sys/block/xvdb/queue/max_sectors_kb
> >
>
> The default value on my domU is 128.
>
> I ran a couple of tests with different values, starting from 40 and up
> to 128, clearing the cache between each tests.
>
> The only value that showed the issue is 128. Even setting max_sectors_kb
> to 127 is enough to get normal behaviour.

Ok, I don't quite follow how it's initialized to 128k, but your
results appear to confirm the default settings are not optimal for the
interface. The patch you identified just submits commands to the max
size the driver asked to use. If the largest size tanks performance,
I think xen-blkfront should register a smaller transfer size, or maybe
some other constraint needs to be refined.

Roger,

I'm a bit out of my depth here in the xen code. Is this something you
may be able to help clarify?

Thanks,
Keith

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Roger Pau Monné-3
On Fri, Jul 21, 2017 at 11:53:33AM -0400, Keith Busch wrote:

> On Fri, Jul 21, 2017 at 12:19:39PM +0200, Benoit Depail wrote:
> > On 07/20/17 19:36, Keith Busch wrote:
> > >
> > > As a test, could you throttle the xvdb queue's max_sectors_kb? If I
> > > followed xen-blkfront correctly, the default should have it set to 44.
> > > Try setting it to 40.
> > >
> > >   echo 40 > /sys/block/xvdb/queue/max_sectors_kb
> > >
> >
> > The default value on my domU is 128.
> >
> > I ran a couple of tests with different values, starting from 40 and up
> > to 128, clearing the cache between each tests.
> >
> > The only value that showed the issue is 128. Even setting max_sectors_kb
> > to 127 is enough to get normal behaviour.
>
> Ok, I don't quite follow how it's initialized to 128k, but your
> results appear to confirm the default settings are not optimal for the
> interface. The patch you identified just submits commands to the max
> size the driver asked to use. If the largest size tanks performance,
> I think xen-blkfront should register a smaller transfer size, or maybe
> some other constraint needs to be refined.
>
> Roger,
>
> I'm a bit out of my depth here in the xen code. Is this something you
> may be able to help clarify?

Hm, I'm not sure I follow either. AFAIK this problem came from
changing the Linux version in the Dom0 (where the backend, blkback is
running), rather than in the DomU right?

Regarding the queue/sectors stuff, blkfront uses several blk_queue
functions to set those parameters, maybe there's something wrong
there, but I cannot really spot what it is:

http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929

In the past the number of pages that could fit in a single ring
request was limited to 11, but some time ago indirect descriptors
where introduced in order to lift this limit, and now requests can
have a much bigger number of pages.

Could you check the max_sectors_kb of the underlying storage you are
using in Dom0?

Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Benoit Depail
On 07/21/17 18:07, Roger Pau Monné wrote:

> On Fri, Jul 21, 2017 at 11:53:33AM -0400, Keith Busch wrote:
>> On Fri, Jul 21, 2017 at 12:19:39PM +0200, Benoit Depail wrote:
>>> On 07/20/17 19:36, Keith Busch wrote:
>>>>
>>>> As a test, could you throttle the xvdb queue's max_sectors_kb? If I
>>>> followed xen-blkfront correctly, the default should have it set to 44.
>>>> Try setting it to 40.
>>>>
>>>>   echo 40 > /sys/block/xvdb/queue/max_sectors_kb
>>>>
>>>
>>> The default value on my domU is 128.
>>>
>>> I ran a couple of tests with different values, starting from 40 and up
>>> to 128, clearing the cache between each tests.
>>>
>>> The only value that showed the issue is 128. Even setting max_sectors_kb
>>> to 127 is enough to get normal behaviour.
>>
>> Ok, I don't quite follow how it's initialized to 128k, but your
>> results appear to confirm the default settings are not optimal for the
>> interface. The patch you identified just submits commands to the max
>> size the driver asked to use. If the largest size tanks performance,
>> I think xen-blkfront should register a smaller transfer size, or maybe
>> some other constraint needs to be refined.
>>
>> Roger,
>>
>> I'm a bit out of my depth here in the xen code. Is this something you
>> may be able to help clarify?
>
> Hm, I'm not sure I follow either. AFAIK this problem came from
> changing the Linux version in the Dom0 (where the backend, blkback is
> running), rather than in the DomU right?
>
> Regarding the queue/sectors stuff, blkfront uses several blk_queue
> functions to set those parameters, maybe there's something wrong
> there, but I cannot really spot what it is:
>
> http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929
>
> In the past the number of pages that could fit in a single ring
> request was limited to 11, but some time ago indirect descriptors
> where introduced in order to lift this limit, and now requests can
> have a much bigger number of pages.
>
> Could you check the max_sectors_kb of the underlying storage you are
> using in Dom0?
>
> Roger.
>
I checked the value for the loop device as well

With 4.4.77 (bad write performance)
$ cat /sys/block/sda/queue/max_sectors_kb
1280
$ cat /sys/block/loop1/queue/max_sectors_kb
127


With 4.1.42 (normal write performance)
$ cat /sys/block/sda/queue/max_sectors_kb
4096
$ cat /sys/block/loop1/queue/max_sectors_kb
127

Regards,

--
Benoit Depail
Senior Infrastructures Architect
NBS System

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Keith Busch
On Fri, Jul 21, 2017 at 07:07:06PM +0200, Benoit Depail wrote:

> On 07/21/17 18:07, Roger Pau Monné wrote:
> >
> > Hm, I'm not sure I follow either. AFAIK this problem came from
> > changing the Linux version in the Dom0 (where the backend, blkback is
> > running), rather than in the DomU right?
> >
> > Regarding the queue/sectors stuff, blkfront uses several blk_queue
> > functions to set those parameters, maybe there's something wrong
> > there, but I cannot really spot what it is:
> >
> > http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929
> >
> > In the past the number of pages that could fit in a single ring
> > request was limited to 11, but some time ago indirect descriptors
> > where introduced in order to lift this limit, and now requests can
> > have a much bigger number of pages.
> >
> > Could you check the max_sectors_kb of the underlying storage you are
> > using in Dom0?
> >
> > Roger.
> >
> I checked the value for the loop device as well
>
> With 4.4.77 (bad write performance)
> $ cat /sys/block/sda/queue/max_sectors_kb
> 1280
> $ cat /sys/block/loop1/queue/max_sectors_kb
> 127
>
>
> With 4.1.42 (normal write performance)
> $ cat /sys/block/sda/queue/max_sectors_kb
> 4096
> $ cat /sys/block/loop1/queue/max_sectors_kb
> 127

Thank you for the confirmations so far. Could you confirm performance dom0
running 4.4.77 with domU running 4.1.42, and the other way around? Would
like to verify if this is just isolated to blkfront.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Benoit Depail
On 07/26/17 00:25, Keith Busch wrote:

> On Fri, Jul 21, 2017 at 07:07:06PM +0200, Benoit Depail wrote:
>> On 07/21/17 18:07, Roger Pau Monné wrote:
>>>
>>> Hm, I'm not sure I follow either. AFAIK this problem came from
>>> changing the Linux version in the Dom0 (where the backend, blkback is
>>> running), rather than in the DomU right?
>>>
>>> Regarding the queue/sectors stuff, blkfront uses several blk_queue
>>> functions to set those parameters, maybe there's something wrong
>>> there, but I cannot really spot what it is:
>>>
>>> http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929
>>>
>>> In the past the number of pages that could fit in a single ring
>>> request was limited to 11, but some time ago indirect descriptors
>>> where introduced in order to lift this limit, and now requests can
>>> have a much bigger number of pages.
>>>
>>> Could you check the max_sectors_kb of the underlying storage you are
>>> using in Dom0?
>>>
>>> Roger.
>>>
>> I checked the value for the loop device as well
>>
>> With 4.4.77 (bad write performance)
>> $ cat /sys/block/sda/queue/max_sectors_kb
>> 1280
>> $ cat /sys/block/loop1/queue/max_sectors_kb
>> 127
>>
>>
>> With 4.1.42 (normal write performance)
>> $ cat /sys/block/sda/queue/max_sectors_kb
>> 4096
>> $ cat /sys/block/loop1/queue/max_sectors_kb
>> 127
>
> Thank you for the confirmations so far. Could you confirm performance dom0
> running 4.4.77 with domU running 4.1.42, and the other way around? Would
> like to verify if this is just isolated to blkfront.
>
Hi,

I've ran the tests, and I can tell that the domU kernel version have no
influence on the performance.

Dom0 with 4.4.77 always shows bad performance, wether the domU runs
4.1.42 or 4.4.77.

Dom0 with 4.1.42 always shows good performance, wether the domU runs
4.1.42 or 4.4.77.

Thanks,

--
Benoit Depail
Senior Infrastructures Architect
NBS System

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: File-based domU - Slow storage write since xen 4.8

Roger Pau Monné-3
On Fri, Jul 28, 2017 at 04:50:27PM +0200, Benoit Depail wrote:

> On 07/26/17 00:25, Keith Busch wrote:
> > On Fri, Jul 21, 2017 at 07:07:06PM +0200, Benoit Depail wrote:
> >> On 07/21/17 18:07, Roger Pau Monné wrote:
> >>>
> >>> Hm, I'm not sure I follow either. AFAIK this problem came from
> >>> changing the Linux version in the Dom0 (where the backend, blkback is
> >>> running), rather than in the DomU right?
> >>>
> >>> Regarding the queue/sectors stuff, blkfront uses several blk_queue
> >>> functions to set those parameters, maybe there's something wrong
> >>> there, but I cannot really spot what it is:
> >>>
> >>> http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929
> >>>
> >>> In the past the number of pages that could fit in a single ring
> >>> request was limited to 11, but some time ago indirect descriptors
> >>> where introduced in order to lift this limit, and now requests can
> >>> have a much bigger number of pages.
> >>>
> >>> Could you check the max_sectors_kb of the underlying storage you are
> >>> using in Dom0?
> >>>
> >>> Roger.
> >>>
> >> I checked the value for the loop device as well
> >>
> >> With 4.4.77 (bad write performance)
> >> $ cat /sys/block/sda/queue/max_sectors_kb
> >> 1280
> >> $ cat /sys/block/loop1/queue/max_sectors_kb
> >> 127
> >>
> >>
> >> With 4.1.42 (normal write performance)
> >> $ cat /sys/block/sda/queue/max_sectors_kb
> >> 4096
> >> $ cat /sys/block/loop1/queue/max_sectors_kb
> >> 127
> >
> > Thank you for the confirmations so far. Could you confirm performance dom0
> > running 4.4.77 with domU running 4.1.42, and the other way around? Would
> > like to verify if this is just isolated to blkfront.
> >
> Hi,
>
> I've ran the tests, and I can tell that the domU kernel version have no
> influence on the performance.
>
> Dom0 with 4.4.77 always shows bad performance, wether the domU runs
> 4.1.42 or 4.4.77.
>
> Dom0 with 4.1.42 always shows good performance, wether the domU runs
> 4.1.42 or 4.4.77.

Hello,

I haven't yet got time to look into this sadly. Can you please try to
use fio [0] in order to run the tests against the loop device in Dom0?

If possible, could you test several combinations of block sizes, I/O
sizes and I/O depths?

Thanks, Roger.

[0] http://git.kernel.dk/?p=fio.git;a=summary

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
12
Loading...