Slow performance inside one VM

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow performance inside one VM

Adam Goryachev-3
I have three windows VM's (domU's) running on the one physical box (dom0).

2 of them are Windows Server 2008R2, one of these is hardly used, but
the other is heavily used.

1 of them is Windows Server 2016, this one is reasonably heavily used
(number of users) but performs well.

The VM with the performance problem has been happening for around 3
months now, but I can't seem to find the issue. I've added additional
memory, but then saw that this wasn't used, so reduced it back a bit.
I've seen all CPU's (within the VM - taskmanager) pegged at 100% for
periods of time, so I've added additional vCPU's and this has stopped
that, but the "performance" issue is still present.

Currently, the config file looks like this:

builder        = 'hvm'
memory        = 20480
name        = "server1"
hostname    = "server1"
localtime    = 1
vcpus        = 22        # How many Virtual CPU's to present
viridian    = 1
disk        = [ 'phy:/dev/ssd/server1,xvda,w' ]
boot        = 'dc'
vif        = ['bridge=xenbr0, mac=00:16:3e:43:ab:94']
vnc        = 1
vncunused    = 0
vncdisplay    = 5
vnclisten    = '0.0.0.0'
stdvga        = 1
#videoram    = 16
usb        = 1
usbdevice    = 'tablet'
acpi        = 1
apic        = 1
on_reboot    = 'restart'
on_poweroff    = 'destroy'
on_crash    = 'restart'

The disk is a whole LV, using a RAID5 (Linux MD) from 3 x Intel SSD

I suspect the issue is a disk issue based on Windows Resource Monitor
showing disk "Response Times" of over 30+ ms, up to 120ms I've seen.

 From the dom0 I can see one of the qemu-system-i386 processes is
sometimes over 50% cpu, but of the 4 CPU's allocated to dom0, none of
them are over 20% utilisation.

  2133 ?        RLsl 1117:45 /usr/bin/qemu-system-i386 -xen-domid 1
-chardev socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-1,server,nowait
-no-shutdown -mon chardev=libxl-cmd,mode=control -chardev
socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-1,server,nowait
-mon chardev=libxenstat-cmd,mode=control -nodefaults -no-user-config
-name server3 -vnc 0.0.0.0:7 -display none -device
cirrus-vga,vgamem_mb=8 -boot order=cd -usb -usbdevice tablet -smp
12,maxcpus=12 -device rtl8139,id=nic0,netdev=net0,mac=00:16:3e:3c:d7:16
-netdev type=tap,id=net0,ifname=vif1.0-emu,script=no,downscript=no
-machine xenfv -m 24568 -drive
file=/dev/ssd/server3,if=ide,index=0,media=disk,format=raw,cache=writeback

However, that process is from server3 (the win2016 domU)

All software is installed from debian packages, versions are:

dpkg -l | grep xen
ii  grub-xen-bin 2.02~beta3-5                            amd64       
GRand Unified Bootloader, version 2 (Xen binaries)
ii  grub-xen-host 2.02~beta3-5                            amd64       
GRand Unified Bootloader, version 2 (Xen host version)
ii  libxen-4.4:amd64 4.4.1-9+deb8u10                        
amd64        Public libs for Xen
ii  libxen-4.8:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
amd64        Public libs for Xen
ii  libxenstore3.0:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
amd64        Xenstore communications library for Xen
ii  xen-hypervisor-4.4-amd64 4.4.1-9+deb8u10                        
amd64        Xen Hypervisor on AMD64
ii  xen-hypervisor-4.8-amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
amd64        Xen Hypervisor on AMD64
ii  xen-linux-system-3.16.0-4-amd64
3.16.51-3                               amd64        Xen system with
Linux 3.16 on 64-bit PCs (meta-package)
ii  xen-linux-system-3.16.0-5-amd64
3.16.51-3+deb8u1                        amd64        Xen system with
Linux 3.16 on 64-bit PCs (meta-package)
ii  xen-linux-system-amd64 4.9+80+deb9u4                          
amd64        Xen system with Linux for 64-bit PCs (dummy package)
ii  xen-system-amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
amd64        Xen System on AMD64 (meta-package)
ii  xen-utils-4.4 4.4.1-9+deb8u10                         amd64       
XEN administrative tools
ii  xen-utils-4.8 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64       
XEN administrative tools
ii  xen-utils-common 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
all          Xen administrative tools - common files
ii  xenstore-utils 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64       
Xenstore command line utilities for Xen

xl info:

host                   : xen3
release                : 4.9.0-6-amd64
version                : #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02)
machine                : x86_64
nr_cpus                : 40
max_cpu_id             : 39
nr_nodes               : 2
cores_per_socket       : 10
threads_per_core       : 2
cpu_mhz                : 2200
hw_caps                :
b7ebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
virt_caps              : hvm hvm_directio
total_memory           : 65403
free_memory            : 8841
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 8
xen_extra              : .3
xen_version            : 4.8.3
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder dom0_mem=6144M,max:6144M
dom0_max_vcpus=4 dom0_vcpus_pin dom0_mem=6144M,max:6144M
dom0_max_vcpus=4 dom0_vcpus_pin gnttab_max_frames=256
cc_compiler            : gcc (Debian 6.3.0-18) 6.3.0 20170516
cc_compile_by          : ijackson
cc_compile_domain      : chiark.greenend.org.uk
cc_compile_date        : Fri Mar  2 16:10:09 UTC 2018
build_id               : dff6bad5189f35adc717d7989e1e2c87b87860cc
xend_config_format     : 4

Within the VM, I've installed the drivers
"Disk drive" displays as "XENSRC PVDISK SCSI Disk Device"
"Storage Controller" displays as "Xen PV Storage Host Adapter" where the
Driver shows Xen Project date 28/2/2017 version 8.2.0.9

Using iostat: /usr/bin/iostat -dmx /dev/sda /dev/sdb /dev/sdc /dev/md0
/dev/md1 /dev/ssd/server1 /dev/ssd/server3 /dev/ssd/server2 1

Shows "% Util" is well under 20% (usually under 5%)

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sdc               1.00    18.00    2.00   47.00     0.01 0.21    
9.41     0.01    0.16    0.00    0.17   0.16   0.80
sda               0.00    18.00    2.00   37.00     0.01 0.17    
9.56     0.01    0.31    0.00    0.32   0.31   1.20
sdb               0.00     2.00    2.00   39.00     0.01 0.12    
6.37     0.02    0.39    0.00    0.41   0.39   1.60
md1               0.00     0.00    0.00   41.00     0.00 0.19    
9.24     0.00    0.00    0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00 0.00    
0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-6              0.00     0.00    0.00   14.00     0.00 0.04    
6.50     0.02    1.14    0.00    1.14   0.57   0.80
dm-10             0.00     0.00    0.00    0.00     0.00 0.00    
0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-18             0.00     0.00    0.00   20.00     0.00 0.14   
14.00     0.01    0.60    0.00    0.60   0.60   1.20

I'm not sure what other information I should provide, and to be honest,
I'm not sure what the bottleneck is. Users are seeing wait times of
greater than 10 seconds to open an application, or new email at times,
while other times it is near instant (less than 2s). All users are
connecting via RDP to this server.

Any hints on how to prove that this is a disk performance issue, or CPU
or memory, or my suspicion, is some xen overhead/inefficiency issue
probably caused by misconfiguration on my part. The system did work
really well prior to January, but we did also add a number of additional
users to the system around that time.

Thank you for any assistance or ideas.

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Slow performance inside one VM

Wiebe Cazemier
----- Original Message -----
> From: "Adam Goryachev" <[hidden email]>
> To: "Xen-users" <[hidden email]>
> Sent: Friday, 6 April, 2018 03:33:03
> Subject: [Xen-users] Slow performance inside one VM

> I have three windows VM's (domU's) running on the one physical box (dom0).
>
> 2 of them are Windows Server 2008R2, one of these is hardly used, but
> the other is heavily used.
>
> 1 of them is Windows Server 2016, this one is reasonably heavily used
> (number of users) but performs well.
>
> The VM with the performance problem has been happening for around 3
> months now, but I can't seem to find the issue. I've added additional
> memory, but then saw that this wasn't used, so reduced it back a bit.
> I've seen all CPU's (within the VM - taskmanager) pegged at 100% for
> periods of time, so I've added additional vCPU's and this has stopped
> that, but the "performance" issue is still present.
>
> Currently, the config file looks like this:
>
> builder        = 'hvm'
> memory        = 20480
> name        = "server1"
> hostname    = "server1"
> localtime    = 1
> vcpus        = 22        # How many Virtual CPU's to present
> viridian    = 1
> disk        = [ 'phy:/dev/ssd/server1,xvda,w' ]
> boot        = 'dc'
> vif        = ['bridge=xenbr0, mac=00:16:3e:43:ab:94']
> vnc        = 1
> vncunused    = 0
> vncdisplay    = 5
> vnclisten    = '0.0.0.0'
> stdvga        = 1
> #videoram    = 16
> usb        = 1
> usbdevice    = 'tablet'
> acpi        = 1
> apic        = 1
> on_reboot    = 'restart'
> on_poweroff    = 'destroy'
> on_crash    = 'restart'
>
> The disk is a whole LV, using a RAID5 (Linux MD) from 3 x Intel SSD
>
> I suspect the issue is a disk issue based on Windows Resource Monitor
> showing disk "Response Times" of over 30+ ms, up to 120ms I've seen.
>
> From the dom0 I can see one of the qemu-system-i386 processes is
> sometimes over 50% cpu, but of the 4 CPU's allocated to dom0, none of
> them are over 20% utilisation.
>
>  2133 ?        RLsl 1117:45 /usr/bin/qemu-system-i386 -xen-domid 1
> -chardev socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-1,server,nowait
> -no-shutdown -mon chardev=libxl-cmd,mode=control -chardev
> socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-1,server,nowait
> -mon chardev=libxenstat-cmd,mode=control -nodefaults -no-user-config
> -name server3 -vnc 0.0.0.0:7 -display none -device
> cirrus-vga,vgamem_mb=8 -boot order=cd -usb -usbdevice tablet -smp
> 12,maxcpus=12 -device rtl8139,id=nic0,netdev=net0,mac=00:16:3e:3c:d7:16
> -netdev type=tap,id=net0,ifname=vif1.0-emu,script=no,downscript=no
> -machine xenfv -m 24568 -drive
> file=/dev/ssd/server3,if=ide,index=0,media=disk,format=raw,cache=writeback
>
> However, that process is from server3 (the win2016 domU)
>
> All software is installed from debian packages, versions are:
>
> dpkg -l | grep xen
> ii  grub-xen-bin 2.02~beta3-5                            amd64
> GRand Unified Bootloader, version 2 (Xen binaries)
> ii  grub-xen-host 2.02~beta3-5                            amd64
> GRand Unified Bootloader, version 2 (Xen host version)
> ii  libxen-4.4:amd64 4.4.1-9+deb8u10
> amd64        Public libs for Xen
> ii  libxen-4.8:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
> amd64        Public libs for Xen
> ii  libxenstore3.0:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
> amd64        Xenstore communications library for Xen
> ii  xen-hypervisor-4.4-amd64 4.4.1-9+deb8u10
> amd64        Xen Hypervisor on AMD64
> ii  xen-hypervisor-4.8-amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
> amd64        Xen Hypervisor on AMD64
> ii  xen-linux-system-3.16.0-4-amd64
> 3.16.51-3                               amd64        Xen system with
> Linux 3.16 on 64-bit PCs (meta-package)
> ii  xen-linux-system-3.16.0-5-amd64
> 3.16.51-3+deb8u1                        amd64        Xen system with
> Linux 3.16 on 64-bit PCs (meta-package)
> ii  xen-linux-system-amd64 4.9+80+deb9u4
> amd64        Xen system with Linux for 64-bit PCs (dummy package)
> ii  xen-system-amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
> amd64        Xen System on AMD64 (meta-package)
> ii  xen-utils-4.4 4.4.1-9+deb8u10                         amd64
> XEN administrative tools
> ii  xen-utils-4.8 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64
> XEN administrative tools
> ii  xen-utils-common 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
> all          Xen administrative tools - common files
> ii  xenstore-utils 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 amd64
> Xenstore command line utilities for Xen
>
> xl info:
>
> host                   : xen3
> release                : 4.9.0-6-amd64
> version                : #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02)
> machine                : x86_64
> nr_cpus                : 40
> max_cpu_id             : 39
> nr_nodes               : 2
> cores_per_socket       : 10
> threads_per_core       : 2
> cpu_mhz                : 2200
> hw_caps                :
> b7ebfbff:77fef3ff:2c100800:00000121:00000001:001cbfbb:00000000:00000100
> virt_caps              : hvm hvm_directio
> total_memory           : 65403
> free_memory            : 8841
> sharing_freed_memory   : 0
> sharing_used_memory    : 0
> outstanding_claims     : 0
> free_cpus              : 0
> xen_major              : 4
> xen_minor              : 8
> xen_extra              : .3
> xen_version            : 4.8.3
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          :
> xen_commandline        : placeholder dom0_mem=6144M,max:6144M
> dom0_max_vcpus=4 dom0_vcpus_pin dom0_mem=6144M,max:6144M
> dom0_max_vcpus=4 dom0_vcpus_pin gnttab_max_frames=256
> cc_compiler            : gcc (Debian 6.3.0-18) 6.3.0 20170516
> cc_compile_by          : ijackson
> cc_compile_domain      : chiark.greenend.org.uk
> cc_compile_date        : Fri Mar  2 16:10:09 UTC 2018
> build_id               : dff6bad5189f35adc717d7989e1e2c87b87860cc
> xend_config_format     : 4
>
> Within the VM, I've installed the drivers
> "Disk drive" displays as "XENSRC PVDISK SCSI Disk Device"
> "Storage Controller" displays as "Xen PV Storage Host Adapter" where the
> Driver shows Xen Project date 28/2/2017 version 8.2.0.9
>
> Using iostat: /usr/bin/iostat -dmx /dev/sda /dev/sdb /dev/sdc /dev/md0
> /dev/md1 /dev/ssd/server1 /dev/ssd/server3 /dev/ssd/server2 1
>
> Shows "% Util" is well under 20% (usually under 5%)
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sdc               1.00    18.00    2.00   47.00     0.01 0.21
> 9.41     0.01    0.16    0.00    0.17   0.16   0.80
> sda               0.00    18.00    2.00   37.00     0.01 0.17
> 9.56     0.01    0.31    0.00    0.32   0.31   1.20
> sdb               0.00     2.00    2.00   39.00     0.01 0.12
> 6.37     0.02    0.39    0.00    0.41   0.39   1.60
> md1               0.00     0.00    0.00   41.00     0.00 0.19
> 9.24     0.00    0.00    0.00    0.00   0.00   0.00
> md0               0.00     0.00    0.00    0.00     0.00 0.00
> 0.00     0.00    0.00    0.00    0.00   0.00   0.00
> dm-6              0.00     0.00    0.00   14.00     0.00 0.04
> 6.50     0.02    1.14    0.00    1.14   0.57   0.80
> dm-10             0.00     0.00    0.00    0.00     0.00 0.00
> 0.00     0.00    0.00    0.00    0.00   0.00   0.00
> dm-18             0.00     0.00    0.00   20.00     0.00 0.14
> 14.00     0.01    0.60    0.00    0.60   0.60   1.20
>
> I'm not sure what other information I should provide, and to be honest,
> I'm not sure what the bottleneck is. Users are seeing wait times of
> greater than 10 seconds to open an application, or new email at times,
> while other times it is near instant (less than 2s). All users are
> connecting via RDP to this server.
>
> Any hints on how to prove that this is a disk performance issue, or CPU
> or memory, or my suspicion, is some xen overhead/inefficiency issue
> probably caused by misconfiguration on my part. The system did work
> really well prior to January, but we did also add a number of additional
> users to the system around that time.
>
> Thank you for any assistance or ideas.
>
> Regards,
> Adam
>

You seem to have covered a lot of bases and I'm not sure what I can offer, especially without being at the server, but perhaps this issue I had will help you:

https://serverfault.com/questions/660575/which-scheduler-to-change-on-lvm-to-benefit-virtual-machines

Look at my own answer to the question.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users