Massive performance regression

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Massive performance regression

Phil Susi
My server is running Ubuntu 16.04 using Xen 4.6.5-0ubuntu1.4.  There
must have been an update in the last few weeks that seems to have caused
a terrible performance regression.  When I try to fire up a testing vm
to test a live cd, it takes several minutes before anything shows up on
the display, and during that time, qemu-system-i386 is burning through
100% of a cpu ( out of 8 ).

Can anyone suggest how to troubleshoot this further?  Could this be
related to spectre?  If so, how to disable its workarounds?

I took a look at the change log and nothing jumps out at me:

xen (4.6.5-0ubuntu1.4) xenial-security; urgency=medium

  * Applying Xen Security Advisories:
    - CVE-2017-14316 / XSA-231
      - xen/mm: make sure node is less than MAX_NUMNODES
    - CVE-2017-14318 / XSA-232
      - grant_table: fix GNTTABOP_cache_flush handling
    - CVE-2017-14317 / XSA-233
      - tools/xenstore: dont unlink connection object twice
    - CVE-2017-14319 / XSA-234
      - gnttab: also validate PTE permissions upon destroy/replace
    - XSA-235
      - arm/mm: release grant lock on xenmem_add_to_physmap_one() error
paths
    - XSA-237
      - x86: don't allow MSI pIRQ mapping on unowned device
      - x86: enforce proper privilege when (un)mapping pIRQ-s
      - x86/MSI: disallow redundant enabling
      - x86/IRQ: conditionally preserve irq <-> pirq mapping on map error
        paths
      - x86/FLASK: fix unmap-domain-IRQ XSM hook
    - XSA-238
      - x86/ioreq server: correctly handle bogus
        XEN_DMOP_{,un}map_io_range_to_ioreq_server arguments
    - XSA-239
      - x86/HVM: prefill partially used variable on emulation paths
    - XSA-240
      - x86: limit linear page table use to a single level
      - x86/mm: Disable PV linear pagetables by default
    - XSA-241
      - x86: don't store possibly stale TLB flush time stamp
    - XSA-242
      - x86: don't allow page_unlock() to drop the last type reference
    - XSA-243
      - x86: Disable the use of auto-translated PV guestsx86: Disable
the use
        of auto-translated PV guests
      - x86/shadow: Don't create self-linear shadow mappings for 4-level
        translated guests
    - XSA-244
      - x86/cpu: Fix IST handling during PCPU bringup
    - XSA-245
      - xen/page_alloc: Cover memory unreserved after boot in
first_valid_mfn
      - xen/arm: Correctly report the memory region in the dummy NUMA
helpers

 -- Stefan Bader <[hidden email]>  Wed, 11 Oct 2017 15:41:03
+0200


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Massive performance regression

Phil Susi
On 2/28/2018 11:14 AM, Phil Susi wrote:
> My server is running Ubuntu 16.04 using Xen 4.6.5-0ubuntu1.4.  There
> must have been an update in the last few weeks that seems to have caused
> a terrible performance regression.  When I try to fire up a testing vm
> to test a live cd, it takes several minutes before anything shows up on
> the display, and during that time, qemu-system-i386 is burning through
> 100% of a cpu ( out of 8 ).

I forgot to list the .cfg:

name="testing"
builder="hvm"
memory=3000
disk=['/dev/mapper/hyper1-testing,,hda','/mnt/bionic-desktop-amd64.iso,,hdc,cdrom']
vfb=['type=vnc']
vnclisten='10.1.1.8'
vif=['']
usbdevice='tablet'
vga="qxl"


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Massive performance regression

Phil Susi
On 2/28/2018 11:52 AM, Phil Susi wrote:
> On 2/28/2018 11:14 AM, Phil Susi wrote:
>> My server is running Ubuntu 16.04 using Xen 4.6.5-0ubuntu1.4.  There
>> must have been an update in the last few weeks that seems to have caused
>> a terrible performance regression.  When I try to fire up a testing vm
>> to test a live cd, it takes several minutes before anything shows up on
>> the display, and during that time, qemu-system-i386 is burning through
>> 100% of a cpu ( out of 8 ).

I found it.  I first tried reverting to an older Dom0 kernel without the
spectre patches to no avail.  Then I noticed that qemu had been patched
for it as well.  After reverting qemu, boot time for a livecd in a domU
went from 4.5 minutes ( where you never even see the LILO menu at all
and the guest kernel keeps reporting cpu soft lockups ) to less than 30
seconds.

Whatever they did to qemu fscked it up.


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users