performance problems...

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

performance problems...

James Harper
I'm using xen-3.0-testing from a week or two ago, and am finding that a
recompile of xen-3.0-testing from hg now is all but killing the
performance in the other domains.

I am running an almost default setup, I've just made some small changes
to bridging.

Any suggestions as to where to start? Or is this a known and solved
problem in the latest -testing?

Thanks

James


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: performance problems...

Ian Pratt
 > I'm using xen-3.0-testing from a week or two ago, and am
> finding that a recompile of xen-3.0-testing from hg now is
> all but killing the performance in the other domains.
>
> I am running an almost default setup, I've just made some
> small changes to bridging.
>
> Any suggestions as to where to start? Or is this a known and
> solved problem in the latest -testing?

There have been no changes in 3.0-testing that are likely relevant. I
don't think anyone else has reported this, so I'd look closely at your
bridging changes. Also please report the test setup in more detail. You
haven't connected dom0 directly to the bridge rather than vif0.0 to the
bridge have you?

Ian

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: performance problems...

James Harper
In reply to this post by James Harper
>  > I'm using xen-3.0-testing from a week or two ago, and am
> > finding that a recompile of xen-3.0-testing from hg now is
> > all but killing the performance in the other domains.
> >
> > I am running an almost default setup, I've just made some
> > small changes to bridging.
> >
> > Any suggestions as to where to start? Or is this a known and
> > solved problem in the latest -testing?
>
> There have been no changes in 3.0-testing that are likely relevant. I
> don't think anyone else has reported this, so I'd look closely at your
> bridging changes. Also please report the test setup in more detail.
You
> haven't connected dom0 directly to the bridge rather than vif0.0 to
the
> bridge have you?

Hmmm... I'm not quite sure what you mean here... I always thought that
in dom0 you just connected eth0 (or whatever you might have renamed it
to) into the bridge. Is this not the case?

'brctl show' gives me:
br0             8000.00508bea6159       no              trunk
br1             8000.00508bea6159       no              br0.2
                                                        vif2.0

I have an Ethernet interface called 'trunk', which is bridged to br0.
br0 then has vlan's on it (the vlan's have to be on the bridge
interface, not on the Ethernet interface, or it doesn't work!), giving
br0.2, which is in turn bridged into br1. The affected DomU is using
br1. None of br0, br1, or trunk have an ip address on them in Dom0. I
have another Ethernet interface called 'lan', which is a gigabit
interface and give's dom0 its connection to the lan, including ATA over
Ethernet. Anything hanging off of 'trunk' is purely for the use of
domU's. Only Dom0 does AoE, All domU's get their disk access via block
devices (which are themselves AoE) exported from dom0.

The slowdown is very very obvious, running a 'make world' in dom0 brings
domU to an almost standstill. Ctrl-Z on the make in dom0 instantly
brings domU back to life. Even if I disable the Ethernet interface in
domU (ifdown eth0), it still runs really really slowly. The system load
goes up in domU too, if that means anything.

Thanks

James

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: performance problems...

James Harper
In reply to this post by James Harper
> > Any suggestions as to where to start? Or is this a known and
> > solved problem in the latest -testing?
>
> There have been no changes in 3.0-testing that are likely relevant. I
> don't think anyone else has reported this, so I'd look closely at your
> bridging changes. Also please report the test setup in more detail.
You
> haven't connected dom0 directly to the bridge rather than vif0.0 to
the
> bridge have you?
>

I just upgraded both my xen machines to the latest (yesterdays hg pull)
and both appear to be doing the same thing now. One of them is SMP
though, so the problem is less apparent, but if do a 'cpuburnP6' on
dom0, the domU's slow down to a crawl.

The problem really does appear to be scheduling related, not network, so
I haven't adjusted the network bridge configuration yet, but will do so
if you tell me it's worth a go.

Thanks

James

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: performance problems...

Ian Pratt
In reply to this post by James Harper
> I just upgraded both my xen machines to the latest
> (yesterdays hg pull) and both appear to be doing the same
> thing now. One of them is SMP though, so the problem is less
> apparent, but if do a 'cpuburnP6' on dom0, the domU's slow
> down to a crawl.

are the guests SMP?

What does 'xm list' show about the relative CPU burn rates?

Ian


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: performance problems...

James Harper
In reply to this post by James Harper
> are the guests SMP?

They are all SMP capable (default -xen kernel config), but except for
one they are all running with 1 vcpu.

> What does 'xm list' show about the relative CPU burn rates?

As I only rebooted recently I don't have any actual figures, but Dom0 is
highish (which makes sense as it was doing a compile), and DomU was low.
Neither had unexpected values in them.

If you can give me the syntax for rolling back to a specific changeset
or time (or I can look it up, but I'm sure you'll know it off the top of
your head :) I can start a binary search on when it all went wrong, and
definitely confirm that the problem didn't exist earlier.

Thanks

James


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: performance problems...

James Harper
In reply to this post by James Harper
Ha! Look at that. I did:

xm sched-sedf Domain-0 0 0 0 1 1
xm sched-sedf mail 0 0 0 1 1

and now I can do a cpuburn (burnP6) in dom0 and you barely notice it.

So... questions...

Sedf must be the default scheduler? I always thought it was bvt? What
are the default settings for it? Obviously they aren't optimal in my
case!

How can I query the current scheduler settings?

Thanks

James

> -----Original Message-----
> From: Mark Nijmeijer [mailto:[hidden email]]
> Sent: Friday, 13 January 2006 12:17
> To: James Harper
> Subject: Re: [Xen-devel] performance problems...
>
>  From the xen-users mailing list, can you try this ?
>
> 2) Scheduling issue
>
> The system was unusable with high CPU load in dom0, all other domains
> were almost dead, and there's apparantly no comprehensive
documentation
> of xen's schedulers (please point out the relevant docs to me if I'm
> wrong, all I found was sedf_scheduler_mini-HOWTO.txt,
> xenwiki/Scheduling and `man xm`, none of which helps to understand
> scheduling well enough to change its parameters). Thanks to Tim
Freeman
> I found an easy setup that works for me: I set the scheduling
parameters
> for all domains via "xm sched-sedf <domID> 0 0 0 1 1" or something
> similar corresponding to example "4 domains with weights 2:3:4:2" from
> the sedf_scheduler_mini-HOWTO.txt with an additional 1 as the
extratime

> parameter if you want to allow a domain to allocate more CPU time if
> other domains are idle.
>
>
>
> James Harper wrote:
>
> >>>Any suggestions as to where to start? Or is this a known and
> >>>solved problem in the latest -testing?
> >>>
> >>>
> >>There have been no changes in 3.0-testing that are likely relevant.
I
> >>don't think anyone else has reported this, so I'd look closely at
your

> >>bridging changes. Also please report the test setup in more detail.
> >>
> >>
> >You
> >
> >
> >>haven't connected dom0 directly to the bridge rather than vif0.0 to
> >>
> >>
> >the
> >
> >
> >>bridge have you?
> >>
> >>
> >>
> >
> >I just upgraded both my xen machines to the latest (yesterdays hg
pull)
> >and both appear to be doing the same thing now. One of them is SMP
> >though, so the problem is less apparent, but if do a 'cpuburnP6' on
> >dom0, the domU's slow down to a crawl.
> >
> >The problem really does appear to be scheduling related, not network,
so
> >I haven't adjusted the network bridge configuration yet, but will do
so

> >if you tell me it's worth a go.
> >
> >Thanks
> >
> >James
> >
> >_______________________________________________
> >Xen-devel mailing list
> >[hidden email]
> >http://lists.xensource.com/xen-devel
> >
> >


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: performance problems...

Keir Fraser

On 13 Jan 2006, at 01:33, James Harper wrote:

> Sedf must be the default scheduler? I always thought it was bvt? What
> are the default settings for it? Obviously they aren't optimal in my
> case!

The default is 75% CPU for domain0, which would explain the unfair
sharing. :-)

  -- Keir


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: performance problems...

James Harper
In reply to this post by James Harper
>
> The default is 75% CPU for domain0, which would explain the unfair
> sharing. :-)
>

I would have said that DomU had less than 25%, based on its nonexistent
performance.

Is it a generally held belief that this arrangement is a good thing? Or
does it just assume that I won't do anything silly like xen recompiles
under dom0?

Thanks

James

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: performance problems...

Keir Fraser

On 13 Jan 2006, at 09:56, James Harper wrote:

>> The default is 75% CPU for domain0, which would explain the unfair
>> sharing. :-)
>>
>
> I would have said that DomU had less than 25%, based on its nonexistent
> performance.
>
> Is it a generally held belief that this arrangement is a good thing? Or
> does it just assume that I won't do anything silly like xen recompiles
> under dom0?

The default basically assumes that dom0 is a server for other domains,
so you want to favour it if it's runnable. There isn't much science
behind that choice of default, I can assure you. :-)

  -- Keir


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel