Ownership of machine pages: Was: [Xen-devel] Essay on an important Xen decision (long)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Ownership of machine pages: Was: [Xen-devel] Essay on an important Xen decision (long)

Dan Magenheimer
> Date: Wed, 11 Jan 2006 15:56:43 +0800
> From: "Tian, Kevin" <[hidden email]>
> Subject: RE: [Xen-devel] Essay on an important Xen decision (long)
> To: "Magenheimer, Dan \(HP Labs Fort Collins\)"
> <[hidden email]>, "xen-devel"
> <[hidden email]>,
> <[hidden email]>
>
> Hi, Dan,
> Good background for discussion.

For some reason, I only got this through xen-devel (which I
only receive in digest form) so I couldn't respond directly.
But this subtopic -- though highly related -- might deserve
a new thread anyway.

> I don't think it's a good design choice by complete takeover
> to dom0. Moving ownership to dom0 doesn¡¯t mean a simple
> move, since memory sub-system is the core/base of Xen

Very true.  But I expect Xen's memory subsystem will need
to evolve considerably to handle things like:

- support of different page sizes
- cacheable vs uncacheable memory
- sparsely populated memory
- NUMA
- hotplug memory
- oversubscription
- memory reserved for firmware

Now might be a very good time to consider alternative
approaches rather than try to hack these one at a time
into the existing memory subsystem of core/base Xen.

> Extra
> context switches are added for any page related operation.

Hmmm... not really.  Page operations (such as alloc and free)
are relatively rare and can be batched (e.g. at domain startup).
And domU's are not going to be making policy decisions...
If dom0 is making policy decisions (e.g. this stick of RAM is
about to die, who do I steal memory from?), dom0 ownership
of the pages may even reduce context switches.

> Also by P==M model, how do you ensure a scalable allocation
> environment after a long run? Any activities within dom0
> which consumes Physical frames, thus actually eats Machine
> frames.

Just like today's balloon driver, a "machine memory policy
driver" would steal all but some fixed number of dom0's pages
such that the dom0 kernel cannot use them for other purposes.

Dan

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ownership of machine pages: Was: [Xen-devel] Essay on an important Xen decision (long)

Keir Fraser

On 11 Jan 2006, at 22:49, Magenheimer, Dan (HP Labs Fort Collins) wrote:

>> Extra
>> context switches are added for any page related operation.
>
> Hmmm... not really.  Page operations (such as alloc and free)
> are relatively rare and can be batched (e.g. at domain startup).
> And domU's are not going to be making policy decisions...
> If dom0 is making policy decisions (e.g. this stick of RAM is
> about to die, who do I steal memory from?), dom0 ownership
> of the pages may even reduce context switches.

In a literal/extreme form of your model, any address space
access/update would need to be translated and/or validated by domain0.
That's certainly going to be a significant overhead. I believe that
there is a balance to be struck between sufficient mechanism in Xen to
ensure good performance while allowing flexible policy expression in
the control plane. I think talking about this at the summit would be
very interesting -- are your ideas solidified enough perhaps even for a
short presentation?

  -- Keir


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ownership of machine pages: Was: [Xen-devel] Essay on an important Xen decision (long)

Dan Magenheimer
In reply to this post by Dan Magenheimer
(Sorry again for the late and off-thread reply... even though
your message is addressed to me personally, to xen-devel,
and to xen-ia64-devel, I didn't get a copy and just saw it
in xen-devel which I only receive digested.  Some strangeness
in the mailing list nodupe feature maybe?  James cc'ed....)

Yes, I think I could do a short presentation about what
I perceive are the issues and my (admittedly half-baked)
solution, but this would probably be better as a discussion
or brainstorming session than a "classroom" session
with 50 people listening.

> In a literal/extreme form of your model, any address space
> access/update would need to be translated and/or validated by
> domain0.

The literal/extreme form of just about *any* model generally
sucks ;-)  I'm thinking that Xen retains lookup tables so
all accesses are low cost, but (most) updates require domain0.
I think (without enough expertise in Xen/x86 memory management)
that the sticky overhead part will be page-flipping for netif:
Is there any "page affinity" for skbuffs or are they random pages?

Dan

> From: Keir Fraser <[hidden email]>
> Subject: Re: Ownership of machine pages: Was: [Xen-devel] Essay on an
> important Xen decision (long)
>
> On 11 Jan 2006, at 22:49, Magenheimer, Dan (HP Labs Fort
> Collins) wrote:
>
> >> Extra
> >> context switches are added for any page related operation.
> >
> > Hmmm... not really.  Page operations (such as alloc and free)
> > are relatively rare and can be batched (e.g. at domain startup).
> > And domU's are not going to be making policy decisions...
> > If dom0 is making policy decisions (e.g. this stick of RAM is
> > about to die, who do I steal memory from?), dom0 ownership
> > of the pages may even reduce context switches.
>
> In a literal/extreme form of your model, any address space
> access/update would need to be translated and/or validated by
> domain0.
> That's certainly going to be a significant overhead. I believe that
> there is a balance to be struck between sufficient mechanism
> in Xen to
> ensure good performance while allowing flexible policy expression in
> the control plane. I think talking about this at the summit would be
> very interesting -- are your ideas solidified enough perhaps
> even for a
> short presentation?

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ownership of machine pages: Was: [Xen-devel] Essay on an important Xen decision (long)

James Bulpin-2
Magenheimer, Dan (HP Labs Fort Collins) wrote:
> (Sorry again for the late and off-thread reply... even though
> your message is addressed to me personally, to xen-devel,
> and to xen-ia64-devel, I didn't get a copy and just saw it
> in xen-devel which I only receive digested.  Some strangeness
> in the mailing list nodupe feature maybe?  James cc'ed....)

If you were on the recipient list of the original mail from Keir then
that should have gone directly to you without passing anywhere near the
list server, if you didn't get that then it's a problem with mail
servers at your or Keir's end. If you've got nodupe set for
xen-ia64-devel then your address being on the message means that you
won't get the message via the list. You still get the message on the
xen-devel digest because nodupe doesn't apply (or make sense) to digests.

Cheers,
James

_______________________________________________
Xen-ia64-devel mailing list
[hidden email]
http://lists.xensource.com/xen-ia64-devel
Reply | Threaded
Open this post in threaded view
|

Re: Ownership of machine pages: Was: [Xen-devel] Essay on an important Xen decision (long)

jyro
Sorry, if I missed something. But I did not read any discussion on Harry Butterworth's idea of generic memory management API (xenidc and USB driver).

My experience is not enough to comment on this topic, but I would like to understand these design decisions. As far as I know, generic API's that provide proper abstractions prove useful (e.g. Linux VirtualFS). There are counter arguments to it (performance degradation mainly / wrong abstractions). But I didn't find any comments on the xenidc API approach.

Apart from that, thanks for elaborate description of some key concepts.

Jayesh

On 1/12/06, James Bulpin <[hidden email]> wrote:
Magenheimer, Dan (HP Labs Fort Collins) wrote:
> (Sorry again for the late and off-thread reply... even though
> your message is addressed to me personally, to xen-devel,
> and to xen-ia64-devel, I didn't get a copy and just saw it
> in xen-devel which I only receive digested.  Some strangeness
> in the mailing list nodupe feature maybe?  James cc'ed....)

If you were on the recipient list of the original mail from Keir then
that should have gone directly to you without passing anywhere near the
list server, if you didn't get that then it's a problem with mail
servers at your or Keir's end. If you've got nodupe set for
xen-ia64-devel then your address being on the message means that you
won't get the message via the list. You still get the message on the
xen-devel digest because nodupe doesn't apply (or make sense) to digests.

Cheers,
James

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel



--
Jayesh


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: Ownership of machine pages: Was: [Xen-devel] Essay on an important Xen decision (long)

Dan Magenheimer
In reply to this post by Dan Magenheimer
If the Xen drivers are rewritten to be more generic (e.g. to handle more
than just p2m), I agree that xenidc should be seriously considered.
I think the most serious impediment to acceptance of xenidc is
fear of breaking drivers (blk and net) that are already stable.
 
My two cents...
Dan


From: Jayesh Salvi [mailto:[hidden email]]
Sent: Thursday, January 12, 2006 9:04 PM
To: James Bulpin
Cc: Magenheimer, Dan (HP Labs Fort Collins); xen-devel; [hidden email]
Subject: Re: Ownership of machine pages: Was: [Xen-devel] Essay on an important Xen decision (long)

Sorry, if I missed something. But I did not read any discussion on Harry Butterworth's idea of generic memory management API (xenidc and USB driver).

My experience is not enough to comment on this topic, but I would like to understand these design decisions. As far as I know, generic API's that provide proper abstractions prove useful (e.g. Linux VirtualFS). There are counter arguments to it (performance degradation mainly / wrong abstractions). But I didn't find any comments on the xenidc API approach.

Apart from that, thanks for elaborate description of some key concepts.

Jayesh

On 1/12/06, James Bulpin <[hidden email]> wrote:
Magenheimer, Dan (HP Labs Fort Collins) wrote:
> (Sorry again for the late and off-thread reply... even though
> your message is addressed to me personally, to xen-devel,
> and to xen-ia64-devel, I didn't get a copy and just saw it
> in xen-devel which I only receive digested.  Some strangeness
> in the mailing list nodupe feature maybe?  James cc'ed....)

If you were on the recipient list of the original mail from Keir then
that should have gone directly to you without passing anywhere near the
list server, if you didn't get that then it's a problem with mail
servers at your or Keir's end. If you've got nodupe set for
xen-ia64-devel then your address being on the message means that you
won't get the message via the list. You still get the message on the
xen-devel digest because nodupe doesn't apply (or make sense) to digests.

Cheers,
James

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel



--
Jayesh


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel