A question about SHARE_PFN_WITH_DOMAIN

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

A question about SHARE_PFN_WITH_DOMAIN

Tian, Kevin
I know the purpose of it is to share xenheap pages with domain, so with
ownership change, type/count info update, etc. But is there any reason
we can not allocate them from domheap instead, to grant ownership
directly, since from the very start we know them shared between Xen and
doman? Currently I just saw several types of pages are following this
path:
        - shared info page
        - grant table
        - trace buffer

Maybe I missed some important hints about usage of domheap and xenheap
there? ;-)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: A question about SHARE_PFN_WITH_DOMAIN

Keir Fraser

On 12 Jan 2006, at 08:20, Tian, Kevin wrote:

> I know the purpose of it is to share xenheap pages with domain, so with
> ownership change, type/count info update, etc. But is there any reason
> we can not allocate them from domheap instead, to grant ownership
> directly, since from the very start we know them shared between Xen and
> doman? Currently I just saw several types of pages are following this
> path:
> - shared info page
> - grant table
> - trace buffer
>
> Maybe I missed some important hints about usage of domheap and xenheap
> there? ;-)

Well, there were originally two reason for allowing a guest to own
xenheap pages:

Firstly, a domain page was only accessible in Xen/x86_32 with
map_domain_page(), which is only intended for very short-term mappings.
I added map_domain_page_global() as a temporary interface until current
offenders who create long-term mappings of domheap pages are fixed. In
fact, we could keep that as a permanent interface (if we created lots
of domains we would probably run out of xen heap before we run out of
domain_page_global mapping space).

Secondly, there is the subtle and thorny issue of domain destruction.
Xen assume that any domain that has a non-zero reference count has a
valid shared_info, for example. If we made shared_info a normal domheap
page, and simply had Xen inc that page's refcnt like any other user, we
would have a problem. Xen could not drop its reference until the domain
refcnt drops to zero, but that won't happen until the domain owns no
more pages!

We could work around that second issue with a new page flag to indicate
pages that should not be freed to the heap when their refcnt drops to
zero. Then Xen could set that flag instead of inc'ing the refcnt. That
would allow the domain refcnt to fall to zero and Xen can then
explicitly clear the flag and free the page in the domain destructor.

There's also the issue of whether these pages should be included in the
domain's page count, and whether they should appear in its pseudo-phys
space. If those issues get special-cased too, which they may need to
for backward compatibility, the solution may end up as complex as the
current xenheap-ownership solution.

  -- Keir


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: A question about SHARE_PFN_WITH_DOMAIN

Tian, Kevin
In reply to this post by Tian, Kevin
>From: Keir Fraser [mailto:[hidden email]]
>Sent: 2006年1月12日 18:33
>On 12 Jan 2006, at 08:20, Tian, Kevin wrote:
>
>> Maybe I missed some important hints about usage of domheap and xenheap
>> there? ;-)
>
>Well, there were originally two reason for allowing a guest to own
>xenheap pages:

Thanks, and that does help!

>
>Firstly, a domain page was only accessible in Xen/x86_32 with
>map_domain_page(), which is only intended for very short-term mappings.
>I added map_domain_page_global() as a temporary interface until current
>offenders who create long-term mappings of domheap pages are fixed. In
>fact, we could keep that as a permanent interface (if we created lots
>of domains we would probably run out of xen heap before we run out of
>domain_page_global mapping space).

A good hint for later development on common code. ;-) I didn't realize access to domheap being an issue on x86_32, since there's enough virtual space on 64bit platform and thus map_domain_page is simply defined as __va on them. Yes, permanent mapping is not scalable here.

>
>Secondly, there is the subtle and thorny issue of domain destruction.
>Xen assume that any domain that has a non-zero reference count has a
>valid shared_info, for example.

Could you please point out where I can find such assumption in the code?

>
>We could work around that second issue with a new page flag to indicate
>pages that should not be freed to the heap when their refcnt drops to
>zero. Then Xen could set that flag instead of inc'ing the refcnt. That
>would allow the domain refcnt to fall to zero and Xen can then
>explicitly clear the flag and free the page in the domain destructor.

Or to set initial refcnt of such type of page to '2', indicating both Xen and domain have reference upon it. Whenever Xen want to drop the refcnt, safe to do since domain still holds on. Later when destructing domain, another drop happens and then freed to heap.

>
>There's also the issue of whether these pages should be included in the
>domain's page count, and whether they should appear in its pseudo-phys
>space. If those issues get special-cased too, which they may need to
>for backward compatibility, the solution may end up as complex as the
>current xenheap-ownership solution.
>
>  -- Keir

I see the necessity now.

Thanks a lot,
Kevin

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: A question about SHARE_PFN_WITH_DOMAIN

Keir Fraser

On 13 Jan 2006, at 07:10, Tian, Kevin wrote:

>> Secondly, there is the subtle and thorny issue of domain destruction.
>> Xen assume that any domain that has a non-zero reference count has a
>> valid shared_info, for example.
>
> Could you please point out where I can find such assumption in the
> code?

Sure. For example, event channel bindings are torn down only when the
domain refcnt falls to zero. If we freed the shared_info page when dom0
kills the domain, the refcnt may remain non-zero for some time after
that (because of mappings of network/block ring pages for example). If
dom0 tries to notify via an event channel, the evtchn code in Xen will
happily dereference the dying domU's shared_info pointer which would no
longer be valid.

So we cannot free shared_info until domain_destruct(), and if Xen held
a reference on shared_info then domain_destruct() would never be
called.

  -- Keir


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: A question about SHARE_PFN_WITH_DOMAIN

Tian, Kevin
In reply to this post by Tian, Kevin
>From: Keir Fraser
>Sent: 2006年1月13日 17:15
>
>On 13 Jan 2006, at 07:10, Tian, Kevin wrote:
>
>>> Secondly, there is the subtle and thorny issue of domain destruction.
>>> Xen assume that any domain that has a non-zero reference count has a
>>> valid shared_info, for example.
>>
>> Could you please point out where I can find such assumption in the
>> code?
>
>Sure. For example, event channel bindings are torn down only when the
>domain refcnt falls to zero. If we freed the shared_info page when dom0
>kills the domain, the refcnt may remain non-zero for some time after
>that (because of mappings of network/block ring pages for example). If
>dom0 tries to notify via an event channel, the evtchn code in Xen will
>happily dereference the dying domU's shared_info pointer which would no
>longer be valid.
>
>So we cannot free shared_info until domain_destruct(), and if Xen held
>a reference on shared_info then domain_destruct() would never be
>called.
>
>  -- Keir

Clear and make sense.

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel