RE: [Xen-ia64-full 68] Fwd: Topics to for Xen Summit discussion

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

RE: [Xen-ia64-full 68] Fwd: Topics to for Xen Summit discussion

Tian, Kevin
(Sending to broader list since we're touching more details now)

Hi, Yamahata,
        Good to see your comments below and following is some in my mind:

>* physical memory support
>  I agree that phys2mach translation should be adopted.

Yes.

>
>  My implementation proposal is as follows.
>  Just phys to mach translation is sufficient, and xen tracks their change.

Not sure what you mean by "sufficient"? ;-) Just a reminder that once p2m concept is added, we have to promise all places aware of existence of this translation if really required, including some core-header files (like DMA) besides para-drivers. Code there has to explicit query this translation by either direct access or hypercall.

>  There are two implementation candidates.
>  1. map xen's translation table to dom0 address space like virtual memmap.
>     Xen handles tlb miss fault to the area instead of dom0.
>     If dom0 get tlb miss in the area, the mfs is considered INVALID_MFN.
>     There is a choice to determine the mapping address.
>     - Xen determines the address, and pass it to dom0 as boot parameter
>     - add a hyper call for dom0 to map the table at the requested address.

Could you give a reason why you want Xen to intercept the tlb miss upon that area, instead of letting dom0 manage it directly?

Since dom0 is anyhow aware of such virtual area, that means dom0 has to allocate and reserve that virtual range explicitly. Xen is not the right one to decide the mapping address out of xenlinux's knowledge. So we still need modify dom0's init code to do such reservation effort. Since that, why not let dom0 setup the mapping itself?

Furthermore, there're several cases that xenlinux may shrink/expand the memmap:
        - Xenlinux may truncate the EFI memmap by some granularity.

        - Xenlinux may expand the max pfn to compensate request from backend which may require some empty physical frames.

Yes, we can add hypercall to let dom0/Xen sync with such info. But there seems no necessity to do that.

>  2. add a new hyper call which translates phys to mach.

For performance reason, I prefer to let dom0 access that table directly.

>
>  The current implementation allocates pages on demand.
>  There is no difference that a page is not yet allocated or a page
>  is not assigned(dom0 explicitly returned a page to xen).
>  I'm not sure whether this cause problems. but if this cause subtle problem
>  the demand allocation for dom0 can be disabled.

I think allocation on demand is originally introduced there due to lacking of balloon support on XEN/IA64. Once we get balloon working after adding p2m support, it's natural to align ia64 domain builder to other arch like x86, meaning to allocate all pages when creating domain. Actually if dom0 can access p2m directly, that also means we have to allocate all pages in the start. ;-)

>  - writable page table
>    I don't see any reason why this is needed.

Neither do I. Just list for possible discussion.

Thanks,
Kevin

>-----Original Message-----
>From: Isaku Yamahata [mailto:[hidden email]]
>Sent: 2006年1月7日 20:31
>To: [hidden email]; [hidden email]; Yang, Fred;
>[hidden email]
>Cc: Tian, Kevin; [hidden email]
>Subject: Re: [Xen-ia64-full 68] Fwd: Topics to for Xen Summit discussion
>
>
>Hi all.
>
>Evantually I can attend the summit and so the IA64 F2F meeting.
>Thank you very much.
>Oguchi-san of Fujitsu forworded the mail and reqeusted comments.
>Here is my comment.
>
>* vhpt management
>  This issue isn't in the agenda.
>  Is it thought as a post next-step issue?
>  I think vhpt management is critical for the SMP-fication,
>  and the non-VTI and VTI codes can be unified.
>
>* physical memory support
>  I agree that phys2mach translation should be adopted.
>
>  My implementation proposal is as follows.
>  Just phys to mach translation is sufficient, and xen tracks their change.
>  There are two implementation candidates.
>  1. map xen's translation table to dom0 address space like virtual memmap.
>     Xen handles tlb miss fault to the area instead of dom0.
>     If dom0 get tlb miss in the area, the mfs is considered INVALID_MFN.
>     There is a choice to determine the mapping address.
>     - Xen determines the address, and pass it to dom0 as boot parameter
>     - add a hyper call for dom0 to map the table at the requested address.
>  2. add a new hyper call which translates phys to mach.
>
>  The current implementation allocates pages on demand.
>  There is no difference that a page is not yet allocated or a page
>  is not assigned(dom0 explicitly returned a page to xen).
>  I'm not sure whether this cause problems. but if this cause subtle problem
>  the demand allocation for dom0 can be disabled.
>
>* more memory enhancement
>  - page reference count
>    This is needed for domU debug, save/restore, migration.
>    Foreign page mapping is also required.
>
>* timer virtualization
>  This is not IA64 specific so that we should discuss about domain
>  time subsystem modification with x86 (and other-arch) people.
>  I think xen time management implementation can be improved.
>
>Thanks.
>
>On Sat, Jan 07, 2006 at 09:23:46AM +0900, Yoshi. Oguchi wrote:
>
>> I forward the mail from Yang, Fred:
>> Team,
>>
>> Attached is the draft foil for proposed topics we can discuss during
>> Summit F2F.  New topics and Feedback, comments are welcomed!
>> We should also find a time to go dinner all together during summit.  :-)
>> Thanks,
>>
>> -Fred
>
>--
>yamahata

_______________________________________________
Xen-ia64-devel mailing list
[hidden email]
http://lists.xensource.com/xen-ia64-devel
Reply | Threaded
Open this post in threaded view
|

Re: RE: [Xen-ia64-full 68] Fwd: Topics to for Xen Summit discussion

Isaku Yamahata

Hi Kevin.

On Mon, Jan 09, 2006 at 03:07:38PM +0800, Tian, Kevin wrote:

> >* physical memory support
> >  I agree that phys2mach translation should be adopted.
>
> Yes.
>
> >
> >  My implementation proposal is as follows.
> >  Just phys to mach translation is sufficient, and xen tracks their change.
>
> Not sure what you mean by "sufficient"? ;-) Just a reminder that once p2m concept is added, we have to promise all places aware of existence of this translation if really required, including some core-header files (like DMA) besides para-drivers. Code there has to explicit query this translation by either direct access or hypercall.

My more detailed idea to implement the phys2mach is as follows.

* observation
  - xenlinux/X86 maintains phys2table conversion by itself. not by xen.
    the memory area used for the table is maintained by xenlinux and
    the table is writable for xenlinux.
  - By grepping xenlinux code, I observed that xenlinux modifiles
    the phys2mach table via set_phys_to_machine() and that
    set_phys_to_machine() calls are paired with hypercalls
    (xen_machphys_update(), memory op, grant table, update va mapping).
    So I gussed dom0 don't have to write the phys2mach table directly.
    (I can easily miss important things, so I may be wrong.)
  - xen/ia64 already tracks phys2mach conversion by struct arch_domain::mm.
    (currently for domU. dom0 is an exception for now)

* my implementation proposal
  - map xen's arch_domain::mm pte pages to dom0 virtual address space
    linearly with read-only protection.
    Since pte pages are managed by xen, I think it is reasonable for xen
    to handles tlb miss by dom0 on the phys2mach table area.
  - leave set_phys_to_machine() as nop.

You seem to have different ideas.
Do you think that it is needed for dom0 to write phys2mach table directly?


> >  There are two implementation candidates.
> >  1. map xen's translation table to dom0 address space like virtual memmap.
> >     Xen handles tlb miss fault to the area instead of dom0.
> >     If dom0 get tlb miss in the area, the mfs is considered INVALID_MFN.
> >     There is a choice to determine the mapping address.
> >     - Xen determines the address, and pass it to dom0 as boot parameter
> >     - add a hyper call for dom0 to map the table at the requested address.
>
> Could you give a reason why you want Xen to intercept the tlb miss upon that area, instead of letting dom0 manage it directly?
>
> Since dom0 is anyhow aware of such virtual area, that means dom0 has to allocate and reserve that virtual range explicitly. Xen is not the right one to decide the mapping address out of xenlinux's knowledge. So we still need modify dom0's init code to do such reservation effort. Since that, why not let dom0 setup the mapping itself?

I hope that the above answers these questions.


> Furthermore, there're several cases that xenlinux may shrink/expand the memmap:
> - Xenlinux may truncate the EFI memmap by some granularity.
>
> - Xenlinux may expand the max pfn to compensate request from backend which may require some empty physical frames.
>
> Yes, we can add hypercall to let dom0/Xen sync with such info. But there seems no necessity to do that.
>
> >  2. add a new hyper call which translates phys to mach.
>
> For performance reason, I prefer to let dom0 access that table directly.

I agreed.


thanks.
--
yamahata

_______________________________________________
Xen-ia64-devel mailing list
[hidden email]
http://lists.xensource.com/xen-ia64-devel
Reply | Threaded
Open this post in threaded view
|

RE: RE: [Xen-ia64-full 68] Fwd: Topics to for Xen Summit discussion

Tian, Kevin
In reply to this post by Tian, Kevin
>From: Isaku Yamahata [mailto:[hidden email]]
>Sent: 2006年1月10日 15:17
>> >  My implementation proposal is as follows.
>> >  Just phys to mach translation is sufficient, and xen tracks their change.
>>
>> Not sure what you mean by "sufficient"? ;-) Just a reminder that once p2m concept
>is added, we have to promise all places aware of existence of this translation if really
>required, including some core-header files (like DMA) besides para-drivers. Code
>there has to explicit query this translation by either direct access or hypercall.
>
>My more detailed idea to implement the phys2mach is as follows.
>
>* observation
>  - xenlinux/X86 maintains phys2table conversion by itself. not by xen.
>    the memory area used for the table is maintained by xenlinux and
>    the table is writable for xenlinux.

Yes.

>  - By grepping xenlinux code, I observed that xenlinux modifiles
>    the phys2mach table via set_phys_to_machine() and that
>    set_phys_to_machine() calls are paired with hypercalls
>    (xen_machphys_update(), memory op, grant table, update va mapping).
>    So I gussed dom0 don't have to write the phys2mach table directly.
>    (I can easily miss important things, so I may be wrong.)

Yes, that's one place you may need to grep carefully. IIRC, sometimes xenlinux may simply change the phys_to_mach table to INVALID_P2M_ENTRY without notification to Xen if guest pfn is free. Anyway, I think final goal is to minimize architecture bias in common driver code, and if you can do that cleanly then it's OK. I think Keir and Ian have clearest vision on this point. ;-)

>  - xen/ia64 already tracks phys2mach conversion by struct arch_domain::mm.
>    (currently for domU. dom0 is an exception for now)
>
>* my implementation proposal
>  - map xen's arch_domain::mm pte pages to dom0 virtual address space
>    linearly with read-only protection.
>    Since pte pages are managed by xen, I think it is reasonable for xen
>    to handles tlb miss by dom0 on the phys2mach table area.
>  - leave set_phys_to_machine() as nop.
>
>You seem to have different ideas.
>Do you think that it is needed for dom0 to write phys2mach table directly?

No, I mostly agree with your approach on this issue with just one difference: I would like to see phys2mach mapping ptes allocated within domain's configured memory, instead of by Xen. Then you need to modify construct_dom0 and control panel to construct those ptes. Then Later Xen just constructs its multi-level page tables with phys2mach table as L1 directly. By this way, dom0 owns the memory and thus dom0 setups the mapping within its own range. I always think dom0 has better knowledge about how allocated resource will be final utilized and better to let dom0 to setup such stuff by its normal way instead of Xen's supplement.

If we do this way, it's natural to let dom0 modify phys2mach table directly, right?

Thanks,
Kevin


_______________________________________________
Xen-ia64-devel mailing list
[hidden email]
http://lists.xensource.com/xen-ia64-devel
Reply | Threaded
Open this post in threaded view
|

Re: RE: [Xen-ia64-full 68] Fwd: Topics to for Xen Summit discussion

Isaku Yamahata
On Tue, Jan 10, 2006 at 03:52:36PM +0800, Tian, Kevin wrote:

> >* my implementation proposal
> >  - map xen's arch_domain::mm pte pages to dom0 virtual address space
> >    linearly with read-only protection.
> >    Since pte pages are managed by xen, I think it is reasonable for xen
> >    to handles tlb miss by dom0 on the phys2mach table area.
> >  - leave set_phys_to_machine() as nop.
> >
> >You seem to have different ideas.
> >Do you think that it is needed for dom0 to write phys2mach table directly?
>
> No, I mostly agree with your approach on this issue with just one difference: I would like to see phys2mach mapping ptes allocated within domain's configured memory, instead of by Xen. Then you need to modify construct_dom0 and control panel to construct those ptes. Then Later Xen just constructs its multi-level page tables with phys2mach table as L1 directly. By this way, dom0 owns the memory and thus dom0 setups the mapping within its own range. I always think dom0 has better knowledge about how allocated resource will be final utilized and better to let dom0 to setup such stuff by its normal way instead of Xen's supplement.
>
> If we do this way, it's natural to let dom0 modify phys2mach table directly, right?

By 'modify' do you mean that dom0 manages how to map
phys2mach pages into the dom0 virtual address space, right?
If so, xen must verify tlb related operations by dom0 to inhibit
dom0 from writing to pte pages.

I think both proposals (yours and mine) may work well in functionality,
I'm not sure which performs better.

- Xen must verify dom0 tlb related operations to protect pte pages.
  i.e. some checks must be added to vcpu_itr_d(), ...
vs
- Xen tlb miss handler check whether fault address is in the phys2mach
  area and handles the fault specifically.
  i.e. The tlb miss handler of xen/ia64 must be modified to handle
  tlb misses in the phys2mach area.

If I misunderstand, please correct.
--
yamahata

_______________________________________________
Xen-ia64-devel mailing list
[hidden email]
http://lists.xensource.com/xen-ia64-devel
Reply | Threaded
Open this post in threaded view
|

RE: RE: [Xen-ia64-full 68] Fwd: Topics to for Xen Summit discussion

Tian, Kevin
In reply to this post by Tian, Kevin
>From: Isaku Yamahata [mailto:[hidden email]]
>Sent: 2006年1月11日 16:45
>
>>
>> If we do this way, it's natural to let dom0 modify phys2mach table directly, right?
>
>By 'modify' do you mean that dom0 manages how to map
>phys2mach pages into the dom0 virtual address space, right?
>If so, xen must verify tlb related operations by dom0 to inhibit
>dom0 from writing to pte pages.

Let's hold until the summit, since Dan raised a good thread on list for discussion now. Taking Dan's terms here, my original suggestion is for P2M model where Xen constructs the phys2mach table within dom0's configured memory and dom0 can read/write that table/ptes completely together with notification to Xen for necessary case (Yes, it's current x86 model). Then xen doesn’t need inhibit dom0 from writing to pte pages.

If we go for VP model here like domU, dom0 even doesn't know phys2mach and thus xen also doesn't need inhibit. Dom0 just needs issue hypercall if actually want machine frame like DMA operation.

So the point is, once we export phys2mach to dom0, is it still necessary to make it read-only to dom0?

Another related question to think is, is it possible to eliminate direct write by dom0, if all modifications to phys2mach are accompanied with a set of hypercalls following? If true, we can encapsulate phys2mach change into hypercall implementations. For example, change phys2mach table within decrease_reservation instead of split for dom0 to write...

Thanks,
Kevin

>
>I think both proposals (yours and mine) may work well in functionality,
>I'm not sure which performs better.
>
>- Xen must verify dom0 tlb related operations to protect pte pages.
>  i.e. some checks must be added to vcpu_itr_d(), ...
>vs
>- Xen tlb miss handler check whether fault address is in the phys2mach
>  area and handles the fault specifically.
>  i.e. The tlb miss handler of xen/ia64 must be modified to handle
>  tlb misses in the phys2mach area.
>
>If I misunderstand, please correct.
>--
>yamahata

_______________________________________________
Xen-ia64-devel mailing list
[hidden email]
http://lists.xensource.com/xen-ia64-devel