Re: how to enable shadow page table? Do I have to run HVM guest systems for shadow paging mode?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: how to enable shadow page table? Do I have to run HVM guest systems for shadow paging mode?

Jeremy Fitzhardinge
Long Wang wrote:
> I am working on a research project based on Xen 3.3.1. I want to use the
> shadow page table for setting guest system memory pages as read-only and/or
> dirty, and perform a copy-on-write mechanism when these memory pages are
> updated. I heard that auto_translated_physmap is not supported any more.
>  

Yes, auto_translated_physmap is not supported by Xen or by the pvops
guest kernels.

> Then how can I enable shadow page table? Do I have to run HVM guest systems
> to have shadow paging mode? Can I enable shadow paging mode in PV?
>  

Well, Xen supports log-dirty mode for doing live migration of PV guests,
which looks to be what you're describing above.  You might want to look
into the existing live migration machinery to see if it meets your needs.

    J

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: how to enable shadow page table? Do I have to run HVM guest systems for shadow paging mode?

Jeremy Fitzhardinge
Long Wang wrote:

> I looked into the live migration code in Xen. I need more than the mechanism
> provided in live migration, as described below:
>
> 1. scan the states of all the memory pages of a domain, set dirty pages as
> read-only, and clear all dirty bits for all the memory pages (so read-only
> and dirty are two different bits). This procedure is done periodically.
>
> 2. When a readonly page is to be updated by the domain, the update is
> trapped into the hypervisor, when I copy the page content to another place,
> and set the page as writable (this page is not a page table page), makes the
> update.
>
> I find that the mechanism used in live migration cannot handle these. How
> can I enable shadow paging in PV? If not, do I have to find a machine with
> vmx-featured processor and run HVM for this machine?
>  

Well, that's an option.  But if you're going to be hacking Xen anyway,
then it sounds like log-dirty is close to what you want.  But Tim is
probably the person to ask about that.

    J

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: how to enable shadow page table? Do I have to run HVM guest systems for shadow paging mode?

Tim Deegan-4
In reply to this post by Jeremy Fitzhardinge
At 16:30 +0000 on 16 Mar (1237221012), Jeremy Fitzhardinge wrote:
> Long Wang wrote:
> > I am working on a research project based on Xen 3.3.1. I want to use the
> > shadow page table for setting guest system memory pages as read-only and/or
> > dirty, and perform a copy-on-write mechanism when these memory pages are
> > updated.

Several people are already working on copy-on-write memory in Xen, or
variations on that theme (Mike Sun, Patrick Colp, John Byrne, and
others) as well as more exotic things like Difference Engine and
Transcendent Memory.  It would probably be better to talk to those
people and try to help their efforts than start from scratch.

Patrick Colp's slides from the most recent Xen Summit give an idea of
one method of implementing it:
http://www.xen.org/files/xensummit_oracle09/VMSnapshots.pdf

Cheers,

Tim.

--
Tim Deegan <[hidden email]>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: Re: how to enable shadow page table? Do I have to run HVM guest systems for shadow paging mode?

Dan Magenheimer-2
For the record, Transcendent Memory is not a copy-on-write
mechanism, doesn't deal with shadow tables or PTE read/write
bits, and is really not exotic at all.  It's a simple (though
paravirtualized) mechanism for optimizing the usage of
unutilized or underutilized physical memory while ensuring
(most of) that memory is synchronously reclaimable for
unexpectedly urgent needs.  :-)

Dan

> -----Original Message-----
> From: Tim Deegan [mailto:[hidden email]]
> Sent: Monday, March 16, 2009 11:41 AM
> To: Jeremy Fitzhardinge
> Cc: Xen-devel; Long Wang
> Subject: [Xen-devel] Re: how to enable shadow page table? Do I have to
> run HVM guest systems for shadow paging mode?
>
>
> At 16:30 +0000 on 16 Mar (1237221012), Jeremy Fitzhardinge wrote:
> > Long Wang wrote:
> > > I am working on a research project based on Xen 3.3.1. I
> want to use the
> > > shadow page table for setting guest system memory pages
> as read-only and/or
> > > dirty, and perform a copy-on-write mechanism when these
> memory pages are
> > > updated.
>
> Several people are already working on copy-on-write memory in Xen, or
> variations on that theme (Mike Sun, Patrick Colp, John Byrne, and
> others) as well as more exotic things like Difference Engine and
> Transcendent Memory.  It would probably be better to talk to those
> people and try to help their efforts than start from scratch.
>
> Patrick Colp's slides from the most recent Xen Summit give an idea of
> one method of implementing it:
> http://www.xen.org/files/xensummit_oracle09/VMSnapshots.pdf
>
> Cheers,
>
> Tim.
>
> --
> Tim Deegan <[hidden email]>
> Principal Software Engineer, Citrix Systems (R&D) Ltd.
> [Company #02300071, SL9 0DZ, UK.]
>
> _______________________________________________
> Xen-devel mailing list
> [hidden email]
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: how to enable shadow page table? Do I have to run HVM guest systems for shadow paging mode?

Tim Deegan-4
At 17:57 +0000 on 16 Mar (1237226223), Dan Magenheimer wrote:
> For the record, Transcendent Memory is not a copy-on-write
> mechanism, doesn't deal with shadow tables or PTE read/write
> bits, and is really not exotic at all.

Indeed.  I was just including it in the current bundle of "fun things
to do with memory management". :)  And since it helps with systems under
memory pressure it might have solved whatever the underlying problem was.
Almost nobody wants to write a copy-on-write mechanism just for the
beauty of the thing.

Cheers,

Tim.

>  It's a simple (though
> paravirtualized) mechanism for optimizing the usage of
> unutilized or underutilized physical memory while ensuring
> (most of) that memory is synchronously reclaimable for
> unexpectedly urgent needs.  :-)
>
> Dan
>
> > -----Original Message-----
> > From: Tim Deegan [mailto:[hidden email]]
> > Sent: Monday, March 16, 2009 11:41 AM
> > To: Jeremy Fitzhardinge
> > Cc: Xen-devel; Long Wang
> > Subject: [Xen-devel] Re: how to enable shadow page table? Do I have to
> > run HVM guest systems for shadow paging mode?
> >
> >
> > At 16:30 +0000 on 16 Mar (1237221012), Jeremy Fitzhardinge wrote:
> > > Long Wang wrote:
> > > > I am working on a research project based on Xen 3.3.1. I
> > want to use the
> > > > shadow page table for setting guest system memory pages
> > as read-only and/or
> > > > dirty, and perform a copy-on-write mechanism when these
> > memory pages are
> > > > updated.
> >
> > Several people are already working on copy-on-write memory in Xen, or
> > variations on that theme (Mike Sun, Patrick Colp, John Byrne, and
> > others) as well as more exotic things like Difference Engine and
> > Transcendent Memory.  It would probably be better to talk to those
> > people and try to help their efforts than start from scratch.
> >
> > Patrick Colp's slides from the most recent Xen Summit give an idea of
> > one method of implementing it:
> > http://www.xen.org/files/xensummit_oracle09/VMSnapshots.pdf
> >
> > Cheers,
> >
> > Tim.
> >
> > --
> > Tim Deegan <[hidden email]>
> > Principal Software Engineer, Citrix Systems (R&D) Ltd.
> > [Company #02300071, SL9 0DZ, UK.]
> >
> > _______________________________________________
> > Xen-devel mailing list
> > [hidden email]
> > http://lists.xensource.com/xen-devel
> >

--
Tim Deegan <[hidden email]>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: how to enable shadow page table? Do I have to run HVM guest systems for shadow paging mode?

Tim Deegan-4
In reply to this post by Tim Deegan-4
At 18:40 +0000 on 16 Mar (1237228854), [hidden email] wrote:
> 2) When I call paging_mode_log_dirty(pdom) in hypervisor, the hypervisor
> crashes. I thought paging_mode_log_dirty() enables the shadow paging as
> well as dirty logging after I had read live migration code. But I was
> wrong. paging_mode_log_dirty() crashes because shadow paging is not
> enabled yet.
>
> Now I need to enable shadow paging mode in my testbed (PV).

You wrote all the code before figuring out how to turn on shadow pagetables?

> I tried the
> auto_translated_physmap in the domU config file. But "xm create" failed
> with :
> Error: (1, 'Internal error', 'xc_dom_boot_domU_map: failed to mmap domU
> pages 0x100+0x2da [mmap, errno=14 (Bad address)]\n')

As Jeremy says, auto_translated_physmap doesn't work on recent Xens or
recent Linuxes.  

> So how can I enable shadow paging? Do I have to run HVM for this to work?

If you want the shadow code to hide a layer of address translations
(i.e. have gfn != mfn) then yes, you need HVM.  

Otherwise, you can just copy what the live-migration tools do to turn on
shadow pagetables for PV guests.  It's just one hypercall.  For example,
in python:

 #!/usr/bin/env python
 import sys
 import xen.lowlevel.xc
 xen.lowlevel.xc.xc().shadow_control(dom=int(sys.argv[1]), op=1)

To turn on log-dirty mode, use op=2.  To turn off shadow mode, op=0.

Cheers,

Tim.

> thanks,
> long
>
>
> > At 16:30 +0000 on 16 Mar (1237221012), Jeremy Fitzhardinge wrote:
> >> Long Wang wrote:
> >> > I am working on a research project based on Xen 3.3.1. I want to use
> >> the
> >> > shadow page table for setting guest system memory pages as read-only
> >> and/or
> >> > dirty, and perform a copy-on-write mechanism when these memory pages
> >> are
> >> > updated.
> >
> > Several people are already working on copy-on-write memory in Xen, or
> > variations on that theme (Mike Sun, Patrick Colp, John Byrne, and
> > others) as well as more exotic things like Difference Engine and
> > Transcendent Memory.  It would probably be better to talk to those
> > people and try to help their efforts than start from scratch.
> >
> > Patrick Colp's slides from the most recent Xen Summit give an idea of
> > one method of implementing it:
> > http://www.xen.org/files/xensummit_oracle09/VMSnapshots.pdf
> >
> > Cheers,
> >
> > Tim.
> >
> > --
> > Tim Deegan <[hidden email]>
> > Principal Software Engineer, Citrix Systems (R&D) Ltd.
> > [Company #02300071, SL9 0DZ, UK.]
> >
> > _______________________________________________
> > Xen-devel mailing list
> > [hidden email]
> > http://lists.xensource.com/xen-devel
> >
>
>

--
Tim Deegan <[hidden email]>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: how to enable shadow page table? Do I have torun HVM guest systems for shadow paging mode?

Tim Deegan-4
Hi,

At 05:03 +0000 on 19 Mar (1237439004), Long Wang wrote:
> Here is another question. In the log-dirty mode, who sets the "dirty" bit of
> a level-1 entry?

Depends what you mean.  The dirty bit of a *guest* PTE is set either
directly by the guest or by guest_walk_tables().

The dirty bit of a *shadow* PTE is usually set by _sh_propagate (since
we don't care about it) except for some optimizations to do with HVM
video RAM mappings, where it's set by the MMU.

The dirty bit in the log-dirty bitmap is set by paging_mark_dirty(),
which is called from anywhere that writes to guest memory or allows the
guest a writeable mapping.  It has nothing to do with the dirty bits in
PTEs.

> The mmu, or _sh_propagate()? As far as I know about the x86
> hardware, the dirty bit is independent of the R/W bit. But it seems that
> _sh_propagate() sets all pages as read-only and then sets the dirty bit when
> handling the page fault.

In order to get the *guest* PTE's dirty bit set correctly, we leave the
*shadow* PTE read-only in order to intercept the guest's first write to
the page.  At that point guest_walk_tables() sets the guest-PTE dirty
bit (as an MMU would if shadows were not in use) and _sh_propagate()
makes the shadow PTE R/W.

> Will the mmu sets the dirty bit for a page when the page is written?

Only for the *shadow* PTE, since that's the only one the MMU ever
sees.

Cheers,

Tim.

--
Tim Deegan <[hidden email]>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: how to enable shadow page table? Do I havetorun HVM guest systems for shadow paging mode?

Tim Deegan-4
Hi,

At 15:55 +0000 on 19 Mar (1237478115), Long Wang wrote:

> I am a little confused. You wrote:
> 1) "The dirty bit of a *shadow* PTE is usually set by _sh_propagate (since
> we don't care about it) except for some optimizations to do with HVM video
> RAM mappings, where it's set by the MMU."
>
> 2) "Only for the *shadow* PTE, since that's the only one the MMU ever
> Sees".
>
> Do you mean that most of *shadow* PTE dirty bits (in PV) are set by
> _sh_propagate() (through a read-only-bit-triggered interception)?

No, I mean that when _sh_propagate() makes a shadow PTE that has the R/W
bit set, it usually sets the Dirty bit as well (to save the MMU the
extra write cycle later).

Cheers,

Tim.

--
Tim Deegan <[hidden email]>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: how to enable shadow page table? Do Ihavetorun HVM guest systems for shadow paging mode?

Tim Deegan-4
At 16:40 +0000 on 19 Mar (1237480807), Long Wang wrote:
> So most of shadow PTE dirty bits are still set by the MMU when the MMU finds
> there is a write to a page and the PTE dirty bit is cleared?

The MMU will indeed do this if it finds this is the case.  However as I
said in almost all shadow PTEs we keep the Dirty bit set all the time so
the MMU has nothing to do.

> Is this
> behavior always performed, or shall I turn on a switch for the MMU to set
> the dirty bit?

Always-on.  The MMU's behaviour is described in the Intel SDMs vol 3
sections 3.6 to 3.12, or the AMD APMs vol 2 chapter 5.

And, again, this has nothing to do with the dirty-page bitmap maintained
by log-dirty mode.

Cheers,

Tim.

> Thanks.
>
> long
>
> -----Original Message-----
> From: Tim Deegan [mailto:[hidden email]]
> Sent: Thursday, March 19, 2009 11:16 AM
> To: Long Wang
> Cc: 'Jeremy Fitzhardinge'; 'Xen-devel'
> Subject: Re: [Xen-devel] Re: how to enable shadow page table? Do Ihavetorun
> HVM guest systems for shadow paging mode?
>
> Hi,
>
> At 15:55 +0000 on 19 Mar (1237478115), Long Wang wrote:
> > I am a little confused. You wrote:
> > 1) "The dirty bit of a *shadow* PTE is usually set by _sh_propagate (since
> > we don't care about it) except for some optimizations to do with HVM video
> > RAM mappings, where it's set by the MMU."
> >
> > 2) "Only for the *shadow* PTE, since that's the only one the MMU ever
> > Sees".
> >
> > Do you mean that most of *shadow* PTE dirty bits (in PV) are set by
> > _sh_propagate() (through a read-only-bit-triggered interception)?
>
> No, I mean that when _sh_propagate() makes a shadow PTE that has the R/W
> bit set, it usually sets the Dirty bit as well (to save the MMU the
> extra write cycle later).
>
> Cheers,
>
> Tim.
>
> --
> Tim Deegan <[hidden email]>
> Principal Software Engineer, Citrix Systems (R&D) Ltd.
> [Company #02300071, SL9 0DZ, UK.]
>

--
Tim Deegan <[hidden email]>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: how to enable shadow page table? Do I have to run HVM guest systems for shadow paging mode?

Alina
This post has NOT been accepted by the mailing list yet.
In reply to this post by Tim Deegan-4
Hello,

I have one question. What argument should we give when we are executing the command: python name_of_the_script ?
I presumed we should use as argument the number of the domain that we want the script to be applied to, in my case I work on Dom0, so the command would become: python name_of_the_script 0, but it resulted into an error: xen.lowlevel.xc.Error: (22, 'Invalid argument').

Thank you for your help,

Alina