Quantcast

Invalidating buffer cache in dom0

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Invalidating buffer cache in dom0

Jacob Gorm Hansen
hi,

I have a domU that dumps some data (a checkpoint) to a virtual device.
Afterwards, I want to create a new domU based on that data, but because
right now my domain bootstrapper does not have a disk driver, I first
build a ramdisk which I give as an argument to my bootstrapper.

This all works fine, but I have the problem that I have to reboot dom0
between each run, because apparently dom0 Linux does not notice that the
contents on disk have changed, and keep giving me stale data from its
buffer cache when I create the ramdisk. I have tested that this is
actually what happens by grepping for a string in the block device
(/dev/hda5), and the value of the string changes between reboots.

Does anyone know of an ioctl or similar that I could use to make Linux
forget about (not flush) buffer cache contents, or how to disable it?

I have tried reading from the block dev with O_DIRECT but that does not
help.

Thanks,
Jacob

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Invalidating buffer cache in dom0

Ian Pratt

> Does anyone know of an ioctl or similar that I could use to
> make Linux forget about (not flush) buffer cache contents, or
> how to disable it?
>
> I have tried reading from the block dev with O_DIRECT but
> that does not help.

Have you tried using raw devices? See "man raw". I've never used them,
but I'd be interested to know whether this works. There's an argument
that we should be using them by default, particularly for people using
using cluster filesystems -- caching stuff in dom0 could be very bad.

Perhaps some of the ocfs2 guys that hang out on the list could comment?

Thanks,
Ian

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Invalidating buffer cache in dom0

Keir Fraser

On 28 May 2005, at 19:49, Ian Pratt wrote:

> Have you tried using raw devices? See "man raw". I've never used them,
> but I'd be interested to know whether this works. There's an argument
> that we should be using them by default, particularly for people using
> using cluster filesystems -- caching stuff in dom0 could be very bad.

This isn't a problem for domU virtual blkdevs: our backend driver
doesn't hook into the page cache, so unless the block device it sends
requests to loops back to the page cache then we will not be affected
by stale buffer cache contents.

Jacob has given a domU direct access to a partition as a virtual
blkdev. It has modified it, and the updates did go straight to disc.
But if dom0 tries to read partition contents itself then it sees stale
page cache data (because the backend completely circumvented the page
cache). I'm surprised that O_DIRECT didn't fix the problem -- I suppose
it's being 'too clever' in this case and checking for potentially dirty
pages in the cache before going direct to disc.

  -- Keir


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Invalidating buffer cache in dom0

Jacob Gorm Hansen
In reply to this post by Ian Pratt
Ian Pratt wrote:

> Have you tried using raw devices? See "man raw". I've never used them,
> but I'd be interested to know whether this works. There's an argument
> that we should be using them by default, particularly for people using
> using cluster filesystems -- caching stuff in dom0 could be very bad.

It is my impression that in recent 2.6 kernels raw devices and O_DIRECT
are the same thing. Actually, I was forced to use O_DIRECT inside my
domU for the self-checkpointing, as otherwise the buffer cache would
inflate the working set and create additional downtime. O_DIRECT works
really well, and minus a ton of bugs I can now self-checkpoint to disk
as well as self-migrate via the network.

O_DIRECT works for writes. However, it seems it still tries to be a
little too clever when reading from a blockdev.

This is not a huge prob for me, just an annoyance that I have to reboot
all the time to flush the cache, but I guess that for a Xen system in
general it would be nice to have consistency of block device contents
between domUs and dom0.

thanks,
Jacob

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Loading...