cache for partition based VBD?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

cache for partition based VBD?

Tim Freeman
For partition backed VBDs, is there still an effective cache in dom0?  The Linux
buffer cache will affect file based VBDs, but does the partition backed VBD get
around it?

I think that the Linux architecture puts the buffer cache in front of all block
device access, not at the VFS layer.  So does that answer the question?  The
dom0 disk backend driver sits "above" the buffer cache?  (sorry for my crude
understanding)

Thanks,
Tim


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

RE: [Xen-users] cache for partition based VBD?

Petersson, Mats
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of
> Tim Freeman
> Sent: 29 November 2005 17:12
> To: [hidden email]
> Cc: [hidden email]
> Subject: [Xen-users] cache for partition based VBD?
>
> For partition backed VBDs, is there still an effective cache
> in dom0?  The Linux buffer cache will affect file based VBDs,
> but does the partition backed VBD get around it?

Let's first start with the facts:  don't know the answer to your
question

Now for some speculation:
The block device cache would be efficient in the sense that it's useful
for every type of file-system, and it's not going to have to be clever
about what sort of accesses are frequent, not so frequent etc. This
gives benefit to the "cache at the device level".

On the other hand, a file-system level cache would also make sense.
Caching directory structurs or file-allocation-table [not necessarily in
the Microsoft sense] would obviously help a whole lot more than caching
some block from the middle of a file that isn't ever going to be read
again. This motivates having a targetted caching mechanism in the
file-system (VFS) layer.

A combination would of course make even more sense - cache important
data structures in the File-system layer, and cache blocks that have
frequent/recent accesses in the block-device.

>
> I think that the Linux architecture puts the buffer cache in
> front of all block device access, not at the VFS layer.  So
> does that answer the question?  The dom0 disk backend driver
> sits "above" the buffer cache?  (sorry for my crude
> understanding)

I would say that this is correct - the Dom0 disk driver would have a
cache in itself, and the DomU would also have a caching layer. But I'm
not SURE about this at all.

If someone can make further comments, it would be great.


--
Mats

>
> Thanks,
> Tim
>
>
> _______________________________________________
> Xen-users mailing list
> [hidden email]
> http://lists.xensource.com/xen-users
>
>


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Xen-users] cache for partition based VBD?

Tim Freeman
Thanks for responding,

On Tue, 29 Nov 2005 18:56:32 +0100
"Petersson, Mats" <[hidden email]> wrote:

> > -----Original Message-----
> > From: [hidden email]
> > [mailto:[hidden email]] On Behalf Of
> > Tim Freeman
> > Sent: 29 November 2005 17:12
> > To: [hidden email]
> > Cc: [hidden email]
> > Subject: [Xen-users] cache for partition based VBD?
> >
> > For partition backed VBDs, is there still an effective cache
> > in dom0?  The Linux buffer cache will affect file based VBDs,
> > but does the partition backed VBD get around it?
>
> Let's first start with the facts:  don't know the answer to your
> question
>
> Now for some speculation:
> The block device cache would be efficient in the sense that it's useful
> for every type of file-system, and it's not going to have to be clever
> about what sort of accesses are frequent, not so frequent etc. This
> gives benefit to the "cache at the device level".

I'm looking into this a little, reading chapter 15 of Robert Love's 'Linux
Kernel Development,' here is a good bit:

"Individual disk blocks can also tie into the page cache, by way of block I/O
buffers [...] this caching is often referred to as the "buffer cache," although
in reality it is not a separate cache and is part of the page cache."

(the difference here is that the page is larger than the disk block and this I/O
buffer actually describes the mapping of block onto a page... also, he notes
back in Linux 2.2 they used to be separate caches)

He explans there is a temporal locality algorithm *by page* not with any
preference for filesystem structures.

> On the other hand, a file-system level cache would also make sense.
> Caching directory structurs or file-allocation-table [not necessarily in
> the Microsoft sense] would obviously help a whole lot more than caching
> some block from the middle of a file that isn't ever going to be read
> again. This motivates having a targetted caching mechanism in the
> file-system (VFS) layer.

This also happening somewhat, found this statement:

"The Linux VFS layer attempts to cache inodes in memory; this cache is
maintained by a linked list of dentry structures. Each dentry contains a
reference to the inode, and the name of the file. "

http://www.kevinboone.com/linux_kernel_file_1.html

Writing+filesystem-cache seems more complicated, but I'm not worried about this
right now, this talk seems to have a lot of good info on that for ext3:

http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html

I am continuing to look for Xen specific device driver + buffer cache (w/
partition based VBD) evidence besides our experiment.  We're going to run a new
test w/ this setup as well, running sar or vmstat in dom0, but I'd still like to
have code/documentation/xen-developer evidence if possible.  Also looking for
ways of clearing the buffer cache entirely (not just flushing writes).

Thanks,
Tim



_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: cache for partition based VBD?

John Madden-4
In reply to this post by Tim Freeman
> For partition backed VBDs, is there still an effective cache in dom0?  The Linux
> buffer cache will affect file based VBDs, but does the partition backed VBD get
> around it?

Filesystem caching shouldn't happen in dom0 for the domU's and I doubt you'd want
it to.  Caching *will* happen at the domU level and that's where you want it --
close to your applications.

Besides, if you can run a dom0 in like 128MB of memory to devote as much as
possible to your domU's, how much caching could you do?

John



--
John Madden
UNIX Systems Engineer
Ivy Tech Community College of Indiana
[hidden email]


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: [Xen-devel] cache for partition based VBD?

Tim Freeman
In reply to this post by Tim Freeman
On Tue, 29 Nov 2005 12:39:25 -0600
Tim Freeman <[hidden email]> wrote:
[...]
>
> > On the other hand, a file-system level cache would also make sense.
> > Caching directory structurs or file-allocation-table [not necessarily in
> > the Microsoft sense] would obviously help a whole lot more than caching
> > some block from the middle of a file that isn't ever going to be read
> > again. This motivates having a targetted caching mechanism in the
> > file-system (VFS) layer.
>
> This also happening somewhat,

Here is a good link, too:
http://www.linux-tutorial.info/modules.php?name=Tutorial&pageid=278

Tim


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: [Xen-users] cache for partition based VBD?

Tim Freeman
In reply to this post by Petersson, Mats
On Tue, 29 Nov 2005 18:56:32 +0100
"Petersson, Mats" <[hidden email]> wrote:
[...]

> > I think that the Linux architecture puts the buffer cache in
> > front of all block device access, not at the VFS layer.  So
> > does that answer the question?  The dom0 disk backend driver
> > sits "above" the buffer cache?  (sorry for my crude
> > understanding)
>
> I would say that this is correct - the Dom0 disk driver would have a
> cache in itself, and the DomU would also have a caching layer. But I'm
> not SURE about this at all.
>
> If someone can make further comments, it would be great.

I found this post that directly addresses the point:

http://lists.xensource.com/archives/html/xen-devel/2005-05/msg01167.html

(now the experimental data makes less sense to me)

Tim


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: [Xen-users] cache for partition based VBD?

Tim Freeman
On Tue, 29 Nov 2005 17:44:34 -0600
Tim Freeman <[hidden email]> wrote:

> On Tue, 29 Nov 2005 18:56:32 +0100
> "Petersson, Mats" <[hidden email]> wrote:
> [...]
> > > I think that the Linux architecture puts the buffer cache in
> > > front of all block device access, not at the VFS layer.  So
> > > does that answer the question?  The dom0 disk backend driver
> > > sits "above" the buffer cache?  (sorry for my crude
> > > understanding)
> >
> > I would say that this is correct - the Dom0 disk driver would have a
> > cache in itself, and the DomU would also have a caching layer. But I'm
> > not SURE about this at all.
> >
> > If someone can make further comments, it would be great.
>
> I found this post that directly addresses the point:
>
> http://lists.xensource.com/archives/html/xen-devel/2005-05/msg01167.html
>
> (now the experimental data makes less sense to me)
>

Regarding that thread (and one of my own questions), I did find this patch for
clearing the page cache entirely (not just flushing pending writes)...

http://lwn.net/Articles/123343/

Tim


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: RE: [Xen-users] cache for partition based VBD?

Mark Williamson
In reply to this post by Petersson, Mats
> The block device cache would be efficient in the sense that it's useful
> for every type of file-system, and it's not going to have to be clever
> about what sort of accesses are frequent, not so frequent etc. This
> gives benefit to the "cache at the device level".
>
> On the other hand, a file-system level cache would also make sense.
> Caching directory structurs or file-allocation-table [not necessarily in
> the Microsoft sense] would obviously help a whole lot more than caching
> some block from the middle of a file that isn't ever going to be read
> again. This motivates having a targetted caching mechanism in the
> file-system (VFS) layer.

Linux does caching at the level of blocks on a device and of contiguous pages
within a file by stuffing them both in the page cache.  Disk-based
filesystems are able to use a cached interface to block storage (when
appropriate) to access their on disk data structures.  Filesystems use the
block-level interface to the page cache to read their own metadata but store
file contents into the page cache in a file-oriented manner.

The VFS manipulates and caches various meta-stuff like inode data generically
(although it has to get this from the filesystem, which in term retrieves
that data from the block layer).

The filesystem is also free to build its own data structures that cache useful
information.  For instance, filesystems may choose to store metadata in
privately allocated structures or (sometimes) in parts of the page cache.  
There are private pointers in the file and inode structures to facilitate
this.

> A combination would of course make even more sense - cache important
> data structures in the File-system layer, and cache blocks that have
> frequent/recent accesses in the block-device.

Yup.

> > I think that the Linux architecture puts the buffer cache in
> > front of all block device access, not at the VFS layer.  So
> > does that answer the question?  The dom0 disk backend driver
> > sits "above" the buffer cache?  (sorry for my crude
> > understanding)
>
> I would say that this is correct - the Dom0 disk driver would have a
> cache in itself, and the DomU would also have a caching layer. But I'm
> not SURE about this at all.
>
> If someone can make further comments, it would be great.

I'm pretty sure our blkback driver uses a non-caching interface to storage: it
requests stuff directly from the block layer, rather than from the
block-level interface to the page cache.  The only way caching gets to happen
in dom0 is if you're using loop devices in which case direct block access is
looped back into the VFS (and therefore the page cache inevitable gets
involved).

Sounds sane?,
Mark

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel
Reply | Threaded
Open this post in threaded view
|

Re: cache for partition based VBD?

Mark Williamson
In reply to this post by Tim Freeman
> For partition backed VBDs, is there still an effective cache in dom0?  The
> Linux buffer cache will affect file based VBDs, but does the partition
> backed VBD get around it?

AFAIK the blkback driver uses uncached interfaces to storage.  Caching only
happens if you're using loop devices in dom0, because that indirects through
the filesystem, which will being use the page cache.

You don't want caching in dom0, really: the domU will manage its own caches
(and do so more effectively than dom0 could!) anyhow, so there's no point
duplicating the effort.

> I think that the Linux architecture puts the buffer cache in front of all
> block device access, not at the VFS layer.  So does that answer the
> question?  The dom0 disk backend driver sits "above" the buffer cache?
> (sorry for my crude understanding)

AFAIK, it sits *alongside* the page cache layer, as another client of the
block system proper.

Cheers,
Mark

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users