ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

Denis Schneider
Hi,
I am using a Cubieboard2 with Xen 4.4.
After building Xen, installing Debian etc. as described there:
http://openmirage.org/wiki/xen-on-cubieboard2, I tried to use Debian
and Gentoo as domU.
Xen and the dom0 kernel were compiled with the Linaro

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

Denis Schneider
Hi,
I am using a Cubieboard2 with Xen 4.4.
After building Xen, installing Debian etc. as described there:
http://openmirage.org/wiki/xen-on-cubieboard2, I tried to use Debian
and Gentoo as domU.
Xen and the dom0 kernel were compiled with the Linaro toolchain:
arm-linux-gnueabihf-gcc (crosstool-NG linaro-1.13.1-4.8-2013.10 -
Linaro GCC 2013.10) 4.8.2 20131014 (prerelease)
The kernel was compiled from
https://github.com/linux-sunxi/linux-sunxi.git, branch sunxi-devel at
a076583266.

Unfortunatly, the machine won't tolerate network load. If I do "nc -l
-p </dev/zero" on domU and "nc $domUIP 1234 > /dev/null" on dom0 or an
external host, I get a constant stream of:
[ 189.507495] xen_add_phys_to_mach_entry: cannot add pfn=0x0006930f ->
mfn=0x0004c3bc: pfn=0x00069310 -> mfn=0x0004c3bc already exists
[ 189.531185] xen_add_phys_to_mach_entry: cannot add pfn=0x0006921d ->
mfn=0x0004c489: pfn=0x0006928f -> mfn=0x0004c489 already exists
[ 189.533478] xen_add_mach_to_phys_entry: cannot add pfn=0x0006921a ->
mfn=0x0004c485: pfn=0x00069283 -> mfn=0x0004c485 already exists
[ 189.536304] xen_add_phys_to_mach_entry: cannot add pfn=0x00069212 ->
mfn=0x0004c485: pfn=0x00069213 -> mfn=0x0004c485 already exists
[ 189.536843] xen_add_mach_to_phys_entry: cannot add pfn=0x00069211 ->
mfn=0x0004c484: pfn=0x0006853a -> mfn=0x0004c484 already exists
[ 189.536909] xen_add_phys_to_mach_entry: cannot add pfn=0x00069208 ->
mfn=0x0004c391: pfn=0x00069210 -> mfn=0x0004c391 already exists
[ 189.539105] xen_add_phys_to_mach_entry: cannot add pfn=0x00069204 ->
mfn=0x0004c485: pfn=0x00069207 -> mfn=0x0004c485 already exists
[ 189.539164] xen_add_mach_to_phys_entry: cannot add pfn=0x0006853f ->
mfn=0x0004c390: pfn=0x0006920a -> mfn=0x0004c390 already exists
[ 189.539569] xen_add_mach_to_phys_entry: cannot add pfn=0x0006852f ->
mfn=0x0004c484: pfn=0x0006853e -> mfn=0x0004c484 already exists
[ 189.541451] xen_add_phys_to_mach_entry: cannot add pfn=0x00068538 ->
mfn=0x0004c484: pfn=0x0006852c -> mfn=0x0004c484 already exists
[ 190.382277] xen_add_mach_to_phys_entry: cannot add pfn=0x00069110 ->
mfn=0x0004c3cc: pfn=0x00069129 -> mfn=0x0004c3cc already exists
[ 190.437460] xen_add_mach_to_phys_entry: cannot add pfn=0x00069280 ->
mfn=0x0004c3ea: pfn=0x00068533 -> mfn=0x0004c3ea already exists
[ 191.022920] xen_add_phys_to_mach_entry: cannot add pfn=0x00069200 ->
mfn=0x0004c3d0: pfn=0x0006920f -> mfn=0x0004c3d0 already exists
[ 191.029744] xen_add_phys_to_mach_entry: cannot add pfn=0x00068532 ->
mfn=0x0004c38e: pfn=0x00068529 -> mfn=0x0004c38e already exists

followed by a panic when disk activity occurs:
[ 300.924382] Unable to handle kernel paging request at virtual address aadd5000
[ 300.924434] pgd = c0004000
[ 300.924450] [aadd5000] *pgd=00000000
[ 300.924474] Internal error: Oops: 2805 [#1] ARM
[ 300.924491] Modules linked in:
[ 300.924521] CPU: 0 PID: 0 Comm: swapper Not tainted 3.15.0-00201-ga076583 #17
[ 300.924546] task: c055d5e0 ti: c0552000 task.ti: c0552000
[ 300.924579] PC is at v7_dma_inv_range+0x30/0x48
[ 300.924601] LR is at __dma_page_dev_to_cpu+0x80/0x124
[ 300.924622] pc : [<c0017ef4>] lr : [<c00140e8>] psr: 40010193
[ 300.924622] sp : c0553d68 ip : c05b01fc fp : 00000000
[ 300.924673] r10: 00001000 r9 : 00000002 r8 : cbb35aa0
[ 300.924693] r7 : 00000000 r6 : 00001000 r5 : c05b01fc r4 : c055d300
[ 300.924714] r3 : 0000003f r2 : 00000040 r1 : aadd6000 r0 : aadd5000
[ 300.924738] Flags: nZcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 300.924763] Control: 10c5387d Table: 6af78059 DAC: 00000015
[ 300.924783] Process swapper (pid: 0, stack limit = 0xc0552240)
[ 300.924804] Stack: (0xc0553d68 to 0xc0554000)
[ 300.924827] 3d60: c0017f84 c0553d84 c03bf560 4add5000 00000000 0004add5
[ 300.924855] 3d80: 4add5000 cb895e10 00000002 c01cdb30 00001000
00000002 00000000 000690da
[ 300.924883] 3da0: 00000482 00000000 00000600 c92af000 00000001
00000001 00000002 cb895e10
[ 300.924911] 3dc0: 00000000 cbb02190 0000249c c01cde34 00001000
00000002 00000000 20010193
[ 300.924939] 3de0: 00000001 cbb10a80 c92af000 cbb10000 00000001
000000b0 00000001 c023b2b4
[ 300.924967] 3e00: 00000000 20010193 c0553e24 c003b9a8 cb82e508
cbb10a80 cbb10000 cbb11640
[ 300.924995] 3e20: 00000001 c023b544 00000000 0000000f cbb10000
c023b90c cbb10000 cbb11640
[ 300.925023] 3e40: 00000000 cbb10000 00000008 d08a2100 00000001
cbb00010 cbae7c10 c024d8d4
[ 300.925051] 3e60: fff27870 ffffffff fff15a00 ffffffff 000361e4
00000000 ffeea214 00000000
[ 300.925079] 3e80: 00000001 00000001 cbb02190 d08a2000 c055a01c
cbb02190 0000249c c024e000
[ 300.925107] 3ea0: 00000000 3b9aca00 00000000 00000001 c055a0e0
cbb03040 c0565c14 00000000
[ 300.925135] 3ec0: 00000000 00000058 cb807c00 c0586790 00000001
c00496a0 00000046 20010193
[ 300.925163] 3ee0: c0561db8 cb807c00 c0565c14 00000000 d0804000
c057df88 00000000 c0552000
[ 300.925192] 3f00: ffffffed c00497cc cb807c00 c004bfe0 c004bf48
00000058 00000058 c0048ec4
[ 300.925220] 3f20: c0565a30 c000f048 d080400c c055a404 c0553f50
c0008540 c000f19c c000f1a0
[ 300.925249] 3f40: 60010013 ffffffff c0553f84 c0011e40 ffffffed
00000000 00000001 00000000
[ 300.925277] 3f60: c055a0b8 0000004c c0552000 00000000 c057df88
00000000 c0552000 ffffffed
[ 300.925305] 3f80: 01000000 c0553f98 c000f19c c000f1a0 60010013
ffffffff c055adcc c0040c5c
[ 300.925333] 3fa0: cbfff5c0 00000000 c058678e c0552000 00000000
c055a000 00000000 c0526a40
[ 300.925361] 3fc0: ffffffff ffffffff c05264ec 00000000 00000000
c0546bc8 00000000 10c5387d
[ 300.925389] 3fe0: c055a04c c0546bc4 c055e6c8 60004059 410fc074
60008070 00000000 00000000
[ 300.925431] [<c0017ef4>] (v7_dma_inv_range) from [<c00140e8>]
(__dma_page_dev_to_cpu+0x80/0x124)
[ 300.925471] [<c00140e8>] (__dma_page_dev_to_cpu) from [<c01cdb30>]
(xen_unmap_single+0xbc/0x108)
[ 300.925508] [<c01cdb30>] (xen_unmap_single) from [<c01cde34>]
(xen_swiotlb_unmap_sg_attrs+0x48/0x68)
[ 300.925544] [<c01cde34>] (xen_swiotlb_unmap_sg_attrs) from
[<c023b2b4>] (ata_sg_clean+0x8c/0x120)
[ 300.925580] [<c023b2b4>] (ata_sg_clean) from [<c023b544>]
(__ata_qc_complete+0x34/0x144)
[ 300.925613] [<c023b544>] (__ata_qc_complete) from [<c023b90c>]
(ata_qc_complete_multiple+0xa0/0xd4)
[ 300.925649] [<c023b90c>] (ata_qc_complete_multiple) from
[<c024d8d4>] (ahci_handle_port_interrupt+0x124/0x5b0)
[ 300.925688] [<c024d8d4>] (ahci_handle_port_interrupt) from
[<c024e000>] (ahci_interrupt+0xc0/0x158)
[ 300.925724] [<c024e000>] (ahci_interrupt) from [<c00496a0>]
(handle_irq_event_percpu+0x38/0x13c)
[ 300.925759] [<c00496a0>] (handle_irq_event_percpu) from [<c00497cc>]
(handle_irq_event+0x28/0x38)
[ 300.925794] [<c00497cc>] (handle_irq_event) from [<c004bfe0>]
(handle_fasteoi_irq+0x98/0x17c)
[ 300.925828] [<c004bfe0>] (handle_fasteoi_irq) from [<c0048ec4>]
(generic_handle_irq+0x2c/0x3c)
[ 300.925862] [<c0048ec4>] (generic_handle_irq) from [<c000f048>]
(handle_IRQ+0x38/0x84)
[ 300.925899] [<c000f048>] (handle_IRQ) from [<c0008540>]
(gic_handle_irq+0x2c/0x54)
[ 300.925929] [<c0008540>] (gic_handle_irq) from [<c0011e40>]
(__irq_svc+0x40/0x50)
[ 300.925953] Exception stack(0xc0553f50 to 0xc0553f98)
[ 300.925973] 3f40: ffffffed 00000000 00000001 00000000
[ 300.926001] 3f60: c055a0b8 0000004c c0552000 00000000 c057df88
00000000 c0552000 ffffffed
[ 300.926028] 3f80: 01000000 c0553f98 c000f19c c000f1a0 60010013 ffffffff
[ 300.926057] [<c0011e40>] (__irq_svc) from [<c000f1a0>]
(arch_cpu_idle+0x2c/0x30)
[ 300.926090] [<c000f1a0>] (arch_cpu_idle) from [<c0040c5c>]
(cpu_startup_entry+0xf8/0x1dc)
[ 300.926125] [<c0040c5c>] (cpu_startup_entry) from [<c0526a40>]
(start_kernel+0x32c/0x338)
[ 300.926155] Code: 1e070f3e e1110003 e1c11003 1e071f3e (ee070f36)
[ 300.926181] ---[ end trace edc3479207e4afcc ]---
[ 300.926200] Kernel panic - not syncing: Fatal exception in interrupt
[ 300.926222] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

disk activity is generated with "find /usr > /dev/null" on domU.
As you see from the panic stack trace, the domU root device is a SATA
HDD connected to the AHCI port.
If you use an SD card, you get a similar panic, but with the mmc
interrupt handler.
A panic can also occur tens of seconds after the network IO seized if
heavy disk IO follows.

I tried compiling the dom0 kernel with Gentoo-gcc 4.7.3 and
Linaro-4.8.2 and Gentoo-4.8.3 without additional CFLAGS. The result is
the same.

To me, it looks as if xen-netback or something in this area corrupts
memory on large amounts of network IO.
Does anybody have a working setup on a Cubieboard2 or similar that is
really stable? Or a clue for me what is really happening here?
I'd happily give more information or try out more things.

I am not the only one with this problem and also tried different power
supplies. The dom0 kernel is stable if used without domU. So it
shouldn't be a hardware problem.

Kind regards,

Denis

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

Ian Campbell-10
On Tue, 2014-07-01 at 16:22 +0200, Denis Schneider wrote:

> Hi,
> I am using a Cubieboard2 with Xen 4.4.
> After building Xen, installing Debian etc. as described there:
> http://openmirage.org/wiki/xen-on-cubieboard2, I tried to use Debian
> and Gentoo as domU.
> Xen and the dom0 kernel were compiled with the Linaro toolchain:
> arm-linux-gnueabihf-gcc (crosstool-NG linaro-1.13.1-4.8-2013.10 -
> Linaro GCC 2013.10) 4.8.2 20131014 (prerelease)
> The kernel was compiled from
> https://github.com/linux-sunxi/linux-sunxi.git, branch sunxi-devel at
> a076583266.

First thing I would recommend would be to try the latest mainline stable
3.15.x release. I think everything needed for a usable sunxi system is
in there already so no need for the sunxi-devel branch

(See http://linux-sunxi.org/Linux_mainlining_effort, I think the only
thing you might miss would be the MMC driver, but I would suggest SATA
anyway...)

The reason I suggest the latest 3.15.x is that there were a few
interesting netback bugs but I think they've all been backported to
stable by now.

Ian.


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

Denis Schneider
Hi an,
thank you for your reply.

2014-07-02 11:56 GMT+02:00 Ian Campbell <[hidden email]>:
> First thing I would recommend would be to try the latest mainline stable
> 3.15.x release. I think everything needed for a usable sunxi system is
> in there already so no need for the sunxi-devel branch

I tried Linus' linux.git/master, which corresponds to 3.16 --
resulting in the same messages and panic.
Besides that, the mainline kernel works quite well.
BTW, git shows that sunxi-devel and mainline Linux v3.15.2 are the
same for drivers/net/xen-netback, though linux.git/master shows some
changes.

The bug can easily be triggered if you access blkback and netback in
parallel (thanks to Maximilian), e.g.
domU: iperf -s & cat /dev/xvda > /dev/null
dom0: iperf -t 3600 -c domU

It does not matter if the underlying dom0 block device is on a SATA,
USB or mmc device. The panic is similar.

> The reason I suggest the latest 3.15.x is that there were a few
> interesting netback bugs but I think they've all been backported to
> stable by now.

I hope that they are all included in linux.git/master @ 16874b2,
regarding xen-netback, those changes occurred from sunxi-devel to
16874b2:
* xen-netback: bookkeep number of active queues in our own module
* net: xen-netback: include linux/vmalloc.h again
* xen-netback: Add support for multiple queues
* xen-netback: Factor queue-specific data into queue struct
* xen-netback: Move grant_copy_op array back into struct xenvif.
* net: get rid of SET_ETHTOOL_OPS

Interestingly, it takes some time until the bug triggers and the time
increased when I switched from linux-sunxi to mainline.

Do you have any idea what happens here? I am a bit clueless what's going on.

Denis

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

Ian Campbell-10
On Thu, 2014-07-03 at 02:01 +0200, Denis Schneider wrote:

> Hi an,
> thank you for your reply.
>
> 2014-07-02 11:56 GMT+02:00 Ian Campbell <[hidden email]>:
> > First thing I would recommend would be to try the latest mainline stable
> > 3.15.x release. I think everything needed for a usable sunxi system is
> > in there already so no need for the sunxi-devel branch
>
> I tried Linus' linux.git/master, which corresponds to 3.16 --
> resulting in the same messages and panic.
> Besides that, the mainline kernel works quite well.
> BTW, git shows that sunxi-devel and mainline Linux v3.15.2 are the
> same for drivers/net/xen-netback, though linux.git/master shows some
> changes.
>
> The bug can easily be triggered if you access blkback and netback in
> parallel (thanks to Maximilian), e.g.
> domU: iperf -s & cat /dev/xvda > /dev/null
> dom0: iperf -t 3600 -c domU
>
> It does not matter if the underlying dom0 block device is on a SATA,
> USB or mmc device. The panic is similar.
>
> > The reason I suggest the latest 3.15.x is that there were a few
> > interesting netback bugs but I think they've all been backported to
> > stable by now.
>
> I hope that they are all included in linux.git/master @ 16874b2,
> regarding xen-netback, those changes occurred from sunxi-devel to
> 16874b2:
> * xen-netback: bookkeep number of active queues in our own module
> * net: xen-netback: include linux/vmalloc.h again
> * xen-netback: Add support for multiple queues
> * xen-netback: Factor queue-specific data into queue struct
> * xen-netback: Move grant_copy_op array back into struct xenvif.
> * net: get rid of SET_ETHTOOL_OPS
>
> Interestingly, it takes some time until the bug triggers and the time
> increased when I switched from linux-sunxi to mainline.
>
> Do you have any idea what happens here? I am a bit clueless what's going on.

Me too. Since there are mach_to_phys messages perhaps Stefano (CCd) has
a clue. Original logs are in
http://lists.xen.org/archives/html/xen-users/2014-07/msg00004.html

Lots of these under network load:

[ 189.507495] xen_add_phys_to_mach_entry: cannot add pfn=0x0006930f ->
mfn=0x0004c3bc: pfn=0x00069310 -> mfn=0x0004c3bc already exists
[ 189.531185] xen_add_phys_to_mach_entry: cannot add pfn=0x0006921d ->
mfn=0x0004c489: pfn=0x0006928f -> mfn=0x0004c489 already exists

Ian.


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

Stefano Stabellini-3
On Thu, 3 Jul 2014, Ian Campbell wrote:

> On Thu, 2014-07-03 at 02:01 +0200, Denis Schneider wrote:
> > Hi an,
> > thank you for your reply.
> >
> > 2014-07-02 11:56 GMT+02:00 Ian Campbell <[hidden email]>:
> > > First thing I would recommend would be to try the latest mainline stable
> > > 3.15.x release. I think everything needed for a usable sunxi system is
> > > in there already so no need for the sunxi-devel branch
> >
> > I tried Linus' linux.git/master, which corresponds to 3.16 --
> > resulting in the same messages and panic.
> > Besides that, the mainline kernel works quite well.
> > BTW, git shows that sunxi-devel and mainline Linux v3.15.2 are the
> > same for drivers/net/xen-netback, though linux.git/master shows some
> > changes.
> >
> > The bug can easily be triggered if you access blkback and netback in
> > parallel (thanks to Maximilian), e.g.
> > domU: iperf -s & cat /dev/xvda > /dev/null
> > dom0: iperf -t 3600 -c domU
> >
> > It does not matter if the underlying dom0 block device is on a SATA,
> > USB or mmc device. The panic is similar.
> >
> > > The reason I suggest the latest 3.15.x is that there were a few
> > > interesting netback bugs but I think they've all been backported to
> > > stable by now.
> >
> > I hope that they are all included in linux.git/master @ 16874b2,
> > regarding xen-netback, those changes occurred from sunxi-devel to
> > 16874b2:
> > * xen-netback: bookkeep number of active queues in our own module
> > * net: xen-netback: include linux/vmalloc.h again
> > * xen-netback: Add support for multiple queues
> > * xen-netback: Factor queue-specific data into queue struct
> > * xen-netback: Move grant_copy_op array back into struct xenvif.
> > * net: get rid of SET_ETHTOOL_OPS
> >
> > Interestingly, it takes some time until the bug triggers and the time
> > increased when I switched from linux-sunxi to mainline.
> >
> > Do you have any idea what happens here? I am a bit clueless what's going on.
>
> Me too. Since there are mach_to_phys messages perhaps Stefano (CCd) has
> a clue. Original logs are in
> http://lists.xen.org/archives/html/xen-users/2014-07/msg00004.html
>
> Lots of these under network load:
>
> [ 189.507495] xen_add_phys_to_mach_entry: cannot add pfn=0x0006930f ->
> mfn=0x0004c3bc: pfn=0x00069310 -> mfn=0x0004c3bc already exists
> [ 189.531185] xen_add_phys_to_mach_entry: cannot add pfn=0x0006921d ->
> mfn=0x0004c489: pfn=0x0006928f -> mfn=0x0004c489 already exists

Unfortunately this is a known issue without a proper solution at the
moment.  It is caused by Zoltan's patch series to switch from grant
copies to grant mappings in netfront/netback, that went in Linux
3.15-rc1.  If you use Linux 3.14 instead, you shouldn't have any
problems.

These are the details: ARM support in Linux cannot deal with multiple
foreign mappings of the same physical page. If the frontend decides to
grant the same page twice, using two different grant references, it is
going to cause problems to the accounting in arch/arm/xen/p2m.c on the
backend side. It is not trivial to come up with a solution because the
data structures in p2m.c are already pretty slow as they are, being able
to account for multiple mappings for a single mfn would slow things down
further.

At the moment I would like a way to disable grant mappings and go back
to grant copies on demand. Maybe we could have a feature flag to change
the behaviour of the network backend?

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: [Xen-devel] ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

David Vrabel
On 03/07/14 18:47, Stefano Stabellini wrote:
>
> At the moment I would like a way to disable grant mappings and go back
> to grant copies on demand. Maybe we could have a feature flag to change
> the behaviour of the network backend?

You must fix the ARM code to handle this properly.

blkback also uses grant maps and depending on the frontend blkback may
grant map the same machine frame multiple time.  We have see frontends
that send such requests.

I can't remember how the ARM code works.  Where is the problematic m2p
lookup?  Perhaps this could be avoided for foreign frames?

David

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: [Xen-devel] ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

Stefano Stabellini-3
On Fri, 4 Jul 2014, David Vrabel wrote:

> On 03/07/14 18:47, Stefano Stabellini wrote:
> >
> > At the moment I would like a way to disable grant mappings and go back
> > to grant copies on demand. Maybe we could have a feature flag to change
> > the behaviour of the network backend?
>
> You must fix the ARM code to handle this properly.
>
> blkback also uses grant maps and depending on the frontend blkback may
> grant map the same machine frame multiple time.  We have see frontends
> that send such requests.

Indeed, it must be fixed properly. The workaround of disabling grant
mappings would be just to buy us some time to come up with the fix.


> I can't remember how the ARM code works.  Where is the problematic m2p
> lookup?

arch/arm/xen/p2m.c


> Perhaps this could be avoided for foreign frames?

Unfortunately no. The whole point of p2m.c is to be able to translate
foreign frames for dma operations.

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: [Xen-devel] ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

David Vrabel
On 04/07/14 15:12, Stefano Stabellini wrote:

> On Fri, 4 Jul 2014, David Vrabel wrote:
>> On 03/07/14 18:47, Stefano Stabellini wrote:
>>>
>>> At the moment I would like a way to disable grant mappings and go back
>>> to grant copies on demand. Maybe we could have a feature flag to change
>>> the behaviour of the network backend?
>>
>> You must fix the ARM code to handle this properly.
>>
>> blkback also uses grant maps and depending on the frontend blkback may
>> grant map the same machine frame multiple time.  We have see frontends
>> that send such requests.
>
> Indeed, it must be fixed properly. The workaround of disabling grant
> mappings would be just to buy us some time to come up with the fix.

It's an expensive workaround though.

>> I can't remember how the ARM code works.  Where is the problematic m2p
>> lookup?
>
> arch/arm/xen/p2m.c
>
>
>> Perhaps this could be avoided for foreign frames?
>
> Unfortunately no. The whole point of p2m.c is to be able to translate
> foreign frames for dma operations.

This is a p2m lookup though, which is fine, yes?  Where, specifically is
a mfn_to_pfn() lookup needed for a foreign MFN.

FWIW, On x86, mfn_to_pfn() on a foreign MFN will return an invalid value
(unless the gntdev device made the mapping and the m2p_override is used).

David

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: [Xen-devel] ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

Stefano Stabellini-3
On Fri, 4 Jul 2014, David Vrabel wrote:

> On 04/07/14 15:12, Stefano Stabellini wrote:
> > On Fri, 4 Jul 2014, David Vrabel wrote:
> >> On 03/07/14 18:47, Stefano Stabellini wrote:
> >>>
> >>> At the moment I would like a way to disable grant mappings and go back
> >>> to grant copies on demand. Maybe we could have a feature flag to change
> >>> the behaviour of the network backend?
> >>
> >> You must fix the ARM code to handle this properly.
> >>
> >> blkback also uses grant maps and depending on the frontend blkback may
> >> grant map the same machine frame multiple time.  We have see frontends
> >> that send such requests.
> >
> > Indeed, it must be fixed properly. The workaround of disabling grant
> > mappings would be just to buy us some time to come up with the fix.
>
> It's an expensive workaround though.

In terms of performances or in terms of code?
If it's the performances you are worried about, we could enable the
workaround just for arm and arm64.


> >> I can't remember how the ARM code works.  Where is the problematic m2p
> >> lookup?
> >
> > arch/arm/xen/p2m.c
> >
> >
> >> Perhaps this could be avoided for foreign frames?
> >
> > Unfortunately no. The whole point of p2m.c is to be able to translate
> > foreign frames for dma operations.
>
> This is a p2m lookup though, which is fine, yes?  Where, specifically is
> a mfn_to_pfn() lookup needed for a foreign MFN.
 
drivers/xen/swiotlb-xen.c:xen_unmap_single. xen_bus_to_phys is based on
the value returned by mfn_to_pfn.


> FWIW, On x86, mfn_to_pfn() on a foreign MFN will return an invalid value
> (unless the gntdev device made the mapping and the m2p_override is used).

Keep in mind that ARM domains are a bit like PVH without IOMMU.

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: [Xen-devel] ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

David Vrabel
On 04/07/14 15:31, Stefano Stabellini wrote:

> On Fri, 4 Jul 2014, David Vrabel wrote:
>> On 04/07/14 15:12, Stefano Stabellini wrote:
>>> On Fri, 4 Jul 2014, David Vrabel wrote:
>>>> On 03/07/14 18:47, Stefano Stabellini wrote:
>>>>>
>>>>> At the moment I would like a way to disable grant mappings and go back
>>>>> to grant copies on demand. Maybe we could have a feature flag to change
>>>>> the behaviour of the network backend?
>>>>
>>>> You must fix the ARM code to handle this properly.
>>>>
>>>> blkback also uses grant maps and depending on the frontend blkback may
>>>> grant map the same machine frame multiple time.  We have see frontends
>>>> that send such requests.
>>>
>>> Indeed, it must be fixed properly. The workaround of disabling grant
>>> mappings would be just to buy us some time to come up with the fix.
>>
>> It's an expensive workaround though.
>
> In terms of performances or in terms of code?
> If it's the performances you are worried about, we could enable the
> workaround just for arm and arm64.

Expensive in terms of effort required to implement and maintain.

>>>> I can't remember how the ARM code works.  Where is the problematic m2p
>>>> lookup?
>>>
>>> arch/arm/xen/p2m.c
>>>
>>>
>>>> Perhaps this could be avoided for foreign frames?
>>>
>>> Unfortunately no. The whole point of p2m.c is to be able to translate
>>> foreign frames for dma operations.
>>
>> This is a p2m lookup though, which is fine, yes?  Where, specifically is
>> a mfn_to_pfn() lookup needed for a foreign MFN.
>  
> drivers/xen/swiotlb-xen.c:xen_unmap_single. xen_bus_to_phys is based on
> the value returned by mfn_to_pfn.

I think you can probably get away with not doing the cache operations on
foreign pages when DMA map/unmapping.  DMA mapped foreign pages are
(currently) either:

a) packets from a guest being Tx'd off host.  These are read-only and
are immediately freed after they're tranmitted.

b) block requests from blkback and these pages are never accessed.

David

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: [Xen-devel] ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

David Vrabel
On 04/07/14 17:36, David Vrabel wrote:

> On 04/07/14 15:31, Stefano Stabellini wrote:
>> On Fri, 4 Jul 2014, David Vrabel wrote:
>>> On 04/07/14 15:12, Stefano Stabellini wrote:
>>>> On Fri, 4 Jul 2014, David Vrabel wrote:
>>>>> On 03/07/14 18:47, Stefano Stabellini wrote:
>>>>>>
>>>>>> At the moment I would like a way to disable grant mappings and go back
>>>>>> to grant copies on demand. Maybe we could have a feature flag to change
>>>>>> the behaviour of the network backend?
>>>>>
>>>>> You must fix the ARM code to handle this properly.
>>>>>
>>>>> blkback also uses grant maps and depending on the frontend blkback may
>>>>> grant map the same machine frame multiple time.  We have see frontends
>>>>> that send such requests.
>>>>
>>>> Indeed, it must be fixed properly. The workaround of disabling grant
>>>> mappings would be just to buy us some time to come up with the fix.
>>>
>>> It's an expensive workaround though.
>>
>> In terms of performances or in terms of code?
>> If it's the performances you are worried about, we could enable the
>> workaround just for arm and arm64.
>
> Expensive in terms of effort required to implement and maintain.
>
>>>>> I can't remember how the ARM code works.  Where is the problematic m2p
>>>>> lookup?
>>>>
>>>> arch/arm/xen/p2m.c
>>>>
>>>>
>>>>> Perhaps this could be avoided for foreign frames?
>>>>
>>>> Unfortunately no. The whole point of p2m.c is to be able to translate
>>>> foreign frames for dma operations.
>>>
>>> This is a p2m lookup though, which is fine, yes?  Where, specifically is
>>> a mfn_to_pfn() lookup needed for a foreign MFN.
>>  
>> drivers/xen/swiotlb-xen.c:xen_unmap_single. xen_bus_to_phys is based on
>> the value returned by mfn_to_pfn.
>
> I think you can probably get away with not doing the cache operations on
> foreign pages when DMA map/unmapping.  DMA mapped foreign pages are
> (currently) either:
>
> a) packets from a guest being Tx'd off host.  These are read-only and
> are immediately freed after they're tranmitted.
>
> b) block requests from blkback and these pages are never accessed.

Something like:

--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -90,17 +90,18 @@ static inline dma_addr_t xen_phys_to_bus(phys_addr_t paddr)
  return dma;
 }
 
-static inline phys_addr_t xen_bus_to_phys(dma_addr_t baddr)
+/* Return true if baddr is the address of a foreign frame. */
+static inline int xen_bus_to_phys(dma_addr_t baddr, phys_addr_t *paddr)
 {
  unsigned long pfn = mfn_to_pfn(PFN_DOWN(baddr));
  dma_addr_t dma = (dma_addr_t)pfn << PAGE_SHIFT;
- phys_addr_t paddr = dma;
 
- BUG_ON(paddr != dma); /* truncation has occurred, should never happen */
+ if (pfn == INVALID_P2M_ENTRY)
+ return true;
 
- paddr |= baddr & ~PAGE_MASK;
+ *paddr = dma | (baddr & ~PAGE_MASK);
 
- return paddr;
+ return false;
 }
 
 static inline dma_addr_t xen_virt_to_bus(void *address)
@@ -443,10 +444,30 @@ static void xen_unmap_single(struct device *hwdev, dma_addr_t dev_addr,
      size_t size, enum dma_data_direction dir,
  struct dma_attrs *attrs)
 {
- phys_addr_t paddr = xen_bus_to_phys(dev_addr);
+ phys_addr_t paddr;
+ bool is_foreign;
 
  BUG_ON(dir == DMA_NONE);
 
+ is_foreign = xen_bus_to_phys(dev_addr, &paddr);
+
+ /*
+ * We can't get the appropriate PFN for a foreign frame since
+ * it may be grant mapped multiple times.
+ *
+ * Assume that the grant unmap will do any appropriate cache
+ * operations, and that the frontend will do any for its own
+ * mappings.
+ *
+ * This does mean there is a window between the DMA unmap and
+ * the grant unmap where the CPU may see stale data (for a
+ * FROM_DEVICE operation), but this is not a problem in
+ * practice with the current users of foreign mappings
+ * (netback and blkback).
+ */
+ if (is_foreign)
+ return;
+
  xen_dma_unmap_page(hwdev, paddr, size, dir, attrs);
 
  /* NOTE: We use dev_addr here, not paddr! */

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: [Xen-devel] ARM: "xen_add_mach_to_phys_entry: cannot add ... already exists and panics"

Stefano Stabellini-3
On Fri, 4 Jul 2014, David Vrabel wrote:

> On 04/07/14 17:36, David Vrabel wrote:
> > On 04/07/14 15:31, Stefano Stabellini wrote:
> >> On Fri, 4 Jul 2014, David Vrabel wrote:
> >>> On 04/07/14 15:12, Stefano Stabellini wrote:
> >>>> On Fri, 4 Jul 2014, David Vrabel wrote:
> >>>>> On 03/07/14 18:47, Stefano Stabellini wrote:
> >>>>>>
> >>>>>> At the moment I would like a way to disable grant mappings and go back
> >>>>>> to grant copies on demand. Maybe we could have a feature flag to change
> >>>>>> the behaviour of the network backend?
> >>>>>
> >>>>> You must fix the ARM code to handle this properly.
> >>>>>
> >>>>> blkback also uses grant maps and depending on the frontend blkback may
> >>>>> grant map the same machine frame multiple time.  We have see frontends
> >>>>> that send such requests.
> >>>>
> >>>> Indeed, it must be fixed properly. The workaround of disabling grant
> >>>> mappings would be just to buy us some time to come up with the fix.
> >>>
> >>> It's an expensive workaround though.
> >>
> >> In terms of performances or in terms of code?
> >> If it's the performances you are worried about, we could enable the
> >> workaround just for arm and arm64.
> >
> > Expensive in terms of effort required to implement and maintain.
> >
> >>>>> I can't remember how the ARM code works.  Where is the problematic m2p
> >>>>> lookup?
> >>>>
> >>>> arch/arm/xen/p2m.c
> >>>>
> >>>>
> >>>>> Perhaps this could be avoided for foreign frames?
> >>>>
> >>>> Unfortunately no. The whole point of p2m.c is to be able to translate
> >>>> foreign frames for dma operations.
> >>>
> >>> This is a p2m lookup though, which is fine, yes?  Where, specifically is
> >>> a mfn_to_pfn() lookup needed for a foreign MFN.
> >>  
> >> drivers/xen/swiotlb-xen.c:xen_unmap_single. xen_bus_to_phys is based on
> >> the value returned by mfn_to_pfn.
> >
> > I think you can probably get away with not doing the cache operations on
> > foreign pages when DMA map/unmapping.  DMA mapped foreign pages are
> > (currently) either:
> >
> > a) packets from a guest being Tx'd off host.  These are read-only and
> > are immediately freed after they're tranmitted.
> >
> > b) block requests from blkback and these pages are never accessed.

Unfortunately it doesn't look like it is possible to skip the call to
__dma_page_dev_to_cpu
(xen_dma_unmap_page->arm_dma_unmap_page->__dma_page_dev_to_cpu), because
otherwise the data written by DMA to system memory won't be visible to
the CPU, at least on ARM32.

In fact the grant_unmap operation might issue flushes, but they are
inner flushes. While here we would need to issue an outer flush at the
very least. In addition to it, __dma_page_dev_to_cpu also calls
dmac_unmap_area, that on v7 becomes v7_dma_inv_range that issues a bunch
of DCCIMVAC.

So in order for this to work on grant_unmap Xen would have to:

- issue an outer flush
- DCCIMVAC the page

Also not all devices need this, only the non-coherent ones.

Given that we assume that Dom0 is mapped 1:1, one way to solve this
would be for Dom0 to map the grant on PFN=MFN too, then issue all the
required flushed on the machine address instead of the physical address.



> Something like:
>
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -90,17 +90,18 @@ static inline dma_addr_t xen_phys_to_bus(phys_addr_t paddr)
>   return dma;
>  }
>  
> -static inline phys_addr_t xen_bus_to_phys(dma_addr_t baddr)
> +/* Return true if baddr is the address of a foreign frame. */
> +static inline int xen_bus_to_phys(dma_addr_t baddr, phys_addr_t *paddr)
>  {
>   unsigned long pfn = mfn_to_pfn(PFN_DOWN(baddr));
>   dma_addr_t dma = (dma_addr_t)pfn << PAGE_SHIFT;
> - phys_addr_t paddr = dma;
>  
> - BUG_ON(paddr != dma); /* truncation has occurred, should never happen */
> + if (pfn == INVALID_P2M_ENTRY)
> + return true;
>  
> - paddr |= baddr & ~PAGE_MASK;
> + *paddr = dma | (baddr & ~PAGE_MASK);
>  
> - return paddr;
> + return false;
>  }
>  
>  static inline dma_addr_t xen_virt_to_bus(void *address)
> @@ -443,10 +444,30 @@ static void xen_unmap_single(struct device *hwdev, dma_addr_t dev_addr,
>       size_t size, enum dma_data_direction dir,
>   struct dma_attrs *attrs)
>  {
> - phys_addr_t paddr = xen_bus_to_phys(dev_addr);
> + phys_addr_t paddr;
> + bool is_foreign;
>  
>   BUG_ON(dir == DMA_NONE);
>  
> + is_foreign = xen_bus_to_phys(dev_addr, &paddr);
> +
> + /*
> + * We can't get the appropriate PFN for a foreign frame since
> + * it may be grant mapped multiple times.
> + *
> + * Assume that the grant unmap will do any appropriate cache
> + * operations, and that the frontend will do any for its own
> + * mappings.
> + *
> + * This does mean there is a window between the DMA unmap and
> + * the grant unmap where the CPU may see stale data (for a
> + * FROM_DEVICE operation), but this is not a problem in
> + * practice with the current users of foreign mappings
> + * (netback and blkback).
> + */
> + if (is_foreign)
> + return;
> +
>   xen_dma_unmap_page(hwdev, paddr, size, dir, attrs);
>  
>   /* NOTE: We use dev_addr here, not paddr! */
>

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users