IOMMU group dissapear in XEN

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

IOMMU group dissapear in XEN

Ivan Radevici

Hi all,


I faced a really strange thing I would like to understand. I'm trying to passthroug a PCI device and tusing arch linux (4.12.3-1-ARCH) for it. When I boot system with the native kernel in kernel.log I can see the following


     [    0.000000] DMAR: IOMMU enabled
     [    0.441136] DMAR: Disabling IOMMU for graphics on this chipset
     [    0.575144] iommu: Adding device 0000:00:00.0 to group 0
     [    0.575159] iommu: Adding device 0000:00:01.0 to group 1
     [    0.575185] iommu: Adding device 0000:00:03.0 to group 2
     [    0.575195] iommu: Adding device 0000:00:03.2 to group 2
     [    0.575205] iommu: Adding device 0000:00:03.3 to group 2
     [    0.575216] iommu: Adding device 0000:00:19.0 to group 3
     [    0.575245] iommu: Adding device 0000:00:1a.0 to group 4
     [    0.575257] iommu: Adding device 0000:00:1a.1 to group 4
     [    0.575267] iommu: Adding device 0000:00:1a.2 to group 4
     [    0.575278] iommu: Adding device 0000:00:1a.7 to group 4
     [    0.575288] iommu: Adding device 0000:00:1b.0 to group 5
     [    0.575318] iommu: Adding device 0000:00:1d.0 to group 6
     [    0.575329] iommu: Adding device 0000:00:1d.1 to group 6
     [    0.575340] iommu: Adding device 0000:00:1d.2 to group 6
     [    0.575353] iommu: Adding device 0000:00:1d.7 to group 6
     [    0.575364] iommu: Adding device 0000:00:1e.0 to group 7
     [    0.575389] iommu: Adding device 0000:00:1f.0 to group 8
     [    0.575401] iommu: Adding device 0000:00:1f.2 to group 8
     [    0.575413] iommu: Adding device 0000:00:1f.3 to group 8
     [    0.575421] iommu: Adding device 0000:01:00.0 to group 1
     [    0.575425] iommu: Adding device 0000:11:00.0 to group 7

when I boot with xen then log shows just

     [    0.000000] DMAR: IOMMU enabled
     [    0.173434] DMAR: Disabling IOMMU for graphics on this chipset

Respectively, with the native kernel there are some iommu groups in /sys/kernel/iommu_groups, but for xen this directory is empty. As I understand absence of the iommu groups means that I can not really pass throug my PCI. Am I right about it?

     xl dmesg
     (XEN) Xen version 4.9.0 (epitt@localdomain) (gcc (GCC) 7.1.1 20170630) debug=n  Thu Aug  3 18:54:42 EEST 2017
     (XEN) Latest ChangeSet:
     (XEN) Bootloader: GRUB 2.02
     (XEN) Command line: /boot/xen-4.9.0.gz xsave=1 intel_iommu=on
...
     (XEN) Initing memory sharing.
     (XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
     (XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
     (XEN) Intel VT-d iommu 2 supported page sizes: 4kB.
     (XEN) Intel VT-d Snoop Control not enabled.
     (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
     (XEN) Intel VT-d Queued Invalidation not enabled.
     (XEN) Intel VT-d Interrupt Remapping not enabled.
     (XEN) Intel VT-d Posted Interrupt not enabled.
     (XEN) Intel VT-d Shared EPT tables not enabled.
     (XEN) I/O virtualisation enabled
     (XEN)  - Dom0 mode: Relaxed
     (XEN) Interrupt remapping disabled
     (XEN) ENABLING IO-APIC IRQs
     (XEN)  -> Using new ACK method

Kernel command line is the same for the both boots (/boot/vmlinuz-linux root=UUID=xxxxxx rw quiet intel_iommu=on console=tty0).

So why iommu groups dissapear in xen?

Thank you in advance for any ideas or suggestions


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: IOMMU group dissapear in XEN

Roger Pau Monné-3
On Thu, Aug 03, 2017 at 06:46:05PM +0000, Radevici Ivan wrote:

> Hi all,
>
>
> I faced a really strange thing I would like to understand. I'm trying to passthroug a PCI device and tusing arch linux (4.12.3-1-ARCH) for it. When I boot system with the native kernel in kernel.log I can see the following
>
>
>      [    0.000000] DMAR: IOMMU enabled
>      [    0.441136] DMAR: Disabling IOMMU for graphics on this chipset
>      [    0.575144] iommu: Adding device 0000:00:00.0 to group 0
>      [    0.575159] iommu: Adding device 0000:00:01.0 to group 1
>      [    0.575185] iommu: Adding device 0000:00:03.0 to group 2
>      [    0.575195] iommu: Adding device 0000:00:03.2 to group 2
>      [    0.575205] iommu: Adding device 0000:00:03.3 to group 2
>      [    0.575216] iommu: Adding device 0000:00:19.0 to group 3
>      [    0.575245] iommu: Adding device 0000:00:1a.0 to group 4
>      [    0.575257] iommu: Adding device 0000:00:1a.1 to group 4
>      [    0.575267] iommu: Adding device 0000:00:1a.2 to group 4
>      [    0.575278] iommu: Adding device 0000:00:1a.7 to group 4
>      [    0.575288] iommu: Adding device 0000:00:1b.0 to group 5
>      [    0.575318] iommu: Adding device 0000:00:1d.0 to group 6
>      [    0.575329] iommu: Adding device 0000:00:1d.1 to group 6
>      [    0.575340] iommu: Adding device 0000:00:1d.2 to group 6
>      [    0.575353] iommu: Adding device 0000:00:1d.7 to group 6
>      [    0.575364] iommu: Adding device 0000:00:1e.0 to group 7
>      [    0.575389] iommu: Adding device 0000:00:1f.0 to group 8
>      [    0.575401] iommu: Adding device 0000:00:1f.2 to group 8
>      [    0.575413] iommu: Adding device 0000:00:1f.3 to group 8
>      [    0.575421] iommu: Adding device 0000:01:00.0 to group 1
>      [    0.575425] iommu: Adding device 0000:11:00.0 to group 7
>
>
> when I boot with xen then log shows just
>
>      [    0.000000] DMAR: IOMMU enabled
>      [    0.173434] DMAR: Disabling IOMMU for graphics on this chipset
>
> Respectively, with the native kernel there are some iommu groups in /sys/kernel/iommu_groups, but for xen this directory is empty. As I understand absence of the iommu groups means that I can not really pass throug my PCI. Am I right about it?

There is no /sys/kernel/iommu_groups because the IOMMU is used by Xen,
and thus Linux is not allowed to use it. Xen hides the IOMMU to Dom0.

From the Xen dmesg you pasted below I think you should be able to do
pci-passthrough just fine. Have you actually tried?

>      xl dmesg
>      (XEN) Xen version 4.9.0 (epitt@localdomain) (gcc (GCC) 7.1.1 20170630) debug=n  Thu Aug  3 18:54:42 EEST 2017
>      (XEN) Latest ChangeSet:
>      (XEN) Bootloader: GRUB 2.02
>      (XEN) Command line: /boot/xen-4.9.0.gz xsave=1 intel_iommu=on
> ...
>      (XEN) Initing memory sharing.
>      (XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
>      (XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
>      (XEN) Intel VT-d iommu 2 supported page sizes: 4kB.
>      (XEN) Intel VT-d Snoop Control not enabled.
>      (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
>      (XEN) Intel VT-d Queued Invalidation not enabled.
>      (XEN) Intel VT-d Interrupt Remapping not enabled.
>      (XEN) Intel VT-d Posted Interrupt not enabled.
>      (XEN) Intel VT-d Shared EPT tables not enabled.
>      (XEN) I/O virtualisation enabled
>      (XEN)  - Dom0 mode: Relaxed
>      (XEN) Interrupt remapping disabled
>      (XEN) ENABLING IO-APIC IRQs
>      (XEN)  -> Using new ACK method

Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: IOMMU group dissapear in XEN

Ivan Radevici
Thanks for your answer.

The question appeared just because I tried it and it doesn't work as should. I can passthrough the card, it can be seen from the guest, I even can install the driver for it, but it fails to pass tests... I thought that it can be caused by this issue with the IOMMU groups

Ivan

On Thu, Aug 03, 2017 at 06:46:05PM +0000, Radevici Ivan wrote:
> Hi all,
>
>
> I faced a really strange thing I would like to understand. I'm trying to passthroug a PCI device and tusing arch linux (4.12.3-1-ARCH) for it. When I boot system with the native kernel in kernel.log I can see the following
>
>
>      [    0.000000] DMAR: IOMMU enabled
>      [    0.441136] DMAR: Disabling IOMMU for graphics on this chipset
>      [    0.575144] iommu: Adding device 0000:00:00.0 to group 0
>      [    0.575159] iommu: Adding device 0000:00:01.0 to group 1
>      [    0.575185] iommu: Adding device 0000:00:03.0 to group 2
>      [    0.575195] iommu: Adding device 0000:00:03.2 to group 2
>      [    0.575205] iommu: Adding device 0000:00:03.3 to group 2
>      [    0.575216] iommu: Adding device 0000:00:19.0 to group 3
>      [    0.575245] iommu: Adding device 0000:00:1a.0 to group 4
>      [    0.575257] iommu: Adding device 0000:00:1a.1 to group 4
>      [    0.575267] iommu: Adding device 0000:00:1a.2 to group 4
>      [    0.575278] iommu: Adding device 0000:00:1a.7 to group 4
>      [    0.575288] iommu: Adding device 0000:00:1b.0 to group 5
>      [    0.575318] iommu: Adding device 0000:00:1d.0 to group 6
>      [    0.575329] iommu: Adding device 0000:00:1d.1 to group 6
>      [    0.575340] iommu: Adding device 0000:00:1d.2 to group 6
>      [    0.575353] iommu: Adding device 0000:00:1d.7 to group 6
>      [    0.575364] iommu: Adding device 0000:00:1e.0 to group 7
>      [    0.575389] iommu: Adding device 0000:00:1f.0 to group 8
>      [    0.575401] iommu: Adding device 0000:00:1f.2 to group 8
>      [    0.575413] iommu: Adding device 0000:00:1f.3 to group 8
>      [    0.575421] iommu: Adding device 0000:01:00.0 to group 1
>      [    0.575425] iommu: Adding device 0000:11:00.0 to group 7
>
>
> when I boot with xen then log shows just
>
>      [    0.000000] DMAR: IOMMU enabled
>      [    0.173434] DMAR: Disabling IOMMU for graphics on this chipset
>
> Respectively, with the native kernel there are some iommu groups in /sys/kernel/iommu_groups, but for xen this directory is empty. As I understand absence of the iommu groups means that I can not really pass throug my PCI. Am I right about it?

There is no /sys/kernel/iommu_groups because the IOMMU is used by Xen,
and thus Linux is not allowed to use it. Xen hides the IOMMU to Dom0.

From the Xen dmesg you pasted below I think you should be able to do
pci-passthrough just fine. Have you actually tried?

>      xl dmesg
>      (XEN) Xen version 4.9.0 (epitt@localdomain) (gcc (GCC) 7.1.1 20170630) debug=n  Thu Aug  3 18:54:42 EEST 2017
>      (XEN) Latest ChangeSet:
>      (XEN) Bootloader: GRUB 2.02
>      (XEN) Command line: /boot/xen-4.9.0.gz xsave=1 intel_iommu=on
> ...
>      (XEN) Initing memory sharing.
>      (XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
>      (XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
>      (XEN) Intel VT-d iommu 2 supported page sizes: 4kB.
>      (XEN) Intel VT-d Snoop Control not enabled.
>      (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
>      (XEN) Intel VT-d Queued Invalidation not enabled.
>      (XEN) Intel VT-d Interrupt Remapping not enabled.
>      (XEN) Intel VT-d Posted Interrupt not enabled.
>      (XEN) Intel VT-d Shared EPT tables not enabled.
>      (XEN) I/O virtualisation enabled
>      (XEN)  - Dom0 mode: Relaxed
>      (XEN) Interrupt remapping disabled
>      (XEN) ENABLING IO-APIC IRQs
>      (XEN)  -> Using new ACK method

Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: IOMMU group dissapear in XEN

Roger Pau Monné-3
On Tue, Aug 29, 2017 at 09:49:42AM +0000, Radevici Ivan wrote:
> Thanks for your answer.
>
> The question appeared just because I tried it and it doesn't work as should. I can passthrough the card, it can be seen from the guest, I even can install the driver for it, but it fails to pass tests... I thought that it can be caused by this issue with the IOMMU groups

Which kind of tests?

Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: IOMMU group dissapear in XEN

Ivan Radevici
In reply to this post by Ivan Radevici


>>On Tue, Aug 29, 2017 at 09:49:42AM +0000, Radevici Ivan wrote:
>>Thanks for your answer.
>>The question appeared just because I tried it and it doesn't work as should. I can passthrough the card, it can be seen from the guest, I even can install the driver for it, >>but it fails to pass tests... I thought that it can be caused by this issue with the IOMMU groups
>Which kind of tests?

>Roger.
I am trying to pass National Instruments card, it has its own software for testing

Ivan

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users