PCI Passthrough - Write-back to unknown field

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

PCI Passthrough - Write-back to unknown field

Ivan Radevici

Hi all!


I'm trying to passthrough a NI PCI-6025E card from Xen version 4.8.0 (Ubuntu 4.8.0-1ubuntu2.2) to HVM Windows 2000. The pass through itself is done accordingly to documentation. Guest OS determines the card as unknown PCI device. Installing the driver allows to recognize device as Data Acquisition Device / PCI-6028E in Windows Device Manager and in NI software. However, running tests from NI test panels fails with "The device is not responding to the selected base address" (in fact it seems to be "overFlowError" with detailed description "Because of system and/or bus-bandwidth limitations, the driver could not read data from the device fast enough to keep up with the device throughput; the onboard device memory reported an overflow error"). QEMU log file contains an error "Write-back to unknown field 0x08 (partially) inhibited (0x000000), If the device doesn't work, try enabling permissive mode (unsafe) and if it helps report the problem to xen-devel". Adding permissive=1 to the pci section of the domain configuration file makes the error from the QEMU log disappear, but the card itself still can not pass the test in the guest system.


I don't have a deep knowledge in device passthroughing, but as far as I understand passthrough means that everything from the virtual machine is just passed to the real one. This leads to a conclusion that the main trouble is to pass the device to the guest, once device is seen by the guest everything should be fine. Am I understanding something wrong?

The device shows in xl pci-assignable-list in Dom0
Relevant part from xl dmesg


(XEN) Initing memory sharing.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 2 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation not enabled.
(XEN) Intel VT-d Interrupt Remapping not enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping disabled


Can you help me to solve/debug this issue, please? Any ideas, guesses or suggestions are welcome.

Thank you in advance


Ivan


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PCI Passthrough - Write-back to unknown field

Roger Pau Monné-3
On Thu, Jul 27, 2017 at 10:36:04AM +0300, Ivan Radevici wrote:

> Hi all!
>
>
> I'm trying to passthrough a NI PCI-6025E card from Xen version 4.8.0 (Ubuntu
> 4.8.0-1ubuntu2.2) to HVM Windows 2000. The pass through itself is done
> accordingly to documentation. Guest OS determines the card as unknown PCI
> device. Installing the driver allows to recognize device as Data Acquisition
> Device / PCI-6028E in Windows Device Manager and in NI software. However,
> running tests from NI test panels fails with "The device is not responding
> to the selected base address" (in fact it seems to be "overFlowError" with
> detailed description "Because of system and/or bus-bandwidth limitations,
> the driver could not read data from the device fast enough to keep up with
> the device throughput; the onboard device memory reported an overflow
> error"). QEMU log file contains an error "Write-back to unknown field 0x08
> (partially) inhibited (0x000000), If the device doesn't work, try enabling
> permissive mode (unsafe) and if it helps report the problem to xen-devel".
> Adding permissive=1 to the pci section of the domain configuration file
> makes the error from the QEMU log disappear, but the card itself still can
> not pass the test in the guest system.
>
>
> I don't have a deep knowledge in device passthroughing, but as far as I
> understand passthrough means that everything from the virtual machine is
> just passed to the real one. This leads to a conclusion that the main
> trouble is to pass the device to the guest, once device is seen by the guest
> everything should be fine. Am I understanding something wrong?
>
> The device shows in xl pci-assignable-list in Dom0
> Relevant part from xl dmesg
>
>
> (XEN) Initing memory sharing.
> (XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
> (XEN) Intel VT-d iommu 2 supported page sizes: 4kB.
> (XEN) Intel VT-d Snoop Control not enabled.
> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
> (XEN) Intel VT-d Queued Invalidation not enabled.
> (XEN) Intel VT-d Interrupt Remapping not enabled.
> (XEN) Intel VT-d Posted Interrupt not enabled.
> (XEN) Intel VT-d Shared EPT tables not enabled.
> (XEN) I/O virtualisation enabled
> (XEN)  - Dom0 mode: Relaxed
> (XEN) Interrupt remapping disabled
>
>
> Can you help me to solve/debug this issue, please? Any ideas, guesses or
> suggestions are welcome.

I would say that your card generates data faster that what the DomU
can extract, so the buffer gets full.

Have you tried pinning the DomU with the card to specific physical
CPUs, and making sure that no other VM is allowed to run there?

For example, provided your box has 4 physical CPUs, and you would like
to give 2 cpus to Dom0 and 2 cpus to the DomU:

xl vcpu-pin Dom0 0 0
xl vcpu-pin Dom0 1 1

xl vcpu-pin DomU 0 2
xl vcpu-pin DomU 1 3

Note that you should create both Dom0 and DomU with only two vcpus.

Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PCI Passthrough - Write-back to unknown field

Ivan Radevici
Thank you for your reply!

Here is the output of vcpu-list
sudo xl vcpu-list
Name                                ID  VCPU   CPU State   Time(s)
Affinity (Hard / Soft)
Domain-0                             0     0    0   -b-      75.7 0 / all
Domain-0                             0     1    1   r--      76.8 1 / all
win2k                                1     0    2   -b-      78.5 2 / all
win2k                                1     1    3   -b-      69.2 3 / all

but the card still fails the test. Also, it looks like cores are mainly
in idle, so probably problem is not there

While I was trying to pin vcpus I remembered about PV on HVM. There was
a chart somewhere in wiki, where it was shown that interrupts and timers
have poor performance on fully virtualised guests. This also may be a
cause for this behaviour. I am thinking about trying PVHVM (or just
simple PV) drivers for Windows2000, if I'll find some. How do you think,
may it help?

Any other options?

Thanks once again

On 27.07.2017 11:48, Roger Pau Monné wrote:

> On Thu, Jul 27, 2017 at 10:36:04AM +0300, Ivan Radevici wrote:
>> Hi all!
>>
>>
>> I'm trying to passthrough a NI PCI-6025E card from Xen version 4.8.0 (Ubuntu
>> 4.8.0-1ubuntu2.2) to HVM Windows 2000. The pass through itself is done
>> accordingly to documentation. Guest OS determines the card as unknown PCI
>> device. Installing the driver allows to recognize device as Data Acquisition
>> Device / PCI-6028E in Windows Device Manager and in NI software. However,
>> running tests from NI test panels fails with "The device is not responding
>> to the selected base address" (in fact it seems to be "overFlowError" with
>> detailed description "Because of system and/or bus-bandwidth limitations,
>> the driver could not read data from the device fast enough to keep up with
>> the device throughput; the onboard device memory reported an overflow
>> error"). QEMU log file contains an error "Write-back to unknown field 0x08
>> (partially) inhibited (0x000000), If the device doesn't work, try enabling
>> permissive mode (unsafe) and if it helps report the problem to xen-devel".
>> Adding permissive=1 to the pci section of the domain configuration file
>> makes the error from the QEMU log disappear, but the card itself still can
>> not pass the test in the guest system.
>>
>>
>> I don't have a deep knowledge in device passthroughing, but as far as I
>> understand passthrough means that everything from the virtual machine is
>> just passed to the real one. This leads to a conclusion that the main
>> trouble is to pass the device to the guest, once device is seen by the guest
>> everything should be fine. Am I understanding something wrong?
>>
>> The device shows in xl pci-assignable-list in Dom0
>> Relevant part from xl dmesg
>>
>>
>> (XEN) Initing memory sharing.
>> (XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
>> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
>> (XEN) Intel VT-d iommu 2 supported page sizes: 4kB.
>> (XEN) Intel VT-d Snoop Control not enabled.
>> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
>> (XEN) Intel VT-d Queued Invalidation not enabled.
>> (XEN) Intel VT-d Interrupt Remapping not enabled.
>> (XEN) Intel VT-d Posted Interrupt not enabled.
>> (XEN) Intel VT-d Shared EPT tables not enabled.
>> (XEN) I/O virtualisation enabled
>> (XEN)  - Dom0 mode: Relaxed
>> (XEN) Interrupt remapping disabled
>>
>>
>> Can you help me to solve/debug this issue, please? Any ideas, guesses or
>> suggestions are welcome.
> I would say that your card generates data faster that what the DomU
> can extract, so the buffer gets full.
>
> Have you tried pinning the DomU with the card to specific physical
> CPUs, and making sure that no other VM is allowed to run there?
>
> For example, provided your box has 4 physical CPUs, and you would like
> to give 2 cpus to Dom0 and 2 cpus to the DomU:
>
> xl vcpu-pin Dom0 0 0
> xl vcpu-pin Dom0 1 1
>
> xl vcpu-pin DomU 0 2
> xl vcpu-pin DomU 1 3
>
> Note that you should create both Dom0 and DomU with only two vcpus.
>
> Roger.


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PCI Passthrough - Write-back to unknown field

Roger Pau Monné-3
On Thu, Jul 27, 2017 at 01:38:36PM +0300, Ivan Radevici wrote:

> Thank you for your reply!
>
> Here is the output of vcpu-list
> sudo xl vcpu-list
> Name                                ID  VCPU   CPU State   Time(s) Affinity
> (Hard / Soft)
> Domain-0                             0     0    0   -b-      75.7 0 / all
> Domain-0                             0     1    1   r--      76.8 1 / all
> win2k                                1     0    2   -b-      78.5 2 / all
> win2k                                1     1    3   -b-      69.2 3 / all
>
> but the card still fails the test. Also, it looks like cores are mainly in
> idle, so probably problem is not there
>
> While I was trying to pin vcpus I remembered about PV on HVM. There was a
> chart somewhere in wiki, where it was shown that interrupts and timers have
> poor performance on fully virtualised guests. This also may be a cause for
> this behaviour. I am thinking about trying PVHVM (or just simple PV) drivers
> for Windows2000, if I'll find some. How do you think, may it help?

There's no such thing as PVHVM for Windows, this is only available for
Linux/FreeBSD. AFAIK on Windows you can only add the PV drivers for
block and network. Might be worth a try, but I doubt it's going to fix
your PCI card issues.

> Any other options?

Try on another box maybe? Your box seems to be missing quite a lot of
IOMMU features, so I guess it's kind of oldish hardware?

Maybe on newer hardware you will no longer see the timeouts, although
it's very hard to tell what's going on exactly without knowing what
the driver is actually doing (or why it's failing exactly).

Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PCI Passthrough - Write-back to unknown field

Ivan Radevici
 
>>On Thu, Jul 27, 2017 at 01:38:36PM +0300, Ivan Radevici wrote:
>> Thank you for your reply!
>>
>> Here is the output of vcpu-list
>> sudo xl vcpu-list
>> Name                                ID  VCPU   CPU State   Time(s) Affinity
>> (Hard / Soft)
>> Domain-0                             0     0    0   -b-      75.7 0 / all
>> Domain-0                             0     1    1   r--      76.8 1 / all
>> win2k                                1     0    2   -b-      78.5 2 / all
>> win2k                                1     1    3   -b-      69.2 3 / all
>>
>> but the card still fails the test. Also, it looks like cores are mainly in
>> idle, so probably problem is not there
>>
> > While I was trying to pin vcpus I remembered about PV on HVM. There was a
>> chart somewhere in wiki, where it was shown that interrupts and timers have
>> poor performance on fully virtualised guests. This also may be a cause for
>> this behaviour. I am thinking about trying PVHVM (or just simple PV) drivers
>> for Windows2000, if I'll find some. How do you think, may it help?
>
>There's no such thing as PVHVM for Windows, this is only available for
>Linux/FreeBSD. AFAIK on Windows you can only add the PV drivers for
>block and network. Might be worth a try, but I doubt it's going to fix
>your PCI card issues.

I am not an expert in it, so I may be completely wrong, but at least there is a xenbus
driver, which "attaches to a virtual device on the PCI bus and provides child devices
for the other paravirtual device drivers to attach to." I understand that this will provide
interface only for the other virtual devices, but may be it will also somehow help.

>
>> Any other options?
>
>Try on another box maybe? Your box seems to be missing quite a lot of
>IOMMU features, so I guess it's kind of oldish hardware?
>

Yes, it is not the new one, but definitely not that old, to allow installation of
Win2k without virtualization

>
>Maybe on newer hardware you will no longer see the timeouts, although
>it's very hard to tell what's going on exactly without knowing what
>the driver is actually doing (or why it's failing exactly).
>
>Roger.

I already wrote to the NI support asking, what can be the cause of the error,
but still no response. Most probably, they'll just tell me that their cards
do not support virtualization

Ivan

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xen.org/xen-users
Loading...