Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Kuba
Dear List,

I am trying to set up a following configuration:
1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
from sources,
2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
using VT-d, exporting block devices via iSCSI to other VMs and physical
machines,
3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
4000) installed on a block device exported from the storage VM (target
on the storage VM, initiator on dom0).

Everything works perfectly (including PCI & GPU passthrough) until I
install GPLPV drivers on the Windows VM. After driver installation,
Windows needs to reboot, boots fine, displays a message that PV SCSI
drivers were installed and needs to reboot again, and then cannot boot.
Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
sometimes BSODs with "unmountable boot volume" message. All of the
following I tried without GPU passthrough to narrow down the problem.

The intriguing part is this:

1. If the storage VM's OS is Linux - it fails with the above symptoms.
2. If the block devices for the storage VM come directly from dom0 (not
via pci-passthrough) - it fails.
2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
9.2-GENERIC) - it all works.
3. If the storage VM's OS is Linux with kernel compiled without Xen
guest support - it works, but is unstable (see below).
4. If the iSCSI target is on a different physical machine - it all works.
5. If the iSCSI target is on dom0 itself - it works.
6. If I attach the AHCI controller to the Windows VM and install
directly on the hard drive - it works.
7. If the block device for Windows VM is a disk, partition, file, LVM
volume or even a ZoL's zvol (and it comes from a dom0 itself, without
iSCSI)- it works.

If I install Windows and the GPLPV drivers on a hard drive attached to
dom0, Windows + GPLPV work perfectly. If I then give the same hard drive
as a block device to the storage VM and re-export it through iSCSI,
Windows usually boots fine, but works unstable. And by unstable I mean
random read/write errors, sometimes programs won't start, ntdll.dll
crashes, and after couple reboots Windows won't boot (just like
mentioned above).

The configurations I would like to achieve makes sense only with PV
drivers on both storage and Windows VM. All of the "components" seem to
work perfectly until all put together, so I am not really sure where the
problem is.

I would be very grateful for any suggestions or ideas that could
possibly help to narrow down the problem. Maybe I am just doing
something wrong (I hope so). Or maybe there is a bug that shows itself
only in such a particular configuration (hope not)?

Yours faithfully,
Kuba



Additional info:

I tried using couple kernels for dom0 (3.2.51, 3.10.x, 3.12.x packaged
and self-compiled). I tried using different versions of open-iscsi and
iscsi target, no luck.

I tried different gplpv drivers:
gplpv_Vista2008x64_0.11.0.372.msi from univention.de
gplpv_Vista2008x64_0.11.0.372.msi from meadowcourt.org
gplpv_Vista2008x64_0.11.0.418.msi from meadowcourt.org
gplpv_Vista2008x64_1.0.1089.msi   from www.ejbdigital.com.au
gplpv_Vista2008x64_1.0.1092.9.msi from www.ejbdigital.com.au

I am working mainly on Xen 4.3.1, but also tried with Xen 4.2.3 and
4.4-RC2 with no noticeable change in behaviour.

I tried using qemu-traditional for both the storage VM and Windows VM,
no luck.

I reinstalled Windows more times than during the XP and 98SE days
combined :)

Typical Storage VM config file:
name='fbsd'
builder='hvm'
vcpus=8
memory=8192
disk=[
'phy:/dev/sdc,xvda,w',
'file:/mnt/fbsd.iso,hdc,r,devtype=cdrom'
]
vif=[
'bridge=xenbr0,mac=00:16:3e:14:b1:1a'
]
boot='c'
pae=1
nx=1
videoram=16
stdvga=1
sdl=0
vnc=1
vnclisten="0.0.0.0"
usb=1
usbdevice="tablet"
localtime=1
xen_platform_pci=1


Typical Windows VM config file:
name='win'
builder='hvm'
vcpus=8
memory=2048
disk=[
'phy:/dev/sdd,hda,w',
'file:/mnt/win.iso,hdc,r,devtype=cdrom'
]
vif=[
'bridge=xenbr0,mac=00:16:3e:14:b1:fc'
]
boot='c'
pae=1
nx=1
videoram=16
stdvga=1
sdl=0
vnc=1
vnclisten="0.0.0.0"
usb=1
usbdevice="tablet"
localtime=1
xen_platform_pci=1
viridian=1

Hardware: Intel S1200BTS, Xeon E3-1230, 16GB ECC RAM.

xl info with dom0 only:
host                   : ws
release                : 3.10.28custom1
version                : #1 SMP Wed Jan 29 20:53:38 CET 2014
machine                : x86_64
nr_cpus                : 8
max_cpu_id             : 127
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 3192
hw_caps                :
bfebfbff:28100800:00000000:00003f00:17bae3ff:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 16108
free_memory            : 13843
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 3
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder dom0_mem=2G
cc_compiler            : gcc (Debian 4.7.2-5) 4.7.2
cc_compile_by          : root
cc_compile_domain      : localdomain
cc_compile_date        : Wed Jan 29 21:53:06 CET 2014
xend_config_format     : 4

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Adam Goryachev-3
On 31/01/14 00:50, Kuba wrote:

> Dear List,
>
> I am trying to set up a following configuration:
> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
> from sources,
> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
> using VT-d, exporting block devices via iSCSI to other VMs and
> physical machines,
> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
> 4000) installed on a block device exported from the storage VM (target
> on the storage VM, initiator on dom0).
>
> Everything works perfectly (including PCI & GPU passthrough) until I
> install GPLPV drivers on the Windows VM. After driver installation,
> Windows needs to reboot, boots fine, displays a message that PV SCSI
> drivers were installed and needs to reboot again, and then cannot
> boot. Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
> sometimes BSODs with "unmountable boot volume" message. All of the
> following I tried without GPU passthrough to narrow down the problem.
>
> The intriguing part is this:
>
> 1. If the storage VM's OS is Linux - it fails with the above symptoms.
> 2. If the block devices for the storage VM come directly from dom0
> (not via pci-passthrough) - it fails.
> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
> 9.2-GENERIC) - it all works.
> 3. If the storage VM's OS is Linux with kernel compiled without Xen
> guest support - it works, but is unstable (see below).
> 4. If the iSCSI target is on a different physical machine - it all works.
> 5. If the iSCSI target is on dom0 itself - it works.
> 6. If I attach the AHCI controller to the Windows VM and install
> directly on the hard drive - it works.
> 7. If the block device for Windows VM is a disk, partition, file, LVM
> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
> iSCSI)- it works.
>
> If I install Windows and the GPLPV drivers on a hard drive attached to
> dom0, Windows + GPLPV work perfectly. If I then give the same hard
> drive as a block device to the storage VM and re-export it through
> iSCSI, Windows usually boots fine, but works unstable. And by unstable
> I mean random read/write errors, sometimes programs won't start,
> ntdll.dll crashes, and after couple reboots Windows won't boot (just
> like mentioned above).
>
> The configurations I would like to achieve makes sense only with PV
> drivers on both storage and Windows VM. All of the "components" seem
> to work perfectly until all put together, so I am not really sure
> where the problem is.
>
> I would be very grateful for any suggestions or ideas that could
> possibly help to narrow down the problem. Maybe I am just doing
> something wrong (I hope so). Or maybe there is a bug that shows itself
> only in such a particular configuration (hope not)?

IMHO, it sounds like a resource issue... the domU providing the iSCSI,
plus the dom0 plus the domU (windows VM) are all asking for CPU, IRQ's,
etc, and someone isn't getting enough in time. It doesn't really help,
but we use a physical iSCSI server, the dom0 then connects to the iSCSI
and provides the disks to the VM's. Maybe look at assigning specific
exclusive CPU's to each dom0 and domU's, and see if you can still
reproduce the issue. Also, make absolutely sure that you don't have two
VM's accessing the same iSCSI.

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Kuba
W dniu 2014-01-30 23:51, Adam Goryachev pisze:

> On 31/01/14 00:50, Kuba wrote:
>> Dear List,
>>
>> I am trying to set up a following configuration:
>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
>> from sources,
>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
>> using VT-d, exporting block devices via iSCSI to other VMs and
>> physical machines,
>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>> 4000) installed on a block device exported from the storage VM (target
>> on the storage VM, initiator on dom0).
>>
>> Everything works perfectly (including PCI & GPU passthrough) until I
>> install GPLPV drivers on the Windows VM. After driver installation,
>> Windows needs to reboot, boots fine, displays a message that PV SCSI
>> drivers were installed and needs to reboot again, and then cannot
>> boot. Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>> sometimes BSODs with "unmountable boot volume" message. All of the
>> following I tried without GPU passthrough to narrow down the problem.
>>
>> The intriguing part is this:
>>
>> 1. If the storage VM's OS is Linux - it fails with the above symptoms.
>> 2. If the block devices for the storage VM come directly from dom0
>> (not via pci-passthrough) - it fails.
>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>> 9.2-GENERIC) - it all works.
>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>> guest support - it works, but is unstable (see below).
>> 4. If the iSCSI target is on a different physical machine - it all works.
>> 5. If the iSCSI target is on dom0 itself - it works.
>> 6. If I attach the AHCI controller to the Windows VM and install
>> directly on the hard drive - it works.
>> 7. If the block device for Windows VM is a disk, partition, file, LVM
>> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
>> iSCSI)- it works.
>>
>> If I install Windows and the GPLPV drivers on a hard drive attached to
>> dom0, Windows + GPLPV work perfectly. If I then give the same hard
>> drive as a block device to the storage VM and re-export it through
>> iSCSI, Windows usually boots fine, but works unstable. And by unstable
>> I mean random read/write errors, sometimes programs won't start,
>> ntdll.dll crashes, and after couple reboots Windows won't boot (just
>> like mentioned above).
>>
>> The configurations I would like to achieve makes sense only with PV
>> drivers on both storage and Windows VM. All of the "components" seem
>> to work perfectly until all put together, so I am not really sure
>> where the problem is.
>>
>> I would be very grateful for any suggestions or ideas that could
>> possibly help to narrow down the problem. Maybe I am just doing
>> something wrong (I hope so). Or maybe there is a bug that shows itself
>> only in such a particular configuration (hope not)?
>
> IMHO, it sounds like a resource issue... the domU providing the iSCSI,
> plus the dom0 plus the domU (windows VM) are all asking for CPU, IRQ's,
> etc, and someone isn't getting enough in time. It doesn't really help,
> but we use a physical iSCSI server, the dom0 then connects to the iSCSI
> and provides the disks to the VM's. Maybe look at assigning specific
> exclusive CPU's to each dom0 and domU's, and see if you can still
> reproduce the issue. Also, make absolutely sure that you don't have two
> VM's accessing the same iSCSI.
>
> Regards,
> Adam
>

Dear Adam,

Thank you for your reply. I will try assigning specific cores to the VMs
this weekend. I will also try to run all this on another machine and try
using something different then Windows for the second VM (why didn't I
think about it earlier?). When iSCSI target is on another physical
machine, everything works like a charm, but the whole point is to make
it all work on a single machine (everything faster than 1 Gbps is way
beyond my reach, while iperf reports ~22 Gbps between dom0 and storage
VM...)

Once again thank you for your suggestions, I will report back with more
results. In the meantime please clarify one thing for me - is there
something inherently wrong with what I'm trying to do?

Best regards,
Kuba

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Adam Goryachev-3
On 31/01/14 11:08, Kuba wrote:

> W dniu 2014-01-30 23:51, Adam Goryachev pisze:
>> On 31/01/14 00:50, Kuba wrote:
>>> Dear List,
>>>
>>> I am trying to set up a following configuration:
>>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
>>> from sources,
>>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
>>> using VT-d, exporting block devices via iSCSI to other VMs and
>>> physical machines,
>>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>>> 4000) installed on a block device exported from the storage VM (target
>>> on the storage VM, initiator on dom0).
>>>
>>> Everything works perfectly (including PCI & GPU passthrough) until I
>>> install GPLPV drivers on the Windows VM. After driver installation,
>>> Windows needs to reboot, boots fine, displays a message that PV SCSI
>>> drivers were installed and needs to reboot again, and then cannot
>>> boot. Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>>> sometimes BSODs with "unmountable boot volume" message. All of the
>>> following I tried without GPU passthrough to narrow down the problem.
>>>
>>> The intriguing part is this:
>>>
>>> 1. If the storage VM's OS is Linux - it fails with the above symptoms.
>>> 2. If the block devices for the storage VM come directly from dom0
>>> (not via pci-passthrough) - it fails.
>>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>>> 9.2-GENERIC) - it all works.
>>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>>> guest support - it works, but is unstable (see below).
>>> 4. If the iSCSI target is on a different physical machine - it all
>>> works.
>>> 5. If the iSCSI target is on dom0 itself - it works.
>>> 6. If I attach the AHCI controller to the Windows VM and install
>>> directly on the hard drive - it works.
>>> 7. If the block device for Windows VM is a disk, partition, file, LVM
>>> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
>>> iSCSI)- it works.
>>>
>>> If I install Windows and the GPLPV drivers on a hard drive attached to
>>> dom0, Windows + GPLPV work perfectly. If I then give the same hard
>>> drive as a block device to the storage VM and re-export it through
>>> iSCSI, Windows usually boots fine, but works unstable. And by unstable
>>> I mean random read/write errors, sometimes programs won't start,
>>> ntdll.dll crashes, and after couple reboots Windows won't boot (just
>>> like mentioned above).
>>>
>>> The configurations I would like to achieve makes sense only with PV
>>> drivers on both storage and Windows VM. All of the "components" seem
>>> to work perfectly until all put together, so I am not really sure
>>> where the problem is.
>>>
>>> I would be very grateful for any suggestions or ideas that could
>>> possibly help to narrow down the problem. Maybe I am just doing
>>> something wrong (I hope so). Or maybe there is a bug that shows itself
>>> only in such a particular configuration (hope not)?
>>
>> IMHO, it sounds like a resource issue... the domU providing the iSCSI,
>> plus the dom0 plus the domU (windows VM) are all asking for CPU, IRQ's,
>> etc, and someone isn't getting enough in time. It doesn't really help,
>> but we use a physical iSCSI server, the dom0 then connects to the iSCSI
>> and provides the disks to the VM's. Maybe look at assigning specific
>> exclusive CPU's to each dom0 and domU's, and see if you can still
>> reproduce the issue. Also, make absolutely sure that you don't have two
>> VM's accessing the same iSCSI.
>>
>> Regards,
>> Adam
>>
>
> Dear Adam,
>
> Thank you for your reply. I will try assigning specific cores to the
> VMs this weekend. I will also try to run all this on another machine
> and try using something different then Windows for the second VM (why
> didn't I think about it earlier?). When iSCSI target is on another
> physical machine, everything works like a charm, but the whole point
> is to make it all work on a single machine (everything faster than 1
> Gbps is way beyond my reach, while iperf reports ~22 Gbps between dom0
> and storage VM...)
>
> Once again thank you for your suggestions, I will report back with
> more results. In the meantime please clarify one thing for me - is
> there something inherently wrong with what I'm trying to do?

I don't see anything "inherently" wrong, but I would question why you
would want to do this? Why not let dom0 use the scsi controller, and
export the disks as physical devices into the various VM's? You are
adding a whole bunch of "ethernet" overhead to both domU's plus the
dom0. Considering the storage is physically local.

The problem you have is the VM's will not be portable to another
machine, because they are both tied to the physical pci devices, and the
block devices are not available on any other physical machine anyway.

A 1Gbps ethernet provides 100MB/s (actually closer to 130MB/s), so
simply bonding 2 x 1Gbps ethernet can usually provide more disk
bandwidth than the disks can provide.

In my setup, the iSCSI server uses 8 x 1Gbps ethernet bonded, and the
xen machines use 2 x 1Gbps bonded for iSCSI, plus 1 x 1Gbps which is
bridged to the domU's and for dom0 "management". You can get a couple of
cheap-ish 1Gbps network cards easily enough, and your disk subsystem
probably won't provide more than 200MB/s anyway (we can get a max of
2.5GB/s read from the disk subsystem, but the limited bandwidth for each
dom0 helps to stop any one domU from stealing all the disk IO). In
practice, I can run 4 x dom0 and obtain over 180MB/s on each of them in
parallel.

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

James Harper
In reply to this post by Kuba
>
> I am trying to set up a following configuration:
> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
> from sources,
> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
> using VT-d, exporting block devices via iSCSI to other VMs and physical
> machines,
> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
> 4000) installed on a block device exported from the storage VM (target
> on the storage VM, initiator on dom0).
>
> Everything works perfectly (including PCI & GPU passthrough) until I
> install GPLPV drivers on the Windows VM. After driver installation,
> Windows needs to reboot, boots fine, displays a message that PV SCSI

(a)

> drivers were installed and needs to reboot again, and then cannot boot.
> Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
> sometimes BSODs with "unmountable boot volume" message. All of the
> following I tried without GPU passthrough to narrow down the problem.
>
> The intriguing part is this:
>
> 1. If the storage VM's OS is Linux - it fails with the above symptoms.
> 2. If the block devices for the storage VM come directly from dom0 (not
> via pci-passthrough) - it fails.
> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
> 9.2-GENERIC) - it all works.
> 3. If the storage VM's OS is Linux with kernel compiled without Xen
> guest support - it works, but is unstable (see below).
> 4. If the iSCSI target is on a different physical machine - it all works.
> 5. If the iSCSI target is on dom0 itself - it works.
> 6. If I attach the AHCI controller to the Windows VM and install
> directly on the hard drive - it works.
> 7. If the block device for Windows VM is a disk, partition, file, LVM
> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
> iSCSI)- it works.
>
> If I install Windows and the GPLPV drivers on a hard drive attached to
> dom0, Windows + GPLPV work perfectly. If I then give the same hard drive
> as a block device to the storage VM and re-export it through iSCSI,

(b)

> Windows usually boots fine, but works unstable. And by unstable I mean
> random read/write errors, sometimes programs won't start, ntdll.dll
> crashes, and after couple reboots Windows won't boot (just like
> mentioned above).
>
> The configurations I would like to achieve makes sense only with PV
> drivers on both storage and Windows VM. All of the "components" seem to
> work perfectly until all put together, so I am not really sure where the
> problem is.
>
> I would be very grateful for any suggestions or ideas that could
> possibly help to narrow down the problem. Maybe I am just doing
> something wrong (I hope so). Or maybe there is a bug that shows itself
> only in such a particular configuration (hope not)?
>

I'm curious about prompting for the pvscsi drivers to be installed. Is this definitely what it is asking for? Pvscsi for gplpv is removed in the latest versions and suffered varying degrees of bitrot in earlier versions. If you have the iscsi initiator in dom0 then exporting a block device to windows via the normal vbd channel should be just fine.

You've gone to great lengths to explain the various things you've tried, but I think I'm a little confused on where the iscsi initiator is in the "doesn't work" scenarios. I'm having a bit of an off day today so it's probably just me, but above I have highlighted the two scenarios... could you fill me in on a few things:

At (a) and (b), is the iscsi initiator in dom0, or are you actually booting windows directly via iscsi?

At (b), with latest debug build of gplpv, can you run debugview from sysinternals.com and see if any interesting messages are displayed before things fall in a heap?

Are any strange logs shown in any of Win DomU, Dom0, or storage DomU?

How big are your disks?

Can you reproduce with only one vcpu?

What bridge are you using? Openvswitch or traditional linux bridge?

What MTU are you using on your storage network? If you are using Jumbo frames can you go back to 1500 (or at least <= 4000)?

Can you turn off scatter gather, Large Send Offload (GSO), and IP Checksum offload on all the iscsi endpoints?

Can you turn on data digest/checksum on iscsi? If all endpoints support it then this would provide additional verification that none of the network packets are getting corrupted.
 
Would driver domain work in your scenario? Then the disk could be attached directly from your storage DomU without accruing all the iscsi overhead. I'm not up with the status of HVM, vbd, and driver domain so I don't know if this is possible.

More questions than answers. Sorry :)

James


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

James Harper
In reply to this post by Adam Goryachev-3
>
> A 1Gbps ethernet provides 100MB/s (actually closer to 130MB/s), so
> simply bonding 2 x 1Gbps ethernet can usually provide more disk
> bandwidth than the disks can provide.
>
> In my setup, the iSCSI server uses 8 x 1Gbps ethernet bonded, and the
> xen machines use 2 x 1Gbps bonded for iSCSI, plus 1 x 1Gbps which is
> bridged to the domU's and for dom0 "management". You can get a couple of
> cheap-ish 1Gbps network cards easily enough, and your disk subsystem
> probably won't provide more than 200MB/s anyway (we can get a max of
> 2.5GB/s read from the disk subsystem, but the limited bandwidth for each
> dom0 helps to stop any one domU from stealing all the disk IO). In
> practice, I can run 4 x dom0 and obtain over 180MB/s on each of them in
> parallel.
>

If you workload is only iSCSI then can you comment on the choice of bonding instead of multipath?

James

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Adam Goryachev-3
On 31/01/14 12:39, James Harper wrote:

>> A 1Gbps ethernet provides 100MB/s (actually closer to 130MB/s), so
>> simply bonding 2 x 1Gbps ethernet can usually provide more disk
>> bandwidth than the disks can provide.
>>
>> In my setup, the iSCSI server uses 8 x 1Gbps ethernet bonded, and the
>> xen machines use 2 x 1Gbps bonded for iSCSI, plus 1 x 1Gbps which is
>> bridged to the domU's and for dom0 "management". You can get a couple of
>> cheap-ish 1Gbps network cards easily enough, and your disk subsystem
>> probably won't provide more than 200MB/s anyway (we can get a max of
>> 2.5GB/s read from the disk subsystem, but the limited bandwidth for each
>> dom0 helps to stop any one domU from stealing all the disk IO). In
>> practice, I can run 4 x dom0 and obtain over 180MB/s on each of them in
>> parallel.
>>
> If you workload is only iSCSI then can you comment on the choice of bonding instead of multipath?
>
> James
I use both bonding (on the iSCSI server to join the 8 x 1Gbps) and
multipath (on the dom0, one path on each interface). That way the server
side can load balance data over any of the 8 ports, and the clients can
load balance over their two ports.

There is probably additional tweaking I could do to improve performance
in all cases, but it has been stable and works, and there are other
(more pressing) demands on time (that very limited resource).

In actual fact, there are two identical iSCSI server, each with 5 x
480GB Intel SSD. They each have 8 x 1Gbps ethernet bonded for their
iSCSI, plus 1 x 1Gbps for a crossover for DRBD to sync the content. An
additional 1 x 1Gbps is for management. The iSCSI servers use heartbeat,
and will failover within about 5 seconds. The dom0 uses multipath, and
if it gets errors on both paths will wait forever until success (which
means windows sees the SCSI layer as being non-responsive). In practice,
the failover time is transparent to windows, and works really well.

Of course, a "nice" failover takes less than one second, and has no
noticeable impact from users etc (ie, if you aren't looking for the
pause, then you won't notice).

This has the added advantage of being able to live migrate domU's from
one physical host to another, which also works well.

The only thing I don't like about the current setup is that LVM layer
(above DRBD and below iSCSI) seems to massively degrade performance.

If you want more details, please ask, but I may have forgotten some of
the details because it was setup over 9 months ago. You can see a lot of
the discussions on the linux-raid mailing list where I got a lot of
assistance getting the networking and disk layers behaving properly (see
the period from January 2013 when debugging started to around April 2013
when I mostly finished).

(PS, thanks to all the users on the linux-raid list, esp Stan Hoeppner
for all your help!)

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Kuba
In reply to this post by Adam Goryachev-3
W dniu 2014-01-31 01:53, Adam Goryachev pisze:

> On 31/01/14 11:08, Kuba wrote:
>> W dniu 2014-01-30 23:51, Adam Goryachev pisze:
>>> On 31/01/14 00:50, Kuba wrote:
>>>> Dear List,
>>>>
>>>> I am trying to set up a following configuration:
>>>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
>>>> from sources,
>>>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
>>>> using VT-d, exporting block devices via iSCSI to other VMs and
>>>> physical machines,
>>>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>>>> 4000) installed on a block device exported from the storage VM (target
>>>> on the storage VM, initiator on dom0).
>>>>
>>>> Everything works perfectly (including PCI & GPU passthrough) until I
>>>> install GPLPV drivers on the Windows VM. After driver installation,
>>>> Windows needs to reboot, boots fine, displays a message that PV SCSI
>>>> drivers were installed and needs to reboot again, and then cannot
>>>> boot. Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>>>> sometimes BSODs with "unmountable boot volume" message. All of the
>>>> following I tried without GPU passthrough to narrow down the problem.
>>>>
>>>> The intriguing part is this:
>>>>
>>>> 1. If the storage VM's OS is Linux - it fails with the above symptoms.
>>>> 2. If the block devices for the storage VM come directly from dom0
>>>> (not via pci-passthrough) - it fails.
>>>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>>>> 9.2-GENERIC) - it all works.
>>>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>>>> guest support - it works, but is unstable (see below).
>>>> 4. If the iSCSI target is on a different physical machine - it all
>>>> works.
>>>> 5. If the iSCSI target is on dom0 itself - it works.
>>>> 6. If I attach the AHCI controller to the Windows VM and install
>>>> directly on the hard drive - it works.
>>>> 7. If the block device for Windows VM is a disk, partition, file, LVM
>>>> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
>>>> iSCSI)- it works.
>>>>
>>>> If I install Windows and the GPLPV drivers on a hard drive attached to
>>>> dom0, Windows + GPLPV work perfectly. If I then give the same hard
>>>> drive as a block device to the storage VM and re-export it through
>>>> iSCSI, Windows usually boots fine, but works unstable. And by unstable
>>>> I mean random read/write errors, sometimes programs won't start,
>>>> ntdll.dll crashes, and after couple reboots Windows won't boot (just
>>>> like mentioned above).
>>>>
>>>> The configurations I would like to achieve makes sense only with PV
>>>> drivers on both storage and Windows VM. All of the "components" seem
>>>> to work perfectly until all put together, so I am not really sure
>>>> where the problem is.
>>>>
>>>> I would be very grateful for any suggestions or ideas that could
>>>> possibly help to narrow down the problem. Maybe I am just doing
>>>> something wrong (I hope so). Or maybe there is a bug that shows itself
>>>> only in such a particular configuration (hope not)?
>>>
>>> IMHO, it sounds like a resource issue... the domU providing the iSCSI,
>>> plus the dom0 plus the domU (windows VM) are all asking for CPU, IRQ's,
>>> etc, and someone isn't getting enough in time. It doesn't really help,
>>> but we use a physical iSCSI server, the dom0 then connects to the iSCSI
>>> and provides the disks to the VM's. Maybe look at assigning specific
>>> exclusive CPU's to each dom0 and domU's, and see if you can still
>>> reproduce the issue. Also, make absolutely sure that you don't have two
>>> VM's accessing the same iSCSI.
>>>
>>> Regards,
>>> Adam
>>>
>>
>> Dear Adam,
>>
>> Thank you for your reply. I will try assigning specific cores to the
>> VMs this weekend. I will also try to run all this on another machine
>> and try using something different then Windows for the second VM (why
>> didn't I think about it earlier?). When iSCSI target is on another
>> physical machine, everything works like a charm, but the whole point
>> is to make it all work on a single machine (everything faster than 1
>> Gbps is way beyond my reach, while iperf reports ~22 Gbps between dom0
>> and storage VM...)
>>
>> Once again thank you for your suggestions, I will report back with
>> more results. In the meantime please clarify one thing for me - is
>> there something inherently wrong with what I'm trying to do?
>
> I don't see anything "inherently" wrong, but I would question why you
> would want to do this? Why not let dom0 use the scsi controller, and
> export the disks as physical devices into the various VM's? You are
> adding a whole bunch of "ethernet" overhead to both domU's plus the
> dom0. Considering the storage is physically local.
>
> The problem you have is the VM's will not be portable to another
> machine, because they are both tied to the physical pci devices, and the
> block devices are not available on any other physical machine anyway.
>
> A 1Gbps ethernet provides 100MB/s (actually closer to 130MB/s), so
> simply bonding 2 x 1Gbps ethernet can usually provide more disk
> bandwidth than the disks can provide.
>
> In my setup, the iSCSI server uses 8 x 1Gbps ethernet bonded, and the
> xen machines use 2 x 1Gbps bonded for iSCSI, plus 1 x 1Gbps which is
> bridged to the domU's and for dom0 "management". You can get a couple of
> cheap-ish 1Gbps network cards easily enough, and your disk subsystem
> probably won't provide more than 200MB/s anyway (we can get a max of
> 2.5GB/s read from the disk subsystem, but the limited bandwidth for each
> dom0 helps to stop any one domU from stealing all the disk IO). In
> practice, I can run 4 x dom0 and obtain over 180MB/s on each of them in
> parallel.
>
> Regards,
> Adam
>

I'd like to achieve several things at once. I'm fairly new to Xen,
(which is an impressive piece of software) and it creates possibilities
I'm only beginning to grasp.

I'm trying to build something a little bit similar to Qubes OS. I need
several VMs for different tasks, some of them Windows-based, some of
them using GPU passthrough, but all of them installed ZFS backed
storage. I'd really like to have a storage VM that is separated from
everything else as much as possible, even from dom0, just like it was a
seperate machine. I'm fairly certain that providing storage space
directly from dom0 would be faster, but that's a trade-off I'm willing
to accept - it's something between dom0 and separate physical machine.
Consequently, if all goes well, I will get following benefits:
a) everything on one physical machine (less power consumption, cheaper,
satisfying performance),
b) ZFS storage for all VMs (data integrity, snapshots, rollbacks, VM
cloning, etc.),
c) Windows on ZFS (an idea that started all this),
d) storage VM separated from everything else,
f) great flexibility.

I'm aware about VM migration issues, but that's another trade-off I'm
willing to accept. Simply put, I'm trying to achieve a lot more using
the same hardware with acceptable performance loss.

Best regards,
Kuba

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Kuba
In reply to this post by James Harper
W dniu 2014-01-31 02:35, James Harper pisze:

>>
>> I am trying to set up a following configuration:
>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
>> from sources,
>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
>> using VT-d, exporting block devices via iSCSI to other VMs and physical
>> machines,
>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>> 4000) installed on a block device exported from the storage VM (target
>> on the storage VM, initiator on dom0).
>>
>> Everything works perfectly (including PCI & GPU passthrough) until I
>> install GPLPV drivers on the Windows VM. After driver installation,
>> Windows needs to reboot, boots fine, displays a message that PV SCSI
>
> (a)
>
>> drivers were installed and needs to reboot again, and then cannot boot.
>> Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>> sometimes BSODs with "unmountable boot volume" message. All of the
>> following I tried without GPU passthrough to narrow down the problem.
>>
>> The intriguing part is this:
>>
>> 1. If the storage VM's OS is Linux - it fails with the above symptoms.
>> 2. If the block devices for the storage VM come directly from dom0 (not
>> via pci-passthrough) - it fails.
>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>> 9.2-GENERIC) - it all works.
>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>> guest support - it works, but is unstable (see below).
>> 4. If the iSCSI target is on a different physical machine - it all works.
>> 5. If the iSCSI target is on dom0 itself - it works.
>> 6. If I attach the AHCI controller to the Windows VM and install
>> directly on the hard drive - it works.
>> 7. If the block device for Windows VM is a disk, partition, file, LVM
>> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
>> iSCSI)- it works.
>>
>> If I install Windows and the GPLPV drivers on a hard drive attached to
>> dom0, Windows + GPLPV work perfectly. If I then give the same hard drive
>> as a block device to the storage VM and re-export it through iSCSI,
>
> (b)
>
>> Windows usually boots fine, but works unstable. And by unstable I mean
>> random read/write errors, sometimes programs won't start, ntdll.dll
>> crashes, and after couple reboots Windows won't boot (just like
>> mentioned above).
>>
>> The configurations I would like to achieve makes sense only with PV
>> drivers on both storage and Windows VM. All of the "components" seem to
>> work perfectly until all put together, so I am not really sure where the
>> problem is.
>>
>> I would be very grateful for any suggestions or ideas that could
>> possibly help to narrow down the problem. Maybe I am just doing
>> something wrong (I hope so). Or maybe there is a bug that shows itself
>> only in such a particular configuration (hope not)?
>>
>
> I'm curious about prompting for the pvscsi drivers to be installed. Is this definitely what it is asking for? Pvscsi for gplpv is removed in the latest versions and suffered varying degrees of bitrot in earlier versions. If you have the iscsi initiator in dom0 then exporting a block device to windows via the normal vbd channel should be just fine.
>
> You've gone to great lengths to explain the various things you've tried, but I think I'm a little confused on where the iscsi initiator is in the "doesn't work" scenarios. I'm having a bit of an off day today so it's probably just me, but above I have highlighted the two scenarios... could you fill me in on a few things:
>
> At (a) and (b), is the iscsi initiator in dom0, or are you actually booting windows directly via iscsi?
>
> At (b), with latest debug build of gplpv, can you run debugview from sysinternals.com and see if any interesting messages are displayed before things fall in a heap?
>
> Are any strange logs shown in any of Win DomU, Dom0, or storage DomU?
>
> How big are your disks?
>
> Can you reproduce with only one vcpu?
>
> What bridge are you using? Openvswitch or traditional linux bridge?
>
> What MTU are you using on your storage network? If you are using Jumbo frames can you go back to 1500 (or at least <= 4000)?
>
> Can you turn off scatter gather, Large Send Offload (GSO), and IP Checksum offload on all the iscsi endpoints?
>
> Can you turn on data digest/checksum on iscsi? If all endpoints support it then this would provide additional verification that none of the network packets are getting corrupted.
>
> Would driver domain work in your scenario? Then the disk could be attached directly from your storage DomU without accruing all the iscsi overhead. I'm not up with the status of HVM, vbd, and driver domain so I don't know if this is possible.
>
> More questions than answers. Sorry :)
>
> James

Dear James,

thank you for your questions - I really appreciate everything that may
help me move closer to solving or isolating the problem.

I'll check what type of driver is used exactly - up until now I always
just installed all drivers included in the package, I thought all of
them were necessary. I'll try installing them without XenScsi.

Do you mean revisions > 1092:85b99b9795a6 by "the latest versions"?
Which version should I use?

Forgive me if the descriptions were unclear. The initiator was always in
dom0. I only moved the target to dom0 or a separate physical machine in
(4) and (5). I didn't boot Windows directly from iSCSI (in fact I tried
couple times, but had some problems with it, so I didn't mention it).

My "disks" (the block devices I dedicated to the Windows VM) were whole
120GB and 240GB SSDs, ~100GB ZVOLs and 50GB LVM volumes.

I'm using traditional linux bridge. I didn't set MTUs explicitly, so I
assume it's 1500, but I will verify this.

I'd love to use a storage driver domain, but the wiki says "It is not
possible to use driver domains with pygrub or HVM guests yet". But the
page is a couple of months old, maybe it's an outdated info? It surely
is worth checking out.

I'll do my best to provide answers to the remaining questions as soon as
possible. Thank you for so many ideas.

Best regards,
Kuba

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Kuba
W dniu 2014-02-01 20:27, Kuba pisze:

> W dniu 2014-01-31 02:35, James Harper pisze:
>>>
>>> I am trying to set up a following configuration:
>>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
>>> from sources,
>>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
>>> using VT-d, exporting block devices via iSCSI to other VMs and physical
>>> machines,
>>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>>> 4000) installed on a block device exported from the storage VM (target
>>> on the storage VM, initiator on dom0).
>>>
>>> Everything works perfectly (including PCI & GPU passthrough) until I
>>> install GPLPV drivers on the Windows VM. After driver installation,
>>> Windows needs to reboot, boots fine, displays a message that PV SCSI
>>
>> (a)
>>
>>> drivers were installed and needs to reboot again, and then cannot boot.
>>> Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>>> sometimes BSODs with "unmountable boot volume" message. All of the
>>> following I tried without GPU passthrough to narrow down the problem.
>>>
>>> The intriguing part is this:
>>>
>>> 1. If the storage VM's OS is Linux - it fails with the above symptoms.
>>> 2. If the block devices for the storage VM come directly from dom0 (not
>>> via pci-passthrough) - it fails.
>>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>>> 9.2-GENERIC) - it all works.
>>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>>> guest support - it works, but is unstable (see below).
>>> 4. If the iSCSI target is on a different physical machine - it all
>>> works.
>>> 5. If the iSCSI target is on dom0 itself - it works.
>>> 6. If I attach the AHCI controller to the Windows VM and install
>>> directly on the hard drive - it works.
>>> 7. If the block device for Windows VM is a disk, partition, file, LVM
>>> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
>>> iSCSI)- it works.
>>>
>>> If I install Windows and the GPLPV drivers on a hard drive attached to
>>> dom0, Windows + GPLPV work perfectly. If I then give the same hard drive
>>> as a block device to the storage VM and re-export it through iSCSI,
>>
>> (b)
>>
>>> Windows usually boots fine, but works unstable. And by unstable I mean
>>> random read/write errors, sometimes programs won't start, ntdll.dll
>>> crashes, and after couple reboots Windows won't boot (just like
>>> mentioned above).
>>>
>>> The configurations I would like to achieve makes sense only with PV
>>> drivers on both storage and Windows VM. All of the "components" seem to
>>> work perfectly until all put together, so I am not really sure where the
>>> problem is.
>>>
>>> I would be very grateful for any suggestions or ideas that could
>>> possibly help to narrow down the problem. Maybe I am just doing
>>> something wrong (I hope so). Or maybe there is a bug that shows itself
>>> only in such a particular configuration (hope not)?
>>>
>>
>> I'm curious about prompting for the pvscsi drivers to be installed. Is
>> this definitely what it is asking for? Pvscsi for gplpv is removed in
>> the latest versions and suffered varying degrees of bitrot in earlier
>> versions. If you have the iscsi initiator in dom0 then exporting a
>> block device to windows via the normal vbd channel should be just fine.
>>
>> You've gone to great lengths to explain the various things you've
>> tried, but I think I'm a little confused on where the iscsi initiator
>> is in the "doesn't work" scenarios. I'm having a bit of an off day
>> today so it's probably just me, but above I have highlighted the two
>> scenarios... could you fill me in on a few things:
>>
>> At (a) and (b), is the iscsi initiator in dom0, or are you actually
>> booting windows directly via iscsi?
>>
>> At (b), with latest debug build of gplpv, can you run debugview from
>> sysinternals.com and see if any interesting messages are displayed
>> before things fall in a heap?
>>
>> Are any strange logs shown in any of Win DomU, Dom0, or storage DomU?
>>
>> How big are your disks?
>>
>> Can you reproduce with only one vcpu?
>>
>> What bridge are you using? Openvswitch or traditional linux bridge?
>>
>> What MTU are you using on your storage network? If you are using Jumbo
>> frames can you go back to 1500 (or at least <= 4000)?
>>
>> Can you turn off scatter gather, Large Send Offload (GSO), and IP
>> Checksum offload on all the iscsi endpoints?
>>
>> Can you turn on data digest/checksum on iscsi? If all endpoints
>> support it then this would provide additional verification that none
>> of the network packets are getting corrupted.
>>
>> Would driver domain work in your scenario? Then the disk could be
>> attached directly from your storage DomU without accruing all the
>> iscsi overhead. I'm not up with the status of HVM, vbd, and driver
>> domain so I don't know if this is possible.
>>
>> More questions than answers. Sorry :)
>>
>> James
>
> Dear James,
>
> thank you for your questions - I really appreciate everything that may
> help me move closer to solving or isolating the problem.
>
> I'll check what type of driver is used exactly - up until now I always
> just installed all drivers included in the package, I thought all of
> them were necessary. I'll try installing them without XenScsi.
>
> Do you mean revisions > 1092:85b99b9795a6 by "the latest versions"?
> Which version should I use?
>
> Forgive me if the descriptions were unclear. The initiator was always in
> dom0. I only moved the target to dom0 or a separate physical machine in
> (4) and (5). I didn't boot Windows directly from iSCSI (in fact I tried
> couple times, but had some problems with it, so I didn't mention it).
>
> My "disks" (the block devices I dedicated to the Windows VM) were whole
> 120GB and 240GB SSDs, ~100GB ZVOLs and 50GB LVM volumes.
>
> I'm using traditional linux bridge. I didn't set MTUs explicitly, so I
> assume it's 1500, but I will verify this.
>
> I'd love to use a storage driver domain, but the wiki says "It is not
> possible to use driver domains with pygrub or HVM guests yet". But the
> page is a couple of months old, maybe it's an outdated info? It surely
> is worth checking out.
>
> I'll do my best to provide answers to the remaining questions as soon as
> possible. Thank you for so many ideas.
>
> Best regards,
> Kuba
>
> _______________________________________________
> Xen-users mailing list
> [hidden email]
> http://lists.xen.org/xen-users
It seems the problems are not related to GPLPV. There is an easy way to
reproduce the issues without Windows and without installing anything,
using only livecds for two DomUs:

1) Set up a Linux Dom0 with Xen 4.3.1 and standard Linux bridge for Dom0
and DomUs
2) Launch DomU "A"
3) Run an iSCSI target inside the "A" DomU exporting any block device or
file
4) In Dom0 log in to the iSCSI target; new block device appears in Dom0
(let's say /dev/sdc)
5) Provide /dev/sdc using "phy:/dev/sdc,xvda,w" to another DomU "B"
6) Any (?) write to the /dev/xvda in DomU "B" destroys data on the iSCSI
target


Some details:
ad 1) tried with Debian 7.3 (kernel 3.2.51 + Xen 4.3.1 compiled from
sources) and Ubuntu Server 13.10 (kernel 3.11.0 + Xen 4.3.1 installed
from Ubuntu package) as Dom0

ad 2) tried with Debian 7.2 livecd, Ubuntu 13.10 Desktop livecd and
FreeBSD 10 livecd as DomU "A"

ad 3) on Linux DomU "A" I used iSCSI Enterprise Target, on FreeBSD I
used native iSCSI target (ctld). The exported block device can be a file
on tmpfs, a disk connected to a SATA controller passed-through via VT-d
or anything else, it doesn't affect the outcome.

ad 4) I tried with open-iscsi initiator. Inside Dom0 the block device
provided by the initiator (/dev/sdc) behaved very well - for example I
could create a file system on it, mount it, run fsck, etc. Everything
was ok.

ad 5) As DomU "B" I ran the same live cds as in 2. Now, let's say we run
"mkfs.ext4 /dev/sdc" on the Dom0 just before launching DomU "B".

ad 6) Then, inside DomU "B", run "mount /dev/xvda /mnt". Everything is
ok up to this point. But now just run "ls /mnt", unmount xvda, shut down
DomU "B" and run fsck on Dom0 again and you will to see some serious
file system problems. Running fsck insided DomU "B" just after
unmounting xvda also shows errors after any operation involving a write
to the underlying block device.

This was tested on two different physical machines independently. Dom0
always had 4 pinned vcpus, each DomU had 1 pinned vcpu.

I was not able to log in from open-iscsi initiator to IET with header
and data digests enabled, but it worked with FreeBSD native iSCSI
target. FreeBSD's target is fairly new so I don't know how stable it is.
I'm attaching relevant dmesg logs from both Linux and FreeBSD DomU "A"
with some errors, as well as config files for both DomUs.

Any ideas and suggestions on what could be the cause and what can I do
will be much appreciated.

Is there any other way to export a block device from a DomU to Dom0 that
would eliminate the need for using iSCSI? A storage driver domain based
on FreeBSD 10 seems to work if it provides a block device to another
FreeBSD 10 DomU running qemu-traditional. Can Dom0 attach itself to a
block device in a similar way?

Best regards,
Kuba

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users

DomU-A-Ubuntu.log (526 bytes) Download Attachment
DomU-A-FreeBSD10.log (7K) Download Attachment
DomU-A.conf (418 bytes) Download Attachment
DomU-B.conf (448 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Roger Pau Monné-3
On 05/02/14 17:13, Kuba wrote:

> W dniu 2014-02-01 20:27, Kuba pisze:
>> W dniu 2014-01-31 02:35, James Harper pisze:
>>>>
>>>> I am trying to set up a following configuration:
>>>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
>>>> from sources,
>>>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
>>>> using VT-d, exporting block devices via iSCSI to other VMs and physical
>>>> machines,
>>>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>>>> 4000) installed on a block device exported from the storage VM (target
>>>> on the storage VM, initiator on dom0).
>>>>
>>>> Everything works perfectly (including PCI & GPU passthrough) until I
>>>> install GPLPV drivers on the Windows VM. After driver installation,
>>>> Windows needs to reboot, boots fine, displays a message that PV SCSI
>>>
>>> (a)
>>>
>>>> drivers were installed and needs to reboot again, and then cannot boot.
>>>> Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>>>> sometimes BSODs with "unmountable boot volume" message. All of the
>>>> following I tried without GPU passthrough to narrow down the problem.
>>>>
>>>> The intriguing part is this:
>>>>
>>>> 1. If the storage VM's OS is Linux - it fails with the above symptoms.
>>>> 2. If the block devices for the storage VM come directly from dom0 (not
>>>> via pci-passthrough) - it fails.
>>>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>>>> 9.2-GENERIC) - it all works.
>>>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>>>> guest support - it works, but is unstable (see below).
>>>> 4. If the iSCSI target is on a different physical machine - it all
>>>> works.
>>>> 5. If the iSCSI target is on dom0 itself - it works.
>>>> 6. If I attach the AHCI controller to the Windows VM and install
>>>> directly on the hard drive - it works.
>>>> 7. If the block device for Windows VM is a disk, partition, file, LVM
>>>> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
>>>> iSCSI)- it works.
>>>>
>>>> If I install Windows and the GPLPV drivers on a hard drive attached to
>>>> dom0, Windows + GPLPV work perfectly. If I then give the same hard
>>>> drive
>>>> as a block device to the storage VM and re-export it through iSCSI,
>>>
>>> (b)
>>>
>>>> Windows usually boots fine, but works unstable. And by unstable I mean
>>>> random read/write errors, sometimes programs won't start, ntdll.dll
>>>> crashes, and after couple reboots Windows won't boot (just like
>>>> mentioned above).
>>>>
>>>> The configurations I would like to achieve makes sense only with PV
>>>> drivers on both storage and Windows VM. All of the "components" seem to
>>>> work perfectly until all put together, so I am not really sure where
>>>> the
>>>> problem is.
>>>>
>>>> I would be very grateful for any suggestions or ideas that could
>>>> possibly help to narrow down the problem. Maybe I am just doing
>>>> something wrong (I hope so). Or maybe there is a bug that shows itself
>>>> only in such a particular configuration (hope not)?
>>>>
>>>
>>> I'm curious about prompting for the pvscsi drivers to be installed. Is
>>> this definitely what it is asking for? Pvscsi for gplpv is removed in
>>> the latest versions and suffered varying degrees of bitrot in earlier
>>> versions. If you have the iscsi initiator in dom0 then exporting a
>>> block device to windows via the normal vbd channel should be just fine.
>>>
>>> You've gone to great lengths to explain the various things you've
>>> tried, but I think I'm a little confused on where the iscsi initiator
>>> is in the "doesn't work" scenarios. I'm having a bit of an off day
>>> today so it's probably just me, but above I have highlighted the two
>>> scenarios... could you fill me in on a few things:
>>>
>>> At (a) and (b), is the iscsi initiator in dom0, or are you actually
>>> booting windows directly via iscsi?
>>>
>>> At (b), with latest debug build of gplpv, can you run debugview from
>>> sysinternals.com and see if any interesting messages are displayed
>>> before things fall in a heap?
>>>
>>> Are any strange logs shown in any of Win DomU, Dom0, or storage DomU?
>>>
>>> How big are your disks?
>>>
>>> Can you reproduce with only one vcpu?
>>>
>>> What bridge are you using? Openvswitch or traditional linux bridge?
>>>
>>> What MTU are you using on your storage network? If you are using Jumbo
>>> frames can you go back to 1500 (or at least <= 4000)?
>>>
>>> Can you turn off scatter gather, Large Send Offload (GSO), and IP
>>> Checksum offload on all the iscsi endpoints?
>>>
>>> Can you turn on data digest/checksum on iscsi? If all endpoints
>>> support it then this would provide additional verification that none
>>> of the network packets are getting corrupted.
>>>
>>> Would driver domain work in your scenario? Then the disk could be
>>> attached directly from your storage DomU without accruing all the
>>> iscsi overhead. I'm not up with the status of HVM, vbd, and driver
>>> domain so I don't know if this is possible.
>>>
>>> More questions than answers. Sorry :)
>>>
>>> James
>>
>> Dear James,
>>
>> thank you for your questions - I really appreciate everything that may
>> help me move closer to solving or isolating the problem.
>>
>> I'll check what type of driver is used exactly - up until now I always
>> just installed all drivers included in the package, I thought all of
>> them were necessary. I'll try installing them without XenScsi.
>>
>> Do you mean revisions > 1092:85b99b9795a6 by "the latest versions"?
>> Which version should I use?
>>
>> Forgive me if the descriptions were unclear. The initiator was always in
>> dom0. I only moved the target to dom0 or a separate physical machine in
>> (4) and (5). I didn't boot Windows directly from iSCSI (in fact I tried
>> couple times, but had some problems with it, so I didn't mention it).
>>
>> My "disks" (the block devices I dedicated to the Windows VM) were whole
>> 120GB and 240GB SSDs, ~100GB ZVOLs and 50GB LVM volumes.
>>
>> I'm using traditional linux bridge. I didn't set MTUs explicitly, so I
>> assume it's 1500, but I will verify this.
>>
>> I'd love to use a storage driver domain, but the wiki says "It is not
>> possible to use driver domains with pygrub or HVM guests yet". But the
>> page is a couple of months old, maybe it's an outdated info? It surely
>> is worth checking out.
>>
>> I'll do my best to provide answers to the remaining questions as soon as
>> possible. Thank you for so many ideas.
>>
>> Best regards,
>> Kuba
>>
>> _______________________________________________
>> Xen-users mailing list
>> [hidden email]
>> http://lists.xen.org/xen-users
>
> It seems the problems are not related to GPLPV. There is an easy way to
> reproduce the issues without Windows and without installing anything,
> using only livecds for two DomUs:
>
> 1) Set up a Linux Dom0 with Xen 4.3.1 and standard Linux bridge for Dom0
> and DomUs

Are you using a Xen build with debugging enabled? I think I might have a
clue of what's happening, because I also saw it. Could you recompile Xen
with debugging enabled and try the same test (iSCSI target on DomU and
initiator on Dom0)?

Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Kuba
W dniu 2014-02-05 17:29, Roger Pau Monné pisze:

> On 05/02/14 17:13, Kuba wrote:
>> W dniu 2014-02-01 20:27, Kuba pisze:
>>> W dniu 2014-01-31 02:35, James Harper pisze:
>>>>>
>>>>> I am trying to set up a following configuration:
>>>>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
>>>>> from sources,
>>>>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
>>>>> using VT-d, exporting block devices via iSCSI to other VMs and physical
>>>>> machines,
>>>>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>>>>> 4000) installed on a block device exported from the storage VM (target
>>>>> on the storage VM, initiator on dom0).
>>>>>
>>>>> Everything works perfectly (including PCI & GPU passthrough) until I
>>>>> install GPLPV drivers on the Windows VM. After driver installation,
>>>>> Windows needs to reboot, boots fine, displays a message that PV SCSI
>>>>
>>>> (a)
>>>>
>>>>> drivers were installed and needs to reboot again, and then cannot boot.
>>>>> Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>>>>> sometimes BSODs with "unmountable boot volume" message. All of the
>>>>> following I tried without GPU passthrough to narrow down the problem.
>>>>>
>>>>> The intriguing part is this:
>>>>>
>>>>> 1. If the storage VM's OS is Linux - it fails with the above symptoms.
>>>>> 2. If the block devices for the storage VM come directly from dom0 (not
>>>>> via pci-passthrough) - it fails.
>>>>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>>>>> 9.2-GENERIC) - it all works.
>>>>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>>>>> guest support - it works, but is unstable (see below).
>>>>> 4. If the iSCSI target is on a different physical machine - it all
>>>>> works.
>>>>> 5. If the iSCSI target is on dom0 itself - it works.
>>>>> 6. If I attach the AHCI controller to the Windows VM and install
>>>>> directly on the hard drive - it works.
>>>>> 7. If the block device for Windows VM is a disk, partition, file, LVM
>>>>> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
>>>>> iSCSI)- it works.
>>>>>
>>>>> If I install Windows and the GPLPV drivers on a hard drive attached to
>>>>> dom0, Windows + GPLPV work perfectly. If I then give the same hard
>>>>> drive
>>>>> as a block device to the storage VM and re-export it through iSCSI,
>>>>
>>>> (b)
>>>>
>>>>> Windows usually boots fine, but works unstable. And by unstable I mean
>>>>> random read/write errors, sometimes programs won't start, ntdll.dll
>>>>> crashes, and after couple reboots Windows won't boot (just like
>>>>> mentioned above).
>>>>>
>>>>> The configurations I would like to achieve makes sense only with PV
>>>>> drivers on both storage and Windows VM. All of the "components" seem to
>>>>> work perfectly until all put together, so I am not really sure where
>>>>> the
>>>>> problem is.
>>>>>
>>>>> I would be very grateful for any suggestions or ideas that could
>>>>> possibly help to narrow down the problem. Maybe I am just doing
>>>>> something wrong (I hope so). Or maybe there is a bug that shows itself
>>>>> only in such a particular configuration (hope not)?
>>>>>
>>>>
>>>> I'm curious about prompting for the pvscsi drivers to be installed. Is
>>>> this definitely what it is asking for? Pvscsi for gplpv is removed in
>>>> the latest versions and suffered varying degrees of bitrot in earlier
>>>> versions. If you have the iscsi initiator in dom0 then exporting a
>>>> block device to windows via the normal vbd channel should be just fine.
>>>>
>>>> You've gone to great lengths to explain the various things you've
>>>> tried, but I think I'm a little confused on where the iscsi initiator
>>>> is in the "doesn't work" scenarios. I'm having a bit of an off day
>>>> today so it's probably just me, but above I have highlighted the two
>>>> scenarios... could you fill me in on a few things:
>>>>
>>>> At (a) and (b), is the iscsi initiator in dom0, or are you actually
>>>> booting windows directly via iscsi?
>>>>
>>>> At (b), with latest debug build of gplpv, can you run debugview from
>>>> sysinternals.com and see if any interesting messages are displayed
>>>> before things fall in a heap?
>>>>
>>>> Are any strange logs shown in any of Win DomU, Dom0, or storage DomU?
>>>>
>>>> How big are your disks?
>>>>
>>>> Can you reproduce with only one vcpu?
>>>>
>>>> What bridge are you using? Openvswitch or traditional linux bridge?
>>>>
>>>> What MTU are you using on your storage network? If you are using Jumbo
>>>> frames can you go back to 1500 (or at least <= 4000)?
>>>>
>>>> Can you turn off scatter gather, Large Send Offload (GSO), and IP
>>>> Checksum offload on all the iscsi endpoints?
>>>>
>>>> Can you turn on data digest/checksum on iscsi? If all endpoints
>>>> support it then this would provide additional verification that none
>>>> of the network packets are getting corrupted.
>>>>
>>>> Would driver domain work in your scenario? Then the disk could be
>>>> attached directly from your storage DomU without accruing all the
>>>> iscsi overhead. I'm not up with the status of HVM, vbd, and driver
>>>> domain so I don't know if this is possible.
>>>>
>>>> More questions than answers. Sorry :)
>>>>
>>>> James
>>>
>>> Dear James,
>>>
>>> thank you for your questions - I really appreciate everything that may
>>> help me move closer to solving or isolating the problem.
>>>
>>> I'll check what type of driver is used exactly - up until now I always
>>> just installed all drivers included in the package, I thought all of
>>> them were necessary. I'll try installing them without XenScsi.
>>>
>>> Do you mean revisions > 1092:85b99b9795a6 by "the latest versions"?
>>> Which version should I use?
>>>
>>> Forgive me if the descriptions were unclear. The initiator was always in
>>> dom0. I only moved the target to dom0 or a separate physical machine in
>>> (4) and (5). I didn't boot Windows directly from iSCSI (in fact I tried
>>> couple times, but had some problems with it, so I didn't mention it).
>>>
>>> My "disks" (the block devices I dedicated to the Windows VM) were whole
>>> 120GB and 240GB SSDs, ~100GB ZVOLs and 50GB LVM volumes.
>>>
>>> I'm using traditional linux bridge. I didn't set MTUs explicitly, so I
>>> assume it's 1500, but I will verify this.
>>>
>>> I'd love to use a storage driver domain, but the wiki says "It is not
>>> possible to use driver domains with pygrub or HVM guests yet". But the
>>> page is a couple of months old, maybe it's an outdated info? It surely
>>> is worth checking out.
>>>
>>> I'll do my best to provide answers to the remaining questions as soon as
>>> possible. Thank you for so many ideas.
>>>
>>> Best regards,
>>> Kuba
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> [hidden email]
>>> http://lists.xen.org/xen-users
>>
>> It seems the problems are not related to GPLPV. There is an easy way to
>> reproduce the issues without Windows and without installing anything,
>> using only livecds for two DomUs:
>>
>> 1) Set up a Linux Dom0 with Xen 4.3.1 and standard Linux bridge for Dom0
>> and DomUs
>
> Are you using a Xen build with debugging enabled? I think I might have a
> clue of what's happening, because I also saw it. Could you recompile Xen
> with debugging enabled and try the same test (iSCSI target on DomU and
> initiator on Dom0)?
>
> Roger.
>
> _______________________________________________
> Xen-users mailing list
> [hidden email]
> http://lists.xen.org/xen-users
>

Of course I could! Please point me to any relevant information on how to
build Xen with debugging enabled and what to do next. I build Xen using
standard ./configure && make world && make install.

Kuba

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Roger Pau Monné-3
On 05/02/14 17:43, Kuba wrote:

> W dniu 2014-02-05 17:29, Roger Pau Monné pisze:
>> On 05/02/14 17:13, Kuba wrote:
>>> W dniu 2014-02-01 20:27, Kuba pisze:
>>>> W dniu 2014-01-31 02:35, James Harper pisze:
>>>>>>
>>>>>> I am trying to set up a following configuration:
>>>>>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
>>>>>> from sources,
>>>>>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
>>>>>> using VT-d, exporting block devices via iSCSI to other VMs and
>>>>>> physical
>>>>>> machines,
>>>>>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>>>>>> 4000) installed on a block device exported from the storage VM
>>>>>> (target
>>>>>> on the storage VM, initiator on dom0).
>>>>>>
>>>>>> Everything works perfectly (including PCI & GPU passthrough) until I
>>>>>> install GPLPV drivers on the Windows VM. After driver installation,
>>>>>> Windows needs to reboot, boots fine, displays a message that PV SCSI
>>>>>
>>>>> (a)
>>>>>
>>>>>> drivers were installed and needs to reboot again, and then cannot
>>>>>> boot.
>>>>>> Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>>>>>> sometimes BSODs with "unmountable boot volume" message. All of the
>>>>>> following I tried without GPU passthrough to narrow down the problem.
>>>>>>
>>>>>> The intriguing part is this:
>>>>>>
>>>>>> 1. If the storage VM's OS is Linux - it fails with the above
>>>>>> symptoms.
>>>>>> 2. If the block devices for the storage VM come directly from dom0
>>>>>> (not
>>>>>> via pci-passthrough) - it fails.
>>>>>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>>>>>> 9.2-GENERIC) - it all works.
>>>>>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>>>>>> guest support - it works, but is unstable (see below).
>>>>>> 4. If the iSCSI target is on a different physical machine - it all
>>>>>> works.
>>>>>> 5. If the iSCSI target is on dom0 itself - it works.
>>>>>> 6. If I attach the AHCI controller to the Windows VM and install
>>>>>> directly on the hard drive - it works.
>>>>>> 7. If the block device for Windows VM is a disk, partition, file, LVM
>>>>>> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
>>>>>> iSCSI)- it works.
>>>>>>
>>>>>> If I install Windows and the GPLPV drivers on a hard drive
>>>>>> attached to
>>>>>> dom0, Windows + GPLPV work perfectly. If I then give the same hard
>>>>>> drive
>>>>>> as a block device to the storage VM and re-export it through iSCSI,
>>>>>
>>>>> (b)
>>>>>
>>>>>> Windows usually boots fine, but works unstable. And by unstable I
>>>>>> mean
>>>>>> random read/write errors, sometimes programs won't start, ntdll.dll
>>>>>> crashes, and after couple reboots Windows won't boot (just like
>>>>>> mentioned above).
>>>>>>
>>>>>> The configurations I would like to achieve makes sense only with PV
>>>>>> drivers on both storage and Windows VM. All of the "components"
>>>>>> seem to
>>>>>> work perfectly until all put together, so I am not really sure where
>>>>>> the
>>>>>> problem is.
>>>>>>
>>>>>> I would be very grateful for any suggestions or ideas that could
>>>>>> possibly help to narrow down the problem. Maybe I am just doing
>>>>>> something wrong (I hope so). Or maybe there is a bug that shows
>>>>>> itself
>>>>>> only in such a particular configuration (hope not)?
>>>>>>
>>>>>
>>>>> I'm curious about prompting for the pvscsi drivers to be installed. Is
>>>>> this definitely what it is asking for? Pvscsi for gplpv is removed in
>>>>> the latest versions and suffered varying degrees of bitrot in earlier
>>>>> versions. If you have the iscsi initiator in dom0 then exporting a
>>>>> block device to windows via the normal vbd channel should be just
>>>>> fine.
>>>>>
>>>>> You've gone to great lengths to explain the various things you've
>>>>> tried, but I think I'm a little confused on where the iscsi initiator
>>>>> is in the "doesn't work" scenarios. I'm having a bit of an off day
>>>>> today so it's probably just me, but above I have highlighted the two
>>>>> scenarios... could you fill me in on a few things:
>>>>>
>>>>> At (a) and (b), is the iscsi initiator in dom0, or are you actually
>>>>> booting windows directly via iscsi?
>>>>>
>>>>> At (b), with latest debug build of gplpv, can you run debugview from
>>>>> sysinternals.com and see if any interesting messages are displayed
>>>>> before things fall in a heap?
>>>>>
>>>>> Are any strange logs shown in any of Win DomU, Dom0, or storage DomU?
>>>>>
>>>>> How big are your disks?
>>>>>
>>>>> Can you reproduce with only one vcpu?
>>>>>
>>>>> What bridge are you using? Openvswitch or traditional linux bridge?
>>>>>
>>>>> What MTU are you using on your storage network? If you are using Jumbo
>>>>> frames can you go back to 1500 (or at least <= 4000)?
>>>>>
>>>>> Can you turn off scatter gather, Large Send Offload (GSO), and IP
>>>>> Checksum offload on all the iscsi endpoints?
>>>>>
>>>>> Can you turn on data digest/checksum on iscsi? If all endpoints
>>>>> support it then this would provide additional verification that none
>>>>> of the network packets are getting corrupted.
>>>>>
>>>>> Would driver domain work in your scenario? Then the disk could be
>>>>> attached directly from your storage DomU without accruing all the
>>>>> iscsi overhead. I'm not up with the status of HVM, vbd, and driver
>>>>> domain so I don't know if this is possible.
>>>>>
>>>>> More questions than answers. Sorry :)
>>>>>
>>>>> James
>>>>
>>>> Dear James,
>>>>
>>>> thank you for your questions - I really appreciate everything that may
>>>> help me move closer to solving or isolating the problem.
>>>>
>>>> I'll check what type of driver is used exactly - up until now I always
>>>> just installed all drivers included in the package, I thought all of
>>>> them were necessary. I'll try installing them without XenScsi.
>>>>
>>>> Do you mean revisions > 1092:85b99b9795a6 by "the latest versions"?
>>>> Which version should I use?
>>>>
>>>> Forgive me if the descriptions were unclear. The initiator was
>>>> always in
>>>> dom0. I only moved the target to dom0 or a separate physical machine in
>>>> (4) and (5). I didn't boot Windows directly from iSCSI (in fact I tried
>>>> couple times, but had some problems with it, so I didn't mention it).
>>>>
>>>> My "disks" (the block devices I dedicated to the Windows VM) were whole
>>>> 120GB and 240GB SSDs, ~100GB ZVOLs and 50GB LVM volumes.
>>>>
>>>> I'm using traditional linux bridge. I didn't set MTUs explicitly, so I
>>>> assume it's 1500, but I will verify this.
>>>>
>>>> I'd love to use a storage driver domain, but the wiki says "It is not
>>>> possible to use driver domains with pygrub or HVM guests yet". But the
>>>> page is a couple of months old, maybe it's an outdated info? It surely
>>>> is worth checking out.
>>>>
>>>> I'll do my best to provide answers to the remaining questions as
>>>> soon as
>>>> possible. Thank you for so many ideas.
>>>>
>>>> Best regards,
>>>> Kuba
>>>>
>>>> _______________________________________________
>>>> Xen-users mailing list
>>>> [hidden email]
>>>> http://lists.xen.org/xen-users
>>>
>>> It seems the problems are not related to GPLPV. There is an easy way to
>>> reproduce the issues without Windows and without installing anything,
>>> using only livecds for two DomUs:
>>>
>>> 1) Set up a Linux Dom0 with Xen 4.3.1 and standard Linux bridge for Dom0
>>> and DomUs
>>
>> Are you using a Xen build with debugging enabled? I think I might have a
>> clue of what's happening, because I also saw it. Could you recompile Xen
>> with debugging enabled and try the same test (iSCSI target on DomU and
>> initiator on Dom0)?
>>
>> Roger.
>>
>> _______________________________________________
>> Xen-users mailing list
>> [hidden email]
>> http://lists.xen.org/xen-users
>>
>
> Of course I could! Please point me to any relevant information on how to
> build Xen with debugging enabled and what to do next. I build Xen using
> standard ./configure && make world && make install.

Just `make debug=y xen` and boot with the resulting xen.gz.

Roger.


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Kuba
W dniu 2014-02-05 17:54, Roger Pau Monné pisze:

> On 05/02/14 17:43, Kuba wrote:
>> W dniu 2014-02-05 17:29, Roger Pau Monné pisze:
>>> On 05/02/14 17:13, Kuba wrote:
>>>> W dniu 2014-02-01 20:27, Kuba pisze:
>>>>> W dniu 2014-01-31 02:35, James Harper pisze:
>>>>>>>
>>>>>>> I am trying to set up a following configuration:
>>>>>>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1 compiled
>>>>>>> from sources,
>>>>>>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller attached
>>>>>>> using VT-d, exporting block devices via iSCSI to other VMs and
>>>>>>> physical
>>>>>>> machines,
>>>>>>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>>>>>>> 4000) installed on a block device exported from the storage VM
>>>>>>> (target
>>>>>>> on the storage VM, initiator on dom0).
>>>>>>>
>>>>>>> Everything works perfectly (including PCI & GPU passthrough) until I
>>>>>>> install GPLPV drivers on the Windows VM. After driver installation,
>>>>>>> Windows needs to reboot, boots fine, displays a message that PV SCSI
>>>>>>
>>>>>> (a)
>>>>>>
>>>>>>> drivers were installed and needs to reboot again, and then cannot
>>>>>>> boot.
>>>>>>> Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>>>>>>> sometimes BSODs with "unmountable boot volume" message. All of the
>>>>>>> following I tried without GPU passthrough to narrow down the problem.
>>>>>>>
>>>>>>> The intriguing part is this:
>>>>>>>
>>>>>>> 1. If the storage VM's OS is Linux - it fails with the above
>>>>>>> symptoms.
>>>>>>> 2. If the block devices for the storage VM come directly from dom0
>>>>>>> (not
>>>>>>> via pci-passthrough) - it fails.
>>>>>>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>>>>>>> 9.2-GENERIC) - it all works.
>>>>>>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>>>>>>> guest support - it works, but is unstable (see below).
>>>>>>> 4. If the iSCSI target is on a different physical machine - it all
>>>>>>> works.
>>>>>>> 5. If the iSCSI target is on dom0 itself - it works.
>>>>>>> 6. If I attach the AHCI controller to the Windows VM and install
>>>>>>> directly on the hard drive - it works.
>>>>>>> 7. If the block device for Windows VM is a disk, partition, file, LVM
>>>>>>> volume or even a ZoL's zvol (and it comes from a dom0 itself, without
>>>>>>> iSCSI)- it works.
>>>>>>>
>>>>>>> If I install Windows and the GPLPV drivers on a hard drive
>>>>>>> attached to
>>>>>>> dom0, Windows + GPLPV work perfectly. If I then give the same hard
>>>>>>> drive
>>>>>>> as a block device to the storage VM and re-export it through iSCSI,
>>>>>>
>>>>>> (b)
>>>>>>
>>>>>>> Windows usually boots fine, but works unstable. And by unstable I
>>>>>>> mean
>>>>>>> random read/write errors, sometimes programs won't start, ntdll.dll
>>>>>>> crashes, and after couple reboots Windows won't boot (just like
>>>>>>> mentioned above).
>>>>>>>
>>>>>>> The configurations I would like to achieve makes sense only with PV
>>>>>>> drivers on both storage and Windows VM. All of the "components"
>>>>>>> seem to
>>>>>>> work perfectly until all put together, so I am not really sure where
>>>>>>> the
>>>>>>> problem is.
>>>>>>>
>>>>>>> I would be very grateful for any suggestions or ideas that could
>>>>>>> possibly help to narrow down the problem. Maybe I am just doing
>>>>>>> something wrong (I hope so). Or maybe there is a bug that shows
>>>>>>> itself
>>>>>>> only in such a particular configuration (hope not)?
>>>>>>>
>>>>>>
>>>>>> I'm curious about prompting for the pvscsi drivers to be installed. Is
>>>>>> this definitely what it is asking for? Pvscsi for gplpv is removed in
>>>>>> the latest versions and suffered varying degrees of bitrot in earlier
>>>>>> versions. If you have the iscsi initiator in dom0 then exporting a
>>>>>> block device to windows via the normal vbd channel should be just
>>>>>> fine.
>>>>>>
>>>>>> You've gone to great lengths to explain the various things you've
>>>>>> tried, but I think I'm a little confused on where the iscsi initiator
>>>>>> is in the "doesn't work" scenarios. I'm having a bit of an off day
>>>>>> today so it's probably just me, but above I have highlighted the two
>>>>>> scenarios... could you fill me in on a few things:
>>>>>>
>>>>>> At (a) and (b), is the iscsi initiator in dom0, or are you actually
>>>>>> booting windows directly via iscsi?
>>>>>>
>>>>>> At (b), with latest debug build of gplpv, can you run debugview from
>>>>>> sysinternals.com and see if any interesting messages are displayed
>>>>>> before things fall in a heap?
>>>>>>
>>>>>> Are any strange logs shown in any of Win DomU, Dom0, or storage DomU?
>>>>>>
>>>>>> How big are your disks?
>>>>>>
>>>>>> Can you reproduce with only one vcpu?
>>>>>>
>>>>>> What bridge are you using? Openvswitch or traditional linux bridge?
>>>>>>
>>>>>> What MTU are you using on your storage network? If you are using Jumbo
>>>>>> frames can you go back to 1500 (or at least <= 4000)?
>>>>>>
>>>>>> Can you turn off scatter gather, Large Send Offload (GSO), and IP
>>>>>> Checksum offload on all the iscsi endpoints?
>>>>>>
>>>>>> Can you turn on data digest/checksum on iscsi? If all endpoints
>>>>>> support it then this would provide additional verification that none
>>>>>> of the network packets are getting corrupted.
>>>>>>
>>>>>> Would driver domain work in your scenario? Then the disk could be
>>>>>> attached directly from your storage DomU without accruing all the
>>>>>> iscsi overhead. I'm not up with the status of HVM, vbd, and driver
>>>>>> domain so I don't know if this is possible.
>>>>>>
>>>>>> More questions than answers. Sorry :)
>>>>>>
>>>>>> James
>>>>>
>>>>> Dear James,
>>>>>
>>>>> thank you for your questions - I really appreciate everything that may
>>>>> help me move closer to solving or isolating the problem.
>>>>>
>>>>> I'll check what type of driver is used exactly - up until now I always
>>>>> just installed all drivers included in the package, I thought all of
>>>>> them were necessary. I'll try installing them without XenScsi.
>>>>>
>>>>> Do you mean revisions > 1092:85b99b9795a6 by "the latest versions"?
>>>>> Which version should I use?
>>>>>
>>>>> Forgive me if the descriptions were unclear. The initiator was
>>>>> always in
>>>>> dom0. I only moved the target to dom0 or a separate physical machine in
>>>>> (4) and (5). I didn't boot Windows directly from iSCSI (in fact I tried
>>>>> couple times, but had some problems with it, so I didn't mention it).
>>>>>
>>>>> My "disks" (the block devices I dedicated to the Windows VM) were whole
>>>>> 120GB and 240GB SSDs, ~100GB ZVOLs and 50GB LVM volumes.
>>>>>
>>>>> I'm using traditional linux bridge. I didn't set MTUs explicitly, so I
>>>>> assume it's 1500, but I will verify this.
>>>>>
>>>>> I'd love to use a storage driver domain, but the wiki says "It is not
>>>>> possible to use driver domains with pygrub or HVM guests yet". But the
>>>>> page is a couple of months old, maybe it's an outdated info? It surely
>>>>> is worth checking out.
>>>>>
>>>>> I'll do my best to provide answers to the remaining questions as
>>>>> soon as
>>>>> possible. Thank you for so many ideas.
>>>>>
>>>>> Best regards,
>>>>> Kuba
>>>>>
>>>>> _______________________________________________
>>>>> Xen-users mailing list
>>>>> [hidden email]
>>>>> http://lists.xen.org/xen-users
>>>>
>>>> It seems the problems are not related to GPLPV. There is an easy way to
>>>> reproduce the issues without Windows and without installing anything,
>>>> using only livecds for two DomUs:
>>>>
>>>> 1) Set up a Linux Dom0 with Xen 4.3.1 and standard Linux bridge for Dom0
>>>> and DomUs
>>>
>>> Are you using a Xen build with debugging enabled? I think I might have a
>>> clue of what's happening, because I also saw it. Could you recompile Xen
>>> with debugging enabled and try the same test (iSCSI target on DomU and
>>> initiator on Dom0)?
>>>
>>> Roger.
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> [hidden email]
>>> http://lists.xen.org/xen-users
>>>
>>
>> Of course I could! Please point me to any relevant information on how to
>> build Xen with debugging enabled and what to do next. I build Xen using
>> standard ./configure && make world && make install.
>
> Just `make debug=y xen` and boot with the resulting xen.gz.
>
> Roger.
>
>
> _______________________________________________
> Xen-users mailing list
> [hidden email]
> http://lists.xen.org/xen-users
>
I ran the test using debug build of Xen. This time I gave the name "tgt"
to the DomU with iSCSI target, and the other domain was named simply
"domu". Sorry for the inconsistency. After logging in to the iSCSI
target from Dom0, I ran "mkfs.ext4 /dev/sdb" (still in Dom0). So far, so
good. Then I launched the other DomU and as soon as I executed
"fsck.ext4 /dev/xvda", some errors appeared in the output of "xl dmesg"
(attached as "xl-dmesg.log"). Surprisingly, the first fsck succeeded.
Unfortunately, executing fsck.ext4 for the second time showed serious
file system errors. The fsck commands were the only things I ran that
touched /dev/xvda. After shutting down "domu", when I tried to log out
from the iSCSI target, an error came up in Dom0's dmesg
("dom0-dmesg.log"). Logs from /var/log/xen/ are also attached.

I will happily run next tests - just tell me what can I do :)

Best regards,
Kuba

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users

xen-debug-logs.zip (31K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

James Harper
In reply to this post by Kuba

>
> Any ideas and suggestions on what could be the cause and what can I do
> will be much appreciated.
>

Can you try turning on iscsi data checksuming? If data corruption in the network stack is the cause (seems likely to me) then this should pick it up.

If your iscsi initiator and/or target don't support data digest, try turning off all the offloads and scatter gather on the network interfaces and see if you can reduce the problem to one of the offloads (large send would be my best guess).

James
_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Roger Pau Monné-3
In reply to this post by Kuba
On 05/02/14 22:54, Kuba wrote:

> W dniu 2014-02-05 17:54, Roger Pau Monné pisze:
>> On 05/02/14 17:43, Kuba wrote:
>>> W dniu 2014-02-05 17:29, Roger Pau Monné pisze:
>>>> On 05/02/14 17:13, Kuba wrote:
>>>>> W dniu 2014-02-01 20:27, Kuba pisze:
>>>>>> W dniu 2014-01-31 02:35, James Harper pisze:
>>>>>>>>
>>>>>>>> I am trying to set up a following configuration:
>>>>>>>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1
>>>>>>>> compiled
>>>>>>>> from sources,
>>>>>>>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller
>>>>>>>> attached
>>>>>>>> using VT-d, exporting block devices via iSCSI to other VMs and
>>>>>>>> physical
>>>>>>>> machines,
>>>>>>>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>>>>>>>> 4000) installed on a block device exported from the storage VM
>>>>>>>> (target
>>>>>>>> on the storage VM, initiator on dom0).
>>>>>>>>
>>>>>>>> Everything works perfectly (including PCI & GPU passthrough)
>>>>>>>> until I
>>>>>>>> install GPLPV drivers on the Windows VM. After driver installation,
>>>>>>>> Windows needs to reboot, boots fine, displays a message that PV
>>>>>>>> SCSI
>>>>>>>
>>>>>>> (a)
>>>>>>>
>>>>>>>> drivers were installed and needs to reboot again, and then cannot
>>>>>>>> boot.
>>>>>>>> Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>>>>>>>> sometimes BSODs with "unmountable boot volume" message. All of the
>>>>>>>> following I tried without GPU passthrough to narrow down the
>>>>>>>> problem.
>>>>>>>>
>>>>>>>> The intriguing part is this:
>>>>>>>>
>>>>>>>> 1. If the storage VM's OS is Linux - it fails with the above
>>>>>>>> symptoms.
>>>>>>>> 2. If the block devices for the storage VM come directly from dom0
>>>>>>>> (not
>>>>>>>> via pci-passthrough) - it fails.
>>>>>>>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>>>>>>>> 9.2-GENERIC) - it all works.
>>>>>>>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>>>>>>>> guest support - it works, but is unstable (see below).
>>>>>>>> 4. If the iSCSI target is on a different physical machine - it all
>>>>>>>> works.
>>>>>>>> 5. If the iSCSI target is on dom0 itself - it works.
>>>>>>>> 6. If I attach the AHCI controller to the Windows VM and install
>>>>>>>> directly on the hard drive - it works.
>>>>>>>> 7. If the block device for Windows VM is a disk, partition,
>>>>>>>> file, LVM
>>>>>>>> volume or even a ZoL's zvol (and it comes from a dom0 itself,
>>>>>>>> without
>>>>>>>> iSCSI)- it works.
>>>>>>>>
>>>>>>>> If I install Windows and the GPLPV drivers on a hard drive
>>>>>>>> attached to
>>>>>>>> dom0, Windows + GPLPV work perfectly. If I then give the same hard
>>>>>>>> drive
>>>>>>>> as a block device to the storage VM and re-export it through iSCSI,
>>>>>>>
>>>>>>> (b)
>>>>>>>
>>>>>>>> Windows usually boots fine, but works unstable. And by unstable I
>>>>>>>> mean
>>>>>>>> random read/write errors, sometimes programs won't start, ntdll.dll
>>>>>>>> crashes, and after couple reboots Windows won't boot (just like
>>>>>>>> mentioned above).
>>>>>>>>
>>>>>>>> The configurations I would like to achieve makes sense only with PV
>>>>>>>> drivers on both storage and Windows VM. All of the "components"
>>>>>>>> seem to
>>>>>>>> work perfectly until all put together, so I am not really sure
>>>>>>>> where
>>>>>>>> the
>>>>>>>> problem is.
>>>>>>>>
>>>>>>>> I would be very grateful for any suggestions or ideas that could
>>>>>>>> possibly help to narrow down the problem. Maybe I am just doing
>>>>>>>> something wrong (I hope so). Or maybe there is a bug that shows
>>>>>>>> itself
>>>>>>>> only in such a particular configuration (hope not)?
>>>>>>>>
>>>>>>>
>>>>>>> I'm curious about prompting for the pvscsi drivers to be
>>>>>>> installed. Is
>>>>>>> this definitely what it is asking for? Pvscsi for gplpv is
>>>>>>> removed in
>>>>>>> the latest versions and suffered varying degrees of bitrot in
>>>>>>> earlier
>>>>>>> versions. If you have the iscsi initiator in dom0 then exporting a
>>>>>>> block device to windows via the normal vbd channel should be just
>>>>>>> fine.
>>>>>>>
>>>>>>> You've gone to great lengths to explain the various things you've
>>>>>>> tried, but I think I'm a little confused on where the iscsi
>>>>>>> initiator
>>>>>>> is in the "doesn't work" scenarios. I'm having a bit of an off day
>>>>>>> today so it's probably just me, but above I have highlighted the two
>>>>>>> scenarios... could you fill me in on a few things:
>>>>>>>
>>>>>>> At (a) and (b), is the iscsi initiator in dom0, or are you actually
>>>>>>> booting windows directly via iscsi?
>>>>>>>
>>>>>>> At (b), with latest debug build of gplpv, can you run debugview from
>>>>>>> sysinternals.com and see if any interesting messages are displayed
>>>>>>> before things fall in a heap?
>>>>>>>
>>>>>>> Are any strange logs shown in any of Win DomU, Dom0, or storage
>>>>>>> DomU?
>>>>>>>
>>>>>>> How big are your disks?
>>>>>>>
>>>>>>> Can you reproduce with only one vcpu?
>>>>>>>
>>>>>>> What bridge are you using? Openvswitch or traditional linux bridge?
>>>>>>>
>>>>>>> What MTU are you using on your storage network? If you are using
>>>>>>> Jumbo
>>>>>>> frames can you go back to 1500 (or at least <= 4000)?
>>>>>>>
>>>>>>> Can you turn off scatter gather, Large Send Offload (GSO), and IP
>>>>>>> Checksum offload on all the iscsi endpoints?
>>>>>>>
>>>>>>> Can you turn on data digest/checksum on iscsi? If all endpoints
>>>>>>> support it then this would provide additional verification that none
>>>>>>> of the network packets are getting corrupted.
>>>>>>>
>>>>>>> Would driver domain work in your scenario? Then the disk could be
>>>>>>> attached directly from your storage DomU without accruing all the
>>>>>>> iscsi overhead. I'm not up with the status of HVM, vbd, and driver
>>>>>>> domain so I don't know if this is possible.
>>>>>>>
>>>>>>> More questions than answers. Sorry :)
>>>>>>>
>>>>>>> James
>>>>>>
>>>>>> Dear James,
>>>>>>
>>>>>> thank you for your questions - I really appreciate everything that
>>>>>> may
>>>>>> help me move closer to solving or isolating the problem.
>>>>>>
>>>>>> I'll check what type of driver is used exactly - up until now I
>>>>>> always
>>>>>> just installed all drivers included in the package, I thought all of
>>>>>> them were necessary. I'll try installing them without XenScsi.
>>>>>>
>>>>>> Do you mean revisions > 1092:85b99b9795a6 by "the latest versions"?
>>>>>> Which version should I use?
>>>>>>
>>>>>> Forgive me if the descriptions were unclear. The initiator was
>>>>>> always in
>>>>>> dom0. I only moved the target to dom0 or a separate physical
>>>>>> machine in
>>>>>> (4) and (5). I didn't boot Windows directly from iSCSI (in fact I
>>>>>> tried
>>>>>> couple times, but had some problems with it, so I didn't mention it).
>>>>>>
>>>>>> My "disks" (the block devices I dedicated to the Windows VM) were
>>>>>> whole
>>>>>> 120GB and 240GB SSDs, ~100GB ZVOLs and 50GB LVM volumes.
>>>>>>
>>>>>> I'm using traditional linux bridge. I didn't set MTUs explicitly,
>>>>>> so I
>>>>>> assume it's 1500, but I will verify this.
>>>>>>
>>>>>> I'd love to use a storage driver domain, but the wiki says "It is not
>>>>>> possible to use driver domains with pygrub or HVM guests yet". But
>>>>>> the
>>>>>> page is a couple of months old, maybe it's an outdated info? It
>>>>>> surely
>>>>>> is worth checking out.
>>>>>>
>>>>>> I'll do my best to provide answers to the remaining questions as
>>>>>> soon as
>>>>>> possible. Thank you for so many ideas.
>>>>>>
>>>>>> Best regards,
>>>>>> Kuba
>>>>>>
>>>>>> _______________________________________________
>>>>>> Xen-users mailing list
>>>>>> [hidden email]
>>>>>> http://lists.xen.org/xen-users
>>>>>
>>>>> It seems the problems are not related to GPLPV. There is an easy
>>>>> way to
>>>>> reproduce the issues without Windows and without installing anything,
>>>>> using only livecds for two DomUs:
>>>>>
>>>>> 1) Set up a Linux Dom0 with Xen 4.3.1 and standard Linux bridge for
>>>>> Dom0
>>>>> and DomUs
>>>>
>>>> Are you using a Xen build with debugging enabled? I think I might
>>>> have a
>>>> clue of what's happening, because I also saw it. Could you recompile
>>>> Xen
>>>> with debugging enabled and try the same test (iSCSI target on DomU and
>>>> initiator on Dom0)?
>>>>
>>>> Roger.
>>>>
>>>> _______________________________________________
>>>> Xen-users mailing list
>>>> [hidden email]
>>>> http://lists.xen.org/xen-users
>>>>
>>>
>>> Of course I could! Please point me to any relevant information on how to
>>> build Xen with debugging enabled and what to do next. I build Xen using
>>> standard ./configure && make world && make install.
>>
>> Just `make debug=y xen` and boot with the resulting xen.gz.
>>
>> Roger.
>>
>>
>> _______________________________________________
>> Xen-users mailing list
>> [hidden email]
>> http://lists.xen.org/xen-users
>>
>
> I ran the test using debug build of Xen. This time I gave the name "tgt"
> to the DomU with iSCSI target, and the other domain was named simply
> "domu". Sorry for the inconsistency. After logging in to the iSCSI
> target from Dom0, I ran "mkfs.ext4 /dev/sdb" (still in Dom0). So far, so
> good. Then I launched the other DomU and as soon as I executed
> "fsck.ext4 /dev/xvda", some errors appeared in the output of "xl dmesg"
> (attached as "xl-dmesg.log"). Surprisingly, the first fsck succeeded.
> Unfortunately, executing fsck.ext4 for the second time showed serious
> file system errors. The fsck commands were the only things I ran that
> touched /dev/xvda. After shutting down "domu", when I tried to log out
> from the iSCSI target, an error came up in Dom0's dmesg
> ("dom0-dmesg.log"). Logs from /var/log/xen/ are also attached.
>
> I will happily run next tests - just tell me what can I do :)

Hello,

This is the same problem I've seen when using a similar setup. The root
of the problem is that blkback maps a grant ref to a memory page in
Dom0, then this memory page ends up in netback, and when netback tries
to issue a GNTTABOP_copy using the mfn of this grant mapped page the
operation fails because Xen detects that the mfn passed doesn't belong
to the guest.

The only way I can think of solving this is that netback detects that
the page is not local and somehow we use it's grant ref instead of mfn
(this means we would need to store the grant ref somewhere in the page).

Roger.


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Kuba
In reply to this post by James Harper
W dniu 2014-02-06 00:07, James Harper pisze:

>
>>
>> Any ideas and suggestions on what could be the cause and what can I do
>> will be much appreciated.
>>
>
> Can you try turning on iscsi data checksuming? If data corruption in the network stack is the cause (seems likely to me) then this should pick it up.
>
> If your iscsi initiator and/or target don't support data digest, try turning off all the offloads and scatter gather on the network interfaces and see if you can reduce the problem to one of the offloads (large send would be my best guess).
>
> James
> _______________________________________________
> Xen-users mailing list
> [hidden email]
> http://lists.xen.org/xen-users
>


I haven't noticed any change after turning off all offloads etc.
reported by ethtool. Sorry, I forgot to mention that earlier.

I couldn't get Open-iSCSI and IET to cooperate with each other with
digests enabled, so I tried with FreeBSD's native target, which "just
worked" - well, at least until I launched the second DomU ;) The logs
are in "DomU-A-FreeBSD10.log" attached in one of my previous posts.

If I understand correctly, the iSCSI-related issues were only symptoms
caused by something Roger seems to have found.

I'm very grateful to you guys for your time and effort to solve my
little problem and (especially) for making Xen what it is today.

Kuba

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Kuba
In reply to this post by Roger Pau Monné-3
W dniu 2014-02-06 09:22, Roger Pau Monné pisze:

> On 05/02/14 22:54, Kuba wrote:
>> W dniu 2014-02-05 17:54, Roger Pau Monné pisze:
>>> On 05/02/14 17:43, Kuba wrote:
>>>> W dniu 2014-02-05 17:29, Roger Pau Monné pisze:
>>>>> On 05/02/14 17:13, Kuba wrote:
>>>>>> W dniu 2014-02-01 20:27, Kuba pisze:
>>>>>>> W dniu 2014-01-31 02:35, James Harper pisze:
>>>>>>>>>
>>>>>>>>> I am trying to set up a following configuration:
>>>>>>>>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1
>>>>>>>>> compiled
>>>>>>>>> from sources,
>>>>>>>>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller
>>>>>>>>> attached
>>>>>>>>> using VT-d, exporting block devices via iSCSI to other VMs and
>>>>>>>>> physical
>>>>>>>>> machines,
>>>>>>>>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough (Quadro
>>>>>>>>> 4000) installed on a block device exported from the storage VM
>>>>>>>>> (target
>>>>>>>>> on the storage VM, initiator on dom0).
>>>>>>>>>
>>>>>>>>> Everything works perfectly (including PCI & GPU passthrough)
>>>>>>>>> until I
>>>>>>>>> install GPLPV drivers on the Windows VM. After driver installation,
>>>>>>>>> Windows needs to reboot, boots fine, displays a message that PV
>>>>>>>>> SCSI
>>>>>>>>
>>>>>>>> (a)
>>>>>>>>
>>>>>>>>> drivers were installed and needs to reboot again, and then cannot
>>>>>>>>> boot.
>>>>>>>>> Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>>>>>>>>> sometimes BSODs with "unmountable boot volume" message. All of the
>>>>>>>>> following I tried without GPU passthrough to narrow down the
>>>>>>>>> problem.
>>>>>>>>>
>>>>>>>>> The intriguing part is this:
>>>>>>>>>
>>>>>>>>> 1. If the storage VM's OS is Linux - it fails with the above
>>>>>>>>> symptoms.
>>>>>>>>> 2. If the block devices for the storage VM come directly from dom0
>>>>>>>>> (not
>>>>>>>>> via pci-passthrough) - it fails.
>>>>>>>>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>>>>>>>>> 9.2-GENERIC) - it all works.
>>>>>>>>> 3. If the storage VM's OS is Linux with kernel compiled without Xen
>>>>>>>>> guest support - it works, but is unstable (see below).
>>>>>>>>> 4. If the iSCSI target is on a different physical machine - it all
>>>>>>>>> works.
>>>>>>>>> 5. If the iSCSI target is on dom0 itself - it works.
>>>>>>>>> 6. If I attach the AHCI controller to the Windows VM and install
>>>>>>>>> directly on the hard drive - it works.
>>>>>>>>> 7. If the block device for Windows VM is a disk, partition,
>>>>>>>>> file, LVM
>>>>>>>>> volume or even a ZoL's zvol (and it comes from a dom0 itself,
>>>>>>>>> without
>>>>>>>>> iSCSI)- it works.
>>>>>>>>>
>>>>>>>>> If I install Windows and the GPLPV drivers on a hard drive
>>>>>>>>> attached to
>>>>>>>>> dom0, Windows + GPLPV work perfectly. If I then give the same hard
>>>>>>>>> drive
>>>>>>>>> as a block device to the storage VM and re-export it through iSCSI,
>>>>>>>>
>>>>>>>> (b)
>>>>>>>>
>>>>>>>>> Windows usually boots fine, but works unstable. And by unstable I
>>>>>>>>> mean
>>>>>>>>> random read/write errors, sometimes programs won't start, ntdll.dll
>>>>>>>>> crashes, and after couple reboots Windows won't boot (just like
>>>>>>>>> mentioned above).
>>>>>>>>>
>>>>>>>>> The configurations I would like to achieve makes sense only with PV
>>>>>>>>> drivers on both storage and Windows VM. All of the "components"
>>>>>>>>> seem to
>>>>>>>>> work perfectly until all put together, so I am not really sure
>>>>>>>>> where
>>>>>>>>> the
>>>>>>>>> problem is.
>>>>>>>>>
>>>>>>>>> I would be very grateful for any suggestions or ideas that could
>>>>>>>>> possibly help to narrow down the problem. Maybe I am just doing
>>>>>>>>> something wrong (I hope so). Or maybe there is a bug that shows
>>>>>>>>> itself
>>>>>>>>> only in such a particular configuration (hope not)?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm curious about prompting for the pvscsi drivers to be
>>>>>>>> installed. Is
>>>>>>>> this definitely what it is asking for? Pvscsi for gplpv is
>>>>>>>> removed in
>>>>>>>> the latest versions and suffered varying degrees of bitrot in
>>>>>>>> earlier
>>>>>>>> versions. If you have the iscsi initiator in dom0 then exporting a
>>>>>>>> block device to windows via the normal vbd channel should be just
>>>>>>>> fine.
>>>>>>>>
>>>>>>>> You've gone to great lengths to explain the various things you've
>>>>>>>> tried, but I think I'm a little confused on where the iscsi
>>>>>>>> initiator
>>>>>>>> is in the "doesn't work" scenarios. I'm having a bit of an off day
>>>>>>>> today so it's probably just me, but above I have highlighted the two
>>>>>>>> scenarios... could you fill me in on a few things:
>>>>>>>>
>>>>>>>> At (a) and (b), is the iscsi initiator in dom0, or are you actually
>>>>>>>> booting windows directly via iscsi?
>>>>>>>>
>>>>>>>> At (b), with latest debug build of gplpv, can you run debugview from
>>>>>>>> sysinternals.com and see if any interesting messages are displayed
>>>>>>>> before things fall in a heap?
>>>>>>>>
>>>>>>>> Are any strange logs shown in any of Win DomU, Dom0, or storage
>>>>>>>> DomU?
>>>>>>>>
>>>>>>>> How big are your disks?
>>>>>>>>
>>>>>>>> Can you reproduce with only one vcpu?
>>>>>>>>
>>>>>>>> What bridge are you using? Openvswitch or traditional linux bridge?
>>>>>>>>
>>>>>>>> What MTU are you using on your storage network? If you are using
>>>>>>>> Jumbo
>>>>>>>> frames can you go back to 1500 (or at least <= 4000)?
>>>>>>>>
>>>>>>>> Can you turn off scatter gather, Large Send Offload (GSO), and IP
>>>>>>>> Checksum offload on all the iscsi endpoints?
>>>>>>>>
>>>>>>>> Can you turn on data digest/checksum on iscsi? If all endpoints
>>>>>>>> support it then this would provide additional verification that none
>>>>>>>> of the network packets are getting corrupted.
>>>>>>>>
>>>>>>>> Would driver domain work in your scenario? Then the disk could be
>>>>>>>> attached directly from your storage DomU without accruing all the
>>>>>>>> iscsi overhead. I'm not up with the status of HVM, vbd, and driver
>>>>>>>> domain so I don't know if this is possible.
>>>>>>>>
>>>>>>>> More questions than answers. Sorry :)
>>>>>>>>
>>>>>>>> James
>>>>>>>
>>>>>>> Dear James,
>>>>>>>
>>>>>>> thank you for your questions - I really appreciate everything that
>>>>>>> may
>>>>>>> help me move closer to solving or isolating the problem.
>>>>>>>
>>>>>>> I'll check what type of driver is used exactly - up until now I
>>>>>>> always
>>>>>>> just installed all drivers included in the package, I thought all of
>>>>>>> them were necessary. I'll try installing them without XenScsi.
>>>>>>>
>>>>>>> Do you mean revisions > 1092:85b99b9795a6 by "the latest versions"?
>>>>>>> Which version should I use?
>>>>>>>
>>>>>>> Forgive me if the descriptions were unclear. The initiator was
>>>>>>> always in
>>>>>>> dom0. I only moved the target to dom0 or a separate physical
>>>>>>> machine in
>>>>>>> (4) and (5). I didn't boot Windows directly from iSCSI (in fact I
>>>>>>> tried
>>>>>>> couple times, but had some problems with it, so I didn't mention it).
>>>>>>>
>>>>>>> My "disks" (the block devices I dedicated to the Windows VM) were
>>>>>>> whole
>>>>>>> 120GB and 240GB SSDs, ~100GB ZVOLs and 50GB LVM volumes.
>>>>>>>
>>>>>>> I'm using traditional linux bridge. I didn't set MTUs explicitly,
>>>>>>> so I
>>>>>>> assume it's 1500, but I will verify this.
>>>>>>>
>>>>>>> I'd love to use a storage driver domain, but the wiki says "It is not
>>>>>>> possible to use driver domains with pygrub or HVM guests yet". But
>>>>>>> the
>>>>>>> page is a couple of months old, maybe it's an outdated info? It
>>>>>>> surely
>>>>>>> is worth checking out.
>>>>>>>
>>>>>>> I'll do my best to provide answers to the remaining questions as
>>>>>>> soon as
>>>>>>> possible. Thank you for so many ideas.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Kuba
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Xen-users mailing list
>>>>>>> [hidden email]
>>>>>>> http://lists.xen.org/xen-users
>>>>>>
>>>>>> It seems the problems are not related to GPLPV. There is an easy
>>>>>> way to
>>>>>> reproduce the issues without Windows and without installing anything,
>>>>>> using only livecds for two DomUs:
>>>>>>
>>>>>> 1) Set up a Linux Dom0 with Xen 4.3.1 and standard Linux bridge for
>>>>>> Dom0
>>>>>> and DomUs
>>>>>
>>>>> Are you using a Xen build with debugging enabled? I think I might
>>>>> have a
>>>>> clue of what's happening, because I also saw it. Could you recompile
>>>>> Xen
>>>>> with debugging enabled and try the same test (iSCSI target on DomU and
>>>>> initiator on Dom0)?
>>>>>
>>>>> Roger.
>>>>>
>>>>> _______________________________________________
>>>>> Xen-users mailing list
>>>>> [hidden email]
>>>>> http://lists.xen.org/xen-users
>>>>>
>>>>
>>>> Of course I could! Please point me to any relevant information on how to
>>>> build Xen with debugging enabled and what to do next. I build Xen using
>>>> standard ./configure && make world && make install.
>>>
>>> Just `make debug=y xen` and boot with the resulting xen.gz.
>>>
>>> Roger.
>>>
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> [hidden email]
>>> http://lists.xen.org/xen-users
>>>
>>
>> I ran the test using debug build of Xen. This time I gave the name "tgt"
>> to the DomU with iSCSI target, and the other domain was named simply
>> "domu". Sorry for the inconsistency. After logging in to the iSCSI
>> target from Dom0, I ran "mkfs.ext4 /dev/sdb" (still in Dom0). So far, so
>> good. Then I launched the other DomU and as soon as I executed
>> "fsck.ext4 /dev/xvda", some errors appeared in the output of "xl dmesg"
>> (attached as "xl-dmesg.log"). Surprisingly, the first fsck succeeded.
>> Unfortunately, executing fsck.ext4 for the second time showed serious
>> file system errors. The fsck commands were the only things I ran that
>> touched /dev/xvda. After shutting down "domu", when I tried to log out
>> from the iSCSI target, an error came up in Dom0's dmesg
>> ("dom0-dmesg.log"). Logs from /var/log/xen/ are also attached.
>>
>> I will happily run next tests - just tell me what can I do :)
>
> Hello,
>
> This is the same problem I've seen when using a similar setup. The root
> of the problem is that blkback maps a grant ref to a memory page in
> Dom0, then this memory page ends up in netback, and when netback tries
> to issue a GNTTABOP_copy using the mfn of this grant mapped page the
> operation fails because Xen detects that the mfn passed doesn't belong
> to the guest.
>
> The only way I can think of solving this is that netback detects that
> the page is not local and somehow we use it's grant ref instead of mfn
> (this means we would need to store the grant ref somewhere in the page).
>
> Roger.
As this is something far beyond my ability to solve, I couldn't resist
to try something else - running FreeBSD 10 as a storage driver domain. I
was able to provide a block device (zvol) from one FreeBSD DomU directly
to another FreeBSD DomU just like described in the wiki (with Qemu
traditional in the second DomU) and install the OS on it. Unfortunately
the second DomU's bios was unable to detect this "disk" and boot from it.

But with this command:
xl block-attach Domain-0
"format=raw,backendtype=phy,backend=fbsd,vdev=xvds,target=/dev/zvol/zroot/vol1"

I was able to attach a block device exported from a DomU to Dom0 without
iSCSI and then using it as a disk for a second DomU (with
disk=['phy:/dev/xvds,xvda,w']). Now, if the second DomU had no PV
drivers (e.g. Windows without GPLPV), everything worked fine. But
running an OS with PV drivers (Linux or Windows+GPLPV) in the second
DomU resulted in a very similar errors in xl dmesg like in the
previously attached logs (see the attachment).

Do I understand correctly that solving the issue you are pointing out
would also allow to use OSes like FreeBSD as storage driver domain for
other PV-enabled DomUs? That would be something!

And most importantly - is there anything I can do to help?

Best regards,
Kuba

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users

xl-dmesg-2.log (87K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Roger Pau Monné-3
On 06/02/14 23:32, Kuba wrote:

> W dniu 2014-02-06 09:22, Roger Pau Monné pisze:
>> On 05/02/14 22:54, Kuba wrote:
>>> W dniu 2014-02-05 17:54, Roger Pau Monné pisze:
>>>> On 05/02/14 17:43, Kuba wrote:
>>>>> W dniu 2014-02-05 17:29, Roger Pau Monné pisze:
>>>>>> On 05/02/14 17:13, Kuba wrote:
>>>>>>> W dniu 2014-02-01 20:27, Kuba pisze:
>>>>>>>> W dniu 2014-01-31 02:35, James Harper pisze:
>>>>>>>>>>
>>>>>>>>>> I am trying to set up a following configuration:
>>>>>>>>>> 1. very simple Linux-based dom0 (Debian 7.3) with Xen 4.3.1
>>>>>>>>>> compiled
>>>>>>>>>> from sources,
>>>>>>>>>> 2. one storage VM (FreeBSD 10, HVM+PV) with SATA controller
>>>>>>>>>> attached
>>>>>>>>>> using VT-d, exporting block devices via iSCSI to other VMs and
>>>>>>>>>> physical
>>>>>>>>>> machines,
>>>>>>>>>> 3. one Windows 7 SP1 64 VM (HVM+GPLPV) with GPU passthrough
>>>>>>>>>> (Quadro
>>>>>>>>>> 4000) installed on a block device exported from the storage VM
>>>>>>>>>> (target
>>>>>>>>>> on the storage VM, initiator on dom0).
>>>>>>>>>>
>>>>>>>>>> Everything works perfectly (including PCI & GPU passthrough)
>>>>>>>>>> until I
>>>>>>>>>> install GPLPV drivers on the Windows VM. After driver
>>>>>>>>>> installation,
>>>>>>>>>> Windows needs to reboot, boots fine, displays a message that PV
>>>>>>>>>> SCSI
>>>>>>>>>
>>>>>>>>> (a)
>>>>>>>>>
>>>>>>>>>> drivers were installed and needs to reboot again, and then cannot
>>>>>>>>>> boot.
>>>>>>>>>> Sometimes it gets stuck at "booting from harddrive" in SeaBIOS,
>>>>>>>>>> sometimes BSODs with "unmountable boot volume" message. All of
>>>>>>>>>> the
>>>>>>>>>> following I tried without GPU passthrough to narrow down the
>>>>>>>>>> problem.
>>>>>>>>>>
>>>>>>>>>> The intriguing part is this:
>>>>>>>>>>
>>>>>>>>>> 1. If the storage VM's OS is Linux - it fails with the above
>>>>>>>>>> symptoms.
>>>>>>>>>> 2. If the block devices for the storage VM come directly from
>>>>>>>>>> dom0
>>>>>>>>>> (not
>>>>>>>>>> via pci-passthrough) - it fails.
>>>>>>>>>> 2. If the storage VM is an HVM without PV drivers (e.g. FreeBSD
>>>>>>>>>> 9.2-GENERIC) - it all works.
>>>>>>>>>> 3. If the storage VM's OS is Linux with kernel compiled
>>>>>>>>>> without Xen
>>>>>>>>>> guest support - it works, but is unstable (see below).
>>>>>>>>>> 4. If the iSCSI target is on a different physical machine - it
>>>>>>>>>> all
>>>>>>>>>> works.
>>>>>>>>>> 5. If the iSCSI target is on dom0 itself - it works.
>>>>>>>>>> 6. If I attach the AHCI controller to the Windows VM and install
>>>>>>>>>> directly on the hard drive - it works.
>>>>>>>>>> 7. If the block device for Windows VM is a disk, partition,
>>>>>>>>>> file, LVM
>>>>>>>>>> volume or even a ZoL's zvol (and it comes from a dom0 itself,
>>>>>>>>>> without
>>>>>>>>>> iSCSI)- it works.
>>>>>>>>>>
>>>>>>>>>> If I install Windows and the GPLPV drivers on a hard drive
>>>>>>>>>> attached to
>>>>>>>>>> dom0, Windows + GPLPV work perfectly. If I then give the same
>>>>>>>>>> hard
>>>>>>>>>> drive
>>>>>>>>>> as a block device to the storage VM and re-export it through
>>>>>>>>>> iSCSI,
>>>>>>>>>
>>>>>>>>> (b)
>>>>>>>>>
>>>>>>>>>> Windows usually boots fine, but works unstable. And by unstable I
>>>>>>>>>> mean
>>>>>>>>>> random read/write errors, sometimes programs won't start,
>>>>>>>>>> ntdll.dll
>>>>>>>>>> crashes, and after couple reboots Windows won't boot (just like
>>>>>>>>>> mentioned above).
>>>>>>>>>>
>>>>>>>>>> The configurations I would like to achieve makes sense only
>>>>>>>>>> with PV
>>>>>>>>>> drivers on both storage and Windows VM. All of the "components"
>>>>>>>>>> seem to
>>>>>>>>>> work perfectly until all put together, so I am not really sure
>>>>>>>>>> where
>>>>>>>>>> the
>>>>>>>>>> problem is.
>>>>>>>>>>
>>>>>>>>>> I would be very grateful for any suggestions or ideas that could
>>>>>>>>>> possibly help to narrow down the problem. Maybe I am just doing
>>>>>>>>>> something wrong (I hope so). Or maybe there is a bug that shows
>>>>>>>>>> itself
>>>>>>>>>> only in such a particular configuration (hope not)?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm curious about prompting for the pvscsi drivers to be
>>>>>>>>> installed. Is
>>>>>>>>> this definitely what it is asking for? Pvscsi for gplpv is
>>>>>>>>> removed in
>>>>>>>>> the latest versions and suffered varying degrees of bitrot in
>>>>>>>>> earlier
>>>>>>>>> versions. If you have the iscsi initiator in dom0 then exporting a
>>>>>>>>> block device to windows via the normal vbd channel should be just
>>>>>>>>> fine.
>>>>>>>>>
>>>>>>>>> You've gone to great lengths to explain the various things you've
>>>>>>>>> tried, but I think I'm a little confused on where the iscsi
>>>>>>>>> initiator
>>>>>>>>> is in the "doesn't work" scenarios. I'm having a bit of an off day
>>>>>>>>> today so it's probably just me, but above I have highlighted
>>>>>>>>> the two
>>>>>>>>> scenarios... could you fill me in on a few things:
>>>>>>>>>
>>>>>>>>> At (a) and (b), is the iscsi initiator in dom0, or are you
>>>>>>>>> actually
>>>>>>>>> booting windows directly via iscsi?
>>>>>>>>>
>>>>>>>>> At (b), with latest debug build of gplpv, can you run debugview
>>>>>>>>> from
>>>>>>>>> sysinternals.com and see if any interesting messages are displayed
>>>>>>>>> before things fall in a heap?
>>>>>>>>>
>>>>>>>>> Are any strange logs shown in any of Win DomU, Dom0, or storage
>>>>>>>>> DomU?
>>>>>>>>>
>>>>>>>>> How big are your disks?
>>>>>>>>>
>>>>>>>>> Can you reproduce with only one vcpu?
>>>>>>>>>
>>>>>>>>> What bridge are you using? Openvswitch or traditional linux
>>>>>>>>> bridge?
>>>>>>>>>
>>>>>>>>> What MTU are you using on your storage network? If you are using
>>>>>>>>> Jumbo
>>>>>>>>> frames can you go back to 1500 (or at least <= 4000)?
>>>>>>>>>
>>>>>>>>> Can you turn off scatter gather, Large Send Offload (GSO), and IP
>>>>>>>>> Checksum offload on all the iscsi endpoints?
>>>>>>>>>
>>>>>>>>> Can you turn on data digest/checksum on iscsi? If all endpoints
>>>>>>>>> support it then this would provide additional verification that
>>>>>>>>> none
>>>>>>>>> of the network packets are getting corrupted.
>>>>>>>>>
>>>>>>>>> Would driver domain work in your scenario? Then the disk could be
>>>>>>>>> attached directly from your storage DomU without accruing all the
>>>>>>>>> iscsi overhead. I'm not up with the status of HVM, vbd, and driver
>>>>>>>>> domain so I don't know if this is possible.
>>>>>>>>>
>>>>>>>>> More questions than answers. Sorry :)
>>>>>>>>>
>>>>>>>>> James
>>>>>>>>
>>>>>>>> Dear James,
>>>>>>>>
>>>>>>>> thank you for your questions - I really appreciate everything that
>>>>>>>> may
>>>>>>>> help me move closer to solving or isolating the problem.
>>>>>>>>
>>>>>>>> I'll check what type of driver is used exactly - up until now I
>>>>>>>> always
>>>>>>>> just installed all drivers included in the package, I thought
>>>>>>>> all of
>>>>>>>> them were necessary. I'll try installing them without XenScsi.
>>>>>>>>
>>>>>>>> Do you mean revisions > 1092:85b99b9795a6 by "the latest versions"?
>>>>>>>> Which version should I use?
>>>>>>>>
>>>>>>>> Forgive me if the descriptions were unclear. The initiator was
>>>>>>>> always in
>>>>>>>> dom0. I only moved the target to dom0 or a separate physical
>>>>>>>> machine in
>>>>>>>> (4) and (5). I didn't boot Windows directly from iSCSI (in fact I
>>>>>>>> tried
>>>>>>>> couple times, but had some problems with it, so I didn't mention
>>>>>>>> it).
>>>>>>>>
>>>>>>>> My "disks" (the block devices I dedicated to the Windows VM) were
>>>>>>>> whole
>>>>>>>> 120GB and 240GB SSDs, ~100GB ZVOLs and 50GB LVM volumes.
>>>>>>>>
>>>>>>>> I'm using traditional linux bridge. I didn't set MTUs explicitly,
>>>>>>>> so I
>>>>>>>> assume it's 1500, but I will verify this.
>>>>>>>>
>>>>>>>> I'd love to use a storage driver domain, but the wiki says "It
>>>>>>>> is not
>>>>>>>> possible to use driver domains with pygrub or HVM guests yet". But
>>>>>>>> the
>>>>>>>> page is a couple of months old, maybe it's an outdated info? It
>>>>>>>> surely
>>>>>>>> is worth checking out.
>>>>>>>>
>>>>>>>> I'll do my best to provide answers to the remaining questions as
>>>>>>>> soon as
>>>>>>>> possible. Thank you for so many ideas.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Kuba
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Xen-users mailing list
>>>>>>>> [hidden email]
>>>>>>>> http://lists.xen.org/xen-users
>>>>>>>
>>>>>>> It seems the problems are not related to GPLPV. There is an easy
>>>>>>> way to
>>>>>>> reproduce the issues without Windows and without installing
>>>>>>> anything,
>>>>>>> using only livecds for two DomUs:
>>>>>>>
>>>>>>> 1) Set up a Linux Dom0 with Xen 4.3.1 and standard Linux bridge for
>>>>>>> Dom0
>>>>>>> and DomUs
>>>>>>
>>>>>> Are you using a Xen build with debugging enabled? I think I might
>>>>>> have a
>>>>>> clue of what's happening, because I also saw it. Could you recompile
>>>>>> Xen
>>>>>> with debugging enabled and try the same test (iSCSI target on DomU
>>>>>> and
>>>>>> initiator on Dom0)?
>>>>>>
>>>>>> Roger.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Xen-users mailing list
>>>>>> [hidden email]
>>>>>> http://lists.xen.org/xen-users
>>>>>>
>>>>>
>>>>> Of course I could! Please point me to any relevant information on
>>>>> how to
>>>>> build Xen with debugging enabled and what to do next. I build Xen
>>>>> using
>>>>> standard ./configure && make world && make install.
>>>>
>>>> Just `make debug=y xen` and boot with the resulting xen.gz.
>>>>
>>>> Roger.
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-users mailing list
>>>> [hidden email]
>>>> http://lists.xen.org/xen-users
>>>>
>>>
>>> I ran the test using debug build of Xen. This time I gave the name "tgt"
>>> to the DomU with iSCSI target, and the other domain was named simply
>>> "domu". Sorry for the inconsistency. After logging in to the iSCSI
>>> target from Dom0, I ran "mkfs.ext4 /dev/sdb" (still in Dom0). So far, so
>>> good. Then I launched the other DomU and as soon as I executed
>>> "fsck.ext4 /dev/xvda", some errors appeared in the output of "xl dmesg"
>>> (attached as "xl-dmesg.log"). Surprisingly, the first fsck succeeded.
>>> Unfortunately, executing fsck.ext4 for the second time showed serious
>>> file system errors. The fsck commands were the only things I ran that
>>> touched /dev/xvda. After shutting down "domu", when I tried to log out
>>> from the iSCSI target, an error came up in Dom0's dmesg
>>> ("dom0-dmesg.log"). Logs from /var/log/xen/ are also attached.
>>>
>>> I will happily run next tests - just tell me what can I do :)
>>
>> Hello,
>>
>> This is the same problem I've seen when using a similar setup. The root
>> of the problem is that blkback maps a grant ref to a memory page in
>> Dom0, then this memory page ends up in netback, and when netback tries
>> to issue a GNTTABOP_copy using the mfn of this grant mapped page the
>> operation fails because Xen detects that the mfn passed doesn't belong
>> to the guest.
>>
>> The only way I can think of solving this is that netback detects that
>> the page is not local and somehow we use it's grant ref instead of mfn
>> (this means we would need to store the grant ref somewhere in the page).
>>
>> Roger.
>
> As this is something far beyond my ability to solve, I couldn't resist
> to try something else - running FreeBSD 10 as a storage driver domain. I
> was able to provide a block device (zvol) from one FreeBSD DomU directly
> to another FreeBSD DomU just like described in the wiki (with Qemu
> traditional in the second DomU) and install the OS on it. Unfortunately
> the second DomU's bios was unable to detect this "disk" and boot from it.
>
> But with this command:
> xl block-attach Domain-0
> "format=raw,backendtype=phy,backend=fbsd,vdev=xvds,target=/dev/zvol/zroot/vol1"
>
>
> I was able to attach a block device exported from a DomU to Dom0 without
> iSCSI and then using it as a disk for a second DomU (with
> disk=['phy:/dev/xvds,xvda,w']). Now, if the second DomU had no PV
> drivers (e.g. Windows without GPLPV), everything worked fine. But
> running an OS with PV drivers (Linux or Windows+GPLPV) in the second
> DomU resulted in a very similar errors in xl dmesg like in the
> previously attached logs (see the attachment).
>
> Do I understand correctly that solving the issue you are pointing out
> would also allow to use OSes like FreeBSD as storage driver domain for
> other PV-enabled DomUs? That would be something!
>
> And most importantly - is there anything I can do to help?

Hello,

Thanks for testing this use-case also. I have not debugged it closely,
but I think what's happening here is that blkback in Dom0 grant maps a
page passed from the DomU, and then the blkfront instance on Dom0 tries
to use gnttab_grant_foreign_access_ref on that page and fails miserably
(because the page doesn't belong to Dom0).

The right way to fix this would be to make blkfront and blkback in the
respective guests connect directly instead of using Dom0 as a proxy.
Mainly the toolstack in Dom0 needs to know you are trying to attach a
disk from a driver domain and DTRT (attach the disk locally to Dom0 for
HVM access, but write the PV disk info in xenstore so that the guest
connects directly to the blkback in the driver domain instead of Dom0).
Are you interested in submitting a patch for libxl to fix this?

Roger.


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

Wei Liu-2
In reply to this post by Roger Pau Monné-3
(Just go through the whole thread...)

On Thu, Feb 06, 2014 at 09:22:04AM +0100, Roger Pau Monné wrote:
[...]

>
> Hello,
>
> This is the same problem I've seen when using a similar setup. The root
> of the problem is that blkback maps a grant ref to a memory page in
> Dom0, then this memory page ends up in netback, and when netback tries
> to issue a GNTTABOP_copy using the mfn of this grant mapped page the
> operation fails because Xen detects that the mfn passed doesn't belong
> to the guest.
>

When did this copy happen? From DomU to Dom0? From Dom0 to TGT? I guess
the latter? Data path from DomU to Dom0 should be handled by blkfront /
blkback, right?

> The only way I can think of solving this is that netback detects that
> the page is not local and somehow we use it's grant ref instead of mfn
> (this means we would need to store the grant ref somewhere in the page).
>

From my reading of blkback code, it always uses mapping, right? Could
there be a path that doesn't use mapping (not necessary now, but also
consider that in the future)?

Unless there's a simple way to embed information that marks the page is
from blkback *and* it is grant, mapped there is nothing netback can
really do about that. Checking every page would incur penalty so we need
to be careful about that.

Or a simpler solution would be -- don't use this setup. My google-fu
tells me that Linux can be booted from iSCSI so probably a valid setup
would be exporting target from TGT, DomU uses it directly, avoiding
proxying through Dom0. (I could be talking crap as I've never played
with iSCSI).


Wei.

> Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
12