Error with DRBD9 and Xen 4.6.5 with HVM Guest

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Error with DRBD9 and Xen 4.6.5 with HVM Guest

Justin Pretorius
Hello

This is my first ever email list help request, apologies if I have not
100% got the protocol correct.

I am currently using Xen on Ubuntu 16.04, and am using a number of PVM
guests successfully using DRBD9 as primary root disk, but when testing
an HVM PfSense (BSD) guest, I am having some issues.

DRBD9 is successfully set up and when creating the guest using the
config as follows

disk=['drbd:disk1-firewall2,hda,w']

The guest will not create with the following info

xl -v create /etc/xen/firewall2.cfg

xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
xc: detail: VIRTUAL MEMORY ARRANGEMENT:
xc: detail:   Loader:   0000000000100000->000000000015b844
xc: detail:   Modules:  0000000000000000->0000000000000000
xc: detail:   TOTAL:    0000000000000000->000000001f800000
xc: detail:   ENTRY:    0000000000100630
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000000fb
xc: detail:   1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x7f979ffaa000 -> 0x7f979fffbdd1
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 74
device model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device
model did not start: -3
libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already exited
libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 74
libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy
guest with domid 74
libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 74
failed

Contents of guest log

qemu-system-i386: -drive
file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback:
Could not open 'disk1-firewall2': No such file or directory

Contents of hotplug log

disk1-firewall2: State change failed: (-12) Device is held open by someone
Command 'drbdsetup secondary disk1-firewall2' terminated with exit code 11

And the guest is not started

If i try to attach to dom 0 :

xl block-list 0
Vdev  BE  handle state evt-ch ring-ref BE-path

xl block-attach 0 'drbd:disk1-firewall2,hda,w'      (no errors or output)

xl block-list 0

Vdev  BE  handle state evt-ch ring-ref BE-path
768   0   0      4     102    8        /local/domain/0/backend/vbd/0/768

xl block-detach 0 768   (no errors or output)

xl block-list 0

Vdev  BE  handle state evt-ch ring-ref BE-path

So it mounts and unmounts fine manually, but not via the hotplug script?
any ideas?

Thanks
Justin

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Error with DRBD9 and Xen 4.6.5 with HVM Guest

Roger Pau Monné-3
On Thu, Sep 06, 2018 at 04:41:11PM +0200, Justin Pretorius wrote:

> Hello
>
> This is my first ever email list help request, apologies if I have not 100%
> got the protocol correct.
>
> I am currently using Xen on Ubuntu 16.04, and am using a number of PVM
> guests successfully using DRBD9 as primary root disk, but when testing an
> HVM PfSense (BSD) guest, I am having some issues.
>
> DRBD9 is successfully set up and when creating the guest using the config as
> follows
>
> disk=['drbd:disk1-firewall2,hda,w']
>
> The guest will not create with the following info
>
> xl -v create /etc/xen/firewall2.cfg
>
> xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
> xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
> xc: detail: VIRTUAL MEMORY ARRANGEMENT:
> xc: detail:   Loader:   0000000000100000->000000000015b844
> xc: detail:   Modules:  0000000000000000->0000000000000000
> xc: detail:   TOTAL:    0000000000000000->000000001f800000
> xc: detail:   ENTRY:    0000000000100630
> xc: detail: PHYSICAL MEMORY ALLOCATION:
> xc: detail:   4KB PAGES: 0x0000000000000200
> xc: detail:   2MB PAGES: 0x00000000000000fb
> xc: detail:   1GB PAGES: 0x0000000000000000
> xc: detail: elf_load_binary: phdr 0 at 0x7f979ffaa000 -> 0x7f979fffbdd1
> domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
> libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 74 device
> model: spawn failed (rc=-3)
> libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device model
> did not start: -3
> libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already exited
> libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 74
> libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy guest
> with domid 74
> libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 74
> failed
>
> Contents of guest log
>
> qemu-system-i386: -drive
> file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback:
> Could not open 'disk1-firewall2': No such file or directory

This is wrong, file= should point to some block device created by the
drbd hotplug script.

>
> Contents of hotplug log
>
> disk1-firewall2: State change failed: (-12) Device is held open by someone
> Command 'drbdsetup secondary disk1-firewall2' terminated with exit code 11

I'm afraid this is due to block hotplug scripts being run twice for
HVM guests, once for the PV disk and another time for the emulated
disk.

Could you please paste the full output of `xl -vvv create ...`, that
should contain all the calls to the hotplug scripts.

> And the guest is not started
>
> If i try to attach to dom 0 :
>
> xl block-list 0
> Vdev  BE  handle state evt-ch ring-ref BE-path
>
> xl block-attach 0 'drbd:disk1-firewall2,hda,w'      (no errors or output)
>
> xl block-list 0
>
> Vdev  BE  handle state evt-ch ring-ref BE-path
> 768   0   0      4     102    8        /local/domain/0/backend/vbd/0/768
>
> xl block-detach 0 768   (no errors or output)
>
> xl block-list 0
>
> Vdev  BE  handle state evt-ch ring-ref BE-path
>
> So it mounts and unmounts fine manually, but not via the hotplug script?

When you attach to Dom0 you are also using the hotplug script.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Error with DRBD9 and Xen 4.6.5 with HVM Guest

Justin Pretorius


On 09/07/18 09:08, Roger Pau Monné wrote:

> On Thu, Sep 06, 2018 at 04:41:11PM +0200, Justin Pretorius wrote:
>> Hello
>>
>> This is my first ever email list help request, apologies if I have not 100%
>> got the protocol correct.
>>
>> I am currently using Xen on Ubuntu 16.04, and am using a number of PVM
>> guests successfully using DRBD9 as primary root disk, but when testing an
>> HVM PfSense (BSD) guest, I am having some issues.
>>
>> DRBD9 is successfully set up and when creating the guest using the config as
>> follows
>>
>> disk=['drbd:disk1-firewall2,hda,w']
>>
>> The guest will not create with the following info
>>
>> xl -v create /etc/xen/firewall2.cfg
>>
>> xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
>> xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
>> xc: detail: VIRTUAL MEMORY ARRANGEMENT:
>> xc: detail:   Loader:   0000000000100000->000000000015b844
>> xc: detail:   Modules:  0000000000000000->0000000000000000
>> xc: detail:   TOTAL:    0000000000000000->000000001f800000
>> xc: detail:   ENTRY:    0000000000100630
>> xc: detail: PHYSICAL MEMORY ALLOCATION:
>> xc: detail:   4KB PAGES: 0x0000000000000200
>> xc: detail:   2MB PAGES: 0x00000000000000fb
>> xc: detail:   1GB PAGES: 0x0000000000000000
>> xc: detail: elf_load_binary: phdr 0 at 0x7f979ffaa000 -> 0x7f979fffbdd1
>> domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
>> libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 74 device
>> model: spawn failed (rc=-3)
>> libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device model
>> did not start: -3
>> libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already exited
>> libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 74
>> libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy guest
>> with domid 74
>> libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 74
>> failed
>>
>> Contents of guest log
>>
>> qemu-system-i386: -drive
>> file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback:
>> Could not open 'disk1-firewall2': No such file or directory
>
> This is wrong, file= should point to some block device created by the
> drbd hotplug script.

Does this mean there is an issue with the handoff from Xen to qemu while
building whichever command is required to start the HVM guest. I have
used the hotplug script in the documented method which works for PV
guests. Are you saying that using the following format

disk=['drbd:disk1-firewall2,hda,w']

Is not the correct way for HVM guests? I have confirmed, that using the
actual dev mount works correctly
(disk=['phy:/dev/vg/disk1-firewall2_00,hda,w']), but this will not live
migrate, or even start on any node besides the last node that was primary.

>
>>
>> Contents of hotplug log
>>
>> disk1-firewall2: State change failed: (-12) Device is held open by someone
>> Command 'drbdsetup secondary disk1-firewall2' terminated with exit code 11
>
> I'm afraid this is due to block hotplug scripts being run twice for
> HVM guests, once for the PV disk and another time for the emulated
> disk.
>

This could be a red herring, the row show here may be from previous
attempts to start the guest. Subsequent creates have not seemed to
create this line. I have deleted the hotplug.log and rerun, and the file
is still empty, I have restarted the xen service, in case a new file
handle was required, but it still is empty after a few failed create
attempts.


> Could you please paste the full output of `xl -vvv create ...`, that
> should contain all the calls to the hotplug scripts.
>

Parsing config from firewall2.cfg
libxl: debug: libxl_create.c:1563:do_domain_create: ao 0x1462fb0:
create: how=(nil) callback=(nil) poller=0x1462960
libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk
vdev=hda spec.backend=unknown
libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses
script=... assuming phy backend
libxl: debug: libxl_device.c:366:libxl__device_disk_set_backend: Disk
vdev=hda, using backend phy
libxl: debug: libxl_create.c:947:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:324:libxl__bootloader_run: not a PV
domain, skipping bootloader
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1460808: deregister unregistered
xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
xc: detail: VIRTUAL MEMORY ARRANGEMENT:
xc: detail:   Loader:   0000000000100000->000000000015b844
xc: detail:   Modules:  0000000000000000->0000000000000000
xc: detail:   TOTAL:    0000000000000000->000000001f800000
xc: detail:   ENTRY:    0000000000100630
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000000fb
xc: detail:   1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x7f53733d5000 -> 0x7f5373426dd1
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk
vdev=hda spec.backend=phy
libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses
script=... assuming phy backend
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0:
register slotnum=3
libxl: debug: libxl_create.c:1586:do_domain_create: ao 0x1462fb0:
inprogress: poller=0x1462960, flags=i
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60
wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event
epath=/local/domain/0/backend/vbd/85/768/state
libxl: debug: libxl_event.c:884:devstate_callback: backend
/local/domain/0/backend/vbd/85/768/state wanted state 2 still waiting
state 1
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60
wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event
epath=/local/domain/0/backend/vbd/85/768/state
libxl: debug: libxl_event.c:880:devstate_callback: backend
/local/domain/0/backend/vbd/85/768/state wanted state 2 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0:
deregister slotnum=3
libxl: debug: libxl_device.c:991:device_backend_callback: calling
device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1463b60: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and
environment ready
libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug
script: /etc/xen/scripts/block-drbd add
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
execute: /etc/xen/scripts/block-drbd add
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1463c60: deregister unregistered
libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec
1, not running hotplug scripts
libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1463c60: deregister unregistered
libxl: debug: libxl_dm.c:1796:libxl__spawn_local_dm: Spawning
device-model /usr/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
/usr/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   85
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-85,server,nowait
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -no-shutdown
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-85,server,nowait
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
chardev=libxenstat-cmd,mode=control
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -nodefaults
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   firewall2
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -vnc
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   :0,to=99
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -display
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   none
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -serial
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   pty
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
cirrus-vga,vgamem_mb=8
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -boot
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   order=c
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -net
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   none
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -machine
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   xenfv
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   504
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback
libxl: debug: libxl_dm.c:1800:libxl__spawn_local_dm: Spawning
device-model /usr/bin/qemu-system-i386 with additional environment:
libxl: debug: libxl_dm.c:1802:libxl__spawn_local_dm:
XEN_QEMU_CONSOLE_LIMIT=1048576
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1:
register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1460b00
wpath=/local/domain/0/device-model/85/state token=3/1: event
epath=/local/domain/0/device-model/85/state
libxl: debug: libxl_exec.c:399:spawn_watch_event: domain 85 device
model: spawn watch p=(null)
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1:
deregister slotnum=3
libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 85
device model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device
model did not start: -3
libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already exited
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2:
register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1467550
wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: event
epath=/local/domain/0/backend/vbd/85/768/state
libxl: debug: libxl_event.c:880:devstate_callback: backend
/local/domain/0/backend/vbd/85/768/state wanted state 6 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2:
deregister slotnum=3
libxl: debug: libxl_device.c:991:device_backend_callback: calling
device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1467550: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and
environment ready
libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug
script: /etc/xen/scripts/block-drbd remove
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
execute: /etc/xen/scripts/block-drbd remove
libxl: debug: libxl_event.c:545:watchfd_callback: watch
epath=/local/domain/0/backend/vbd/85/768/state token=3/2: empty slot
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1467650: deregister unregistered
libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec
1, not running hotplug scripts
libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1467650: deregister unregistered
libxl: debug: libxl_linux.c:267:libxl__get_hotplug_script_info:
backend_kind 6, no need to execute scripts
libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to
execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
w=0x1467ca0: deregister unregistered
libxl: debug: libxl.c:1738:devices_destroy_cb: forked pid 26093 for
destroy of domain 85
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462fb0:
complete, rc=-3
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462fb0: destroy
libxl: debug: libxl.c:1477:libxl_domain_destroy: ao 0x1462250: create:
how=(nil) callback=(nil) poller=0x1462960
libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 85
libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy
guest with domid 85
libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 85
failed
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462250:
complete, rc=-21
libxl: debug: libxl.c:1486:libxl_domain_destroy: ao 0x1462250:
inprogress: poller=0x1462960, flags=ic
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462250: destroy
xc: debug: hypercall buffer: total allocations:648 total releases:648
xc: debug: hypercall buffer: current allocations:0 maximum allocations:3
xc: debug: hypercall buffer: cache current size:3
xc: debug: hypercall buffer: cache hits:632 misses:3 toobig:13



>> And the guest is not started
>>
>> If i try to attach to dom 0 :
>>
>> xl block-list 0
>> Vdev  BE  handle state evt-ch ring-ref BE-path
>>
>> xl block-attach 0 'drbd:disk1-firewall2,hda,w'      (no errors or output)
>>
>> xl block-list 0
>>
>> Vdev  BE  handle state evt-ch ring-ref BE-path
>> 768   0   0      4     102    8        /local/domain/0/backend/vbd/0/768
>>
>> xl block-detach 0 768   (no errors or output)
>>
>> xl block-list 0
>>
>> Vdev  BE  handle state evt-ch ring-ref BE-path
>>
>> So it mounts and unmounts fine manually, but not via the hotplug script?
>
> When you attach to Dom0 you are also using the hotplug script.
>

OK, which is possibly a useless test then, as the hotplug script then
works as expected for PV, and when run manually via block-attach, so I
think this confirms the script in fact works, just not in the HVM case.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Error with DRBD9 and Xen 4.6.5 with HVM Guest

Justin Pretorius
I also did a bit of debugging in the hotplug script (block-drbd), and in
the "add" case statement, at the point where write_dev "$dev" is called,
$dev is set as "/dev/drbd116". If this is the case, why is qemu getting
instantiated as "-drive file=disk1-firewall2" instead of "-drive
file=/dev/drbd116"? or is this just a red herring ?

On 09/07/18 11:28, Justin Pretorius wrote:

>
>
> On 09/07/18 09:08, Roger Pau Monné wrote:
>> On Thu, Sep 06, 2018 at 04:41:11PM +0200, Justin Pretorius wrote:
>>> Hello
>>>
>>> This is my first ever email list help request, apologies if I have
>>> not 100%
>>> got the protocol correct.
>>>
>>> I am currently using Xen on Ubuntu 16.04, and am using a number of PVM
>>> guests successfully using DRBD9 as primary root disk, but when
>>> testing an
>>> HVM PfSense (BSD) guest, I am having some issues.
>>>
>>> DRBD9 is successfully set up and when creating the guest using the
>>> config as
>>> follows
>>>
>>> disk=['drbd:disk1-firewall2,hda,w']
>>>
>>> The guest will not create with the following info
>>>
>>> xl -v create /etc/xen/firewall2.cfg
>>>
>>> xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
>>> xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
>>> xc: detail: VIRTUAL MEMORY ARRANGEMENT:
>>> xc: detail:   Loader:   0000000000100000->000000000015b844
>>> xc: detail:   Modules:  0000000000000000->0000000000000000
>>> xc: detail:   TOTAL:    0000000000000000->000000001f800000
>>> xc: detail:   ENTRY:    0000000000100630
>>> xc: detail: PHYSICAL MEMORY ALLOCATION:
>>> xc: detail:   4KB PAGES: 0x0000000000000200
>>> xc: detail:   2MB PAGES: 0x00000000000000fb
>>> xc: detail:   1GB PAGES: 0x0000000000000000
>>> xc: detail: elf_load_binary: phdr 0 at 0x7f979ffaa000 -> 0x7f979fffbdd1
>>> domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
>>> libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 74
>>> device
>>> model: spawn failed (rc=-3)
>>> libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device
>>> model
>>> did not start: -3
>>> libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already
>>> exited
>>> libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 74
>>> libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy
>>> guest
>>> with domid 74
>>> libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 74
>>> failed
>>>
>>> Contents of guest log
>>>
>>> qemu-system-i386: -drive
>>> file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback:
>>>
>>> Could not open 'disk1-firewall2': No such file or directory
>>
>> This is wrong, file= should point to some block device created by the
>> drbd hotplug script.
>
> Does this mean there is an issue with the handoff from Xen to qemu while
> building whichever command is required to start the HVM guest. I have
> used the hotplug script in the documented method which works for PV
> guests. Are you saying that using the following format
>
> disk=['drbd:disk1-firewall2,hda,w']
>
> Is not the correct way for HVM guests? I have confirmed, that using the
> actual dev mount works correctly
> (disk=['phy:/dev/vg/disk1-firewall2_00,hda,w']), but this will not live
> migrate, or even start on any node besides the last node that was primary.
>
>>
>>>
>>> Contents of hotplug log
>>>
>>> disk1-firewall2: State change failed: (-12) Device is held open by
>>> someone
>>> Command 'drbdsetup secondary disk1-firewall2' terminated with exit
>>> code 11
>>
>> I'm afraid this is due to block hotplug scripts being run twice for
>> HVM guests, once for the PV disk and another time for the emulated
>> disk.
>>
>
> This could be a red herring, the row show here may be from previous
> attempts to start the guest. Subsequent creates have not seemed to
> create this line. I have deleted the hotplug.log and rerun, and the file
> is still empty, I have restarted the xen service, in case a new file
> handle was required, but it still is empty after a few failed create
> attempts.
>
>
>> Could you please paste the full output of `xl -vvv create ...`, that
>> should contain all the calls to the hotplug scripts.
>>
>
> Parsing config from firewall2.cfg
> libxl: debug: libxl_create.c:1563:do_domain_create: ao 0x1462fb0:
> create: how=(nil) callback=(nil) poller=0x1462960
> libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk
> vdev=hda spec.backend=unknown
> libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses
> script=... assuming phy backend
> libxl: debug: libxl_device.c:366:libxl__device_disk_set_backend: Disk
> vdev=hda, using backend phy
> libxl: debug: libxl_create.c:947:initiate_domain_create: running bootloader
> libxl: debug: libxl_bootloader.c:324:libxl__bootloader_run: not a PV
> domain, skipping bootloader
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> w=0x1460808: deregister unregistered
> xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
> xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
> xc: detail: VIRTUAL MEMORY ARRANGEMENT:
> xc: detail:   Loader:   0000000000100000->000000000015b844
> xc: detail:   Modules:  0000000000000000->0000000000000000
> xc: detail:   TOTAL:    0000000000000000->000000001f800000
> xc: detail:   ENTRY:    0000000000100630
> xc: detail: PHYSICAL MEMORY ALLOCATION:
> xc: detail:   4KB PAGES: 0x0000000000000200
> xc: detail:   2MB PAGES: 0x00000000000000fb
> xc: detail:   1GB PAGES: 0x0000000000000000
> xc: detail: elf_load_binary: phdr 0 at 0x7f53733d5000 -> 0x7f5373426dd1
> domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
> libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk
> vdev=hda spec.backend=phy
> libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses
> script=... assuming phy backend
> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
> w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0:
> register slotnum=3
> libxl: debug: libxl_create.c:1586:do_domain_create: ao 0x1462fb0:
> inprogress: poller=0x1462960, flags=i
> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60
> wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event
> epath=/local/domain/0/backend/vbd/85/768/state
> libxl: debug: libxl_event.c:884:devstate_callback: backend
> /local/domain/0/backend/vbd/85/768/state wanted state 2 still waiting
> state 1
> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60
> wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event
> epath=/local/domain/0/backend/vbd/85/768/state
> libxl: debug: libxl_event.c:880:devstate_callback: backend
> /local/domain/0/backend/vbd/85/768/state wanted state 2 ok
> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
> w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0:
> deregister slotnum=3
> libxl: debug: libxl_device.c:991:device_backend_callback: calling
> device_backend_cleanup
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> w=0x1463b60: deregister unregistered
> libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and
> environment ready
> libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug
> script: /etc/xen/scripts/block-drbd add
> libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
> execute: /etc/xen/scripts/block-drbd add
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> w=0x1463c60: deregister unregistered
> libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec
> 1, not running hotplug scripts
> libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to
> execute
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> w=0x1463c60: deregister unregistered
> libxl: debug: libxl_dm.c:1796:libxl__spawn_local_dm: Spawning
> device-model /usr/bin/qemu-system-i386 with arguments:
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
> /usr/bin/qemu-system-i386
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -xen-domid
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   85
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -chardev
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
> socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-85,server,nowait
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -no-shutdown
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -mon
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
> chardev=libxl-cmd,mode=control
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -chardev
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
> socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-85,server,nowait
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -mon
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
> chardev=libxenstat-cmd,mode=control
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -nodefaults
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -name
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   firewall2
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -vnc
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   :0,to=99
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -display
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   none
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -serial
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   pty
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -device
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: cirrus-vga,vgamem_mb=8
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -boot
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   order=c
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -net
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   none
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -machine
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   xenfv
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -m
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   504
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -drive
> libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:
> file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback
> libxl: debug: libxl_dm.c:1800:libxl__spawn_local_dm: Spawning
> device-model /usr/bin/qemu-system-i386 with additional environment:
> libxl: debug: libxl_dm.c:1802:libxl__spawn_local_dm:
> XEN_QEMU_CONSOLE_LIMIT=1048576
> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
> w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1:
> register slotnum=3
> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1460b00
> wpath=/local/domain/0/device-model/85/state token=3/1: event
> epath=/local/domain/0/device-model/85/state
> libxl: debug: libxl_exec.c:399:spawn_watch_event: domain 85 device
> model: spawn watch p=(null)
> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
> w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1:
> deregister slotnum=3
> libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 85
> device model: spawn failed (rc=-3)
> libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device
> model did not start: -3
> libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already
> exited
> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
> w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2:
> register slotnum=3
> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1467550
> wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: event
> epath=/local/domain/0/backend/vbd/85/768/state
> libxl: debug: libxl_event.c:880:devstate_callback: backend
> /local/domain/0/backend/vbd/85/768/state wanted state 6 ok
> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
> w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2:
> deregister slotnum=3
> libxl: debug: libxl_device.c:991:device_backend_callback: calling
> device_backend_cleanup
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> w=0x1467550: deregister unregistered
> libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and
> environment ready
> libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug
> script: /etc/xen/scripts/block-drbd remove
> libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
> execute: /etc/xen/scripts/block-drbd remove
> libxl: debug: libxl_event.c:545:watchfd_callback: watch
> epath=/local/domain/0/backend/vbd/85/768/state token=3/2: empty slot
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> w=0x1467650: deregister unregistered
> libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec
> 1, not running hotplug scripts
> libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to
> execute
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> w=0x1467650: deregister unregistered
> libxl: debug: libxl_linux.c:267:libxl__get_hotplug_script_info:
> backend_kind 6, no need to execute scripts
> libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to
> execute
> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
> w=0x1467ca0: deregister unregistered
> libxl: debug: libxl.c:1738:devices_destroy_cb: forked pid 26093 for
> destroy of domain 85
> libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462fb0:
> complete, rc=-3
> libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462fb0: destroy
> libxl: debug: libxl.c:1477:libxl_domain_destroy: ao 0x1462250: create:
> how=(nil) callback=(nil) poller=0x1462960
> libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 85
> libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy
> guest with domid 85
> libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 85
> failed
> libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462250:
> complete, rc=-21
> libxl: debug: libxl.c:1486:libxl_domain_destroy: ao 0x1462250:
> inprogress: poller=0x1462960, flags=ic
> libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462250: destroy
> xc: debug: hypercall buffer: total allocations:648 total releases:648
> xc: debug: hypercall buffer: current allocations:0 maximum allocations:3
> xc: debug: hypercall buffer: cache current size:3
> xc: debug: hypercall buffer: cache hits:632 misses:3 toobig:13
>
>
>
>>> And the guest is not started
>>>
>>> If i try to attach to dom 0 :
>>>
>>> xl block-list 0
>>> Vdev  BE  handle state evt-ch ring-ref BE-path
>>>
>>> xl block-attach 0 'drbd:disk1-firewall2,hda,w'      (no errors or
>>> output)
>>>
>>> xl block-list 0
>>>
>>> Vdev  BE  handle state evt-ch ring-ref BE-path
>>> 768   0   0      4     102    8        /local/domain/0/backend/vbd/0/768
>>>
>>> xl block-detach 0 768   (no errors or output)
>>>
>>> xl block-list 0
>>>
>>> Vdev  BE  handle state evt-ch ring-ref BE-path
>>>
>>> So it mounts and unmounts fine manually, but not via the hotplug script?
>>
>> When you attach to Dom0 you are also using the hotplug script.
>>
>
> OK, which is possibly a useless test then, as the hotplug script then
> works as expected for PV, and when run manually via block-attach, so I
> think this confirms the script in fact works, just not in the HVM case.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Error with DRBD9 and Xen 4.6.5 with HVM Guest

Roger Pau Monné-3
In reply to this post by Justin Pretorius
On Fri, Sep 07, 2018 at 11:28:15AM +0200, Justin Pretorius wrote:

>
>
> On 09/07/18 09:08, Roger Pau Monné wrote:
> > On Thu, Sep 06, 2018 at 04:41:11PM +0200, Justin Pretorius wrote:
> > > Hello
> > >
> > > This is my first ever email list help request, apologies if I have not 100%
> > > got the protocol correct.
> > >
> > > I am currently using Xen on Ubuntu 16.04, and am using a number of PVM
> > > guests successfully using DRBD9 as primary root disk, but when testing an
> > > HVM PfSense (BSD) guest, I am having some issues.
>
> > > DRBD9 is successfully set up and when creating the guest using the config as
> > > follows
> > >
> > > disk=['drbd:disk1-firewall2,hda,w']

Now I remember that Xen 4.6 doesn't support block hotplug scripts with
HVM guests, you will have to use 4.7 or newer in order for this to
work.

Roger.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Error with DRBD9 and Xen 4.6.5 with HVM Guest

Justin Pretorius
Thanks for this Roger

I'll upgrade and check

Regards

Justin

On 2018/09/10 16:22, Roger Pau Monné wrote:

> On Fri, Sep 07, 2018 at 11:28:15AM +0200, Justin Pretorius wrote:
>>
>> On 09/07/18 09:08, Roger Pau Monné wrote:
>>> On Thu, Sep 06, 2018 at 04:41:11PM +0200, Justin Pretorius wrote:
>>>> Hello
>>>>
>>>> This is my first ever email list help request, apologies if I have not 100%
>>>> got the protocol correct.
>>>>
>>>> I am currently using Xen on Ubuntu 16.04, and am using a number of PVM
>>>> guests successfully using DRBD9 as primary root disk, but when testing an
>>>> HVM PfSense (BSD) guest, I am having some issues.
>>>> DRBD9 is successfully set up and when creating the guest using the config as
>>>> follows
>>>>
>>>> disk=['drbd:disk1-firewall2,hda,w']
> Now I remember that Xen 4.6 doesn't support block hotplug scripts with
> HVM guests, you will have to use 4.7 or newer in order for this to
> work.
>
> Roger.


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users