Xen with Ceph over libvirt not working ...

classic Classic list List threaded Threaded
4 messages Options
thg
Reply | Threaded
Open this post in threaded view
|

Xen with Ceph over libvirt not working ...

thg
Hi everybody,

long time ago, when I was on this list, Xen 3.0 was not yet released :-)

But, I'm still with Xen :-D

In the past I was using Xen with DRBD and GlusterFS as storage, but now,
I have to go with Ceph too.

Software I'm using:
- CentOS 7.5.1804
- Kernel: 4.9.86-30.el7             @centos-virt-xen-48
- Xen: 4.8.3-5.el7                  @centos-virt-xen-48
- libvirt-xen: 4.1.0-2.xen48.el7    @centos-virt-xen-48
- Ceph: 2:12.2.5-0.el7              @Ceph

Xen itself works, as does Ceph too, but using the Ceph-RBD from Xen via
libvirt fails - accessing them directly after doing "rbd map" and
defining the "/dev/rbdNN" as physical device in the vm-config works, but
with many vm's it is too "confusing".

As far as I understood, there is still no Ceph-support in Xen, since it
was announced in 2013, so the way to go is with libvirt?


I was following this guide, to setup Ceph with libvirt:
<http://docs.ceph.com/docs/master/rbd/libvirt/>:

# ceph auth get-or-create client.libvirt mon 'profile rbd' osd 'profile
rbd pool=devel-pool'
-> [client.libvirt]
       key = AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
# ceph auth ls
-> ...
   client.libvirt
       key: AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
       caps: [mon] profile rbd
       caps: [osd] profile rbd pool=devel-pool
       ...
# vi secret.xml
->
<secret ephemeral='no' private='no'>
        <usage type='ceph'>
                <name>client.libvirt secret</name>
        </usage>
</secret>

# virsh secret-define --file secret.xml
-> Secret 07f3a0fe-0000-1111-2222-333333333333 created
# ceph auth get-key client.libvirt > client.libvirt.key
# cat client.libvirt.key
-> AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
# virsh secret-set-value --secret 07f3a0fe-0000-1111-2222-333333333333
--base64 $(cat client.libvirt.key)
-> Secret value set

# vi xml/testvm3.xml
->
<domain type='xen'>
  <name>testvm3</name>
  ...
  <devices>
    <disk type='network' device='disk'>
      <source protocol='rbd' name='devel-pool/testvm3.rbd'>
        <host name="10.20.30.1" port="6789"/>
        <host name="10.20.30.2" port="6789"/>
        <host name="10.20.30.3" port="6789"/>
      </source>
      <auth username='libvirt'>
        <secret type='ceph' uuid='07f3a0fe-0000-1111-2222-333333333333'/>
      </auth>
      <target dev='xvda' bus='xen'/>
    </disk>
    ...

# virsh define xml/testvm3.xml
-> Domain testvm3 defined from xml/testvm3.xml
# virsh start --console testvm3
error: Failed to start domain testvm3
error: internal error: libxenlight failed to create new domain 'testvm3'


So "somthing" goes wrong:

# cat /var/log/libvirt/libxl/libxl-driver.log
-> ...
2018-05-20 15:28:15.270+0000: libxl:
libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult
logfile /var/log/xen/bootloader.7.log
2018-05-20 15:28:15.270+0000: libxl:
libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [26640]
exited with error status 1
2018-05-20 15:28:15.271+0000: libxl:
libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3

# cat /var/log/xen/bootloader.7.log
->
Traceback (most recent call last):
  File "/usr/lib64/xen/bin/pygrub", line 896, in <module>
    part_offs = get_partition_offsets(file)
  File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets
    image_type = identify_disk_image(file)
  File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image
    fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory:
'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789'


So, as far as I "read" the logs, Xen does not find the RBD-device, but I
have no clue, how I can solve this :-( Any idea how it might work?


Thanks a lot,
--
kind regards,

thg

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Xen with Ceph over libvirt not working ...

Florian Heigl
Hi,

we hacked something like 5 years ago:
Maybe it still works.
It's without a libvirt layer, libvirt integration is another story.

(Noone cared back then, just as noone cares now... so I also don't care. 
But maybe it helps you nonetheless. It worked...)

Flo


Am 20.05.2018 um 22:42 schrieb thg <[hidden email]>:

Hi everybody,

long time ago, when I was on this list, Xen 3.0 was not yet released :-)

But, I'm still with Xen :-D

In the past I was using Xen with DRBD and GlusterFS as storage, but now,
I have to go with Ceph too.

Software I'm using:
- CentOS 7.5.1804
- Kernel: 4.9.86-30.el7             @centos-virt-xen-48
- Xen: 4.8.3-5.el7                  @centos-virt-xen-48
- libvirt-xen: 4.1.0-2.xen48.el7    @centos-virt-xen-48
- Ceph: 2:12.2.5-0.el7              @Ceph

Xen itself works, as does Ceph too, but using the Ceph-RBD from Xen via
libvirt fails - accessing them directly after doing "rbd map" and
defining the "/dev/rbdNN" as physical device in the vm-config works, but
with many vm's it is too "confusing".

As far as I understood, there is still no Ceph-support in Xen, since it
was announced in 2013, so the way to go is with libvirt?


I was following this guide, to setup Ceph with libvirt:
<http://docs.ceph.com/docs/master/rbd/libvirt/>:

# ceph auth get-or-create client.libvirt mon 'profile rbd' osd 'profile
rbd pool=devel-pool'
-> [client.libvirt]
      key = AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
# ceph auth ls
-> ...
  client.libvirt
      key: AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
      caps: [mon] profile rbd
      caps: [osd] profile rbd pool=devel-pool
      ...
# vi secret.xml
->
<secret ephemeral='no' private='no'>
       <usage type='ceph'>
               <name>client.libvirt secret</name>
       </usage>
</secret>

# virsh secret-define --file secret.xml
-> Secret 07f3a0fe-0000-1111-2222-333333333333 created
# ceph auth get-key client.libvirt > client.libvirt.key
# cat client.libvirt.key
-> AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==
# virsh secret-set-value --secret 07f3a0fe-0000-1111-2222-333333333333
--base64 $(cat client.libvirt.key)
-> Secret value set

# vi xml/testvm3.xml
->
<domain type='xen'>
 <name>testvm3</name>
 ...
 <devices>
   <disk type='network' device='disk'>
     <source protocol='rbd' name='devel-pool/testvm3.rbd'>
       <host name="10.20.30.1" port="6789"/>
       <host name="10.20.30.2" port="6789"/>
       <host name="10.20.30.3" port="6789"/>
     </source>
     <auth username='libvirt'>
       <secret type='ceph' uuid='07f3a0fe-0000-1111-2222-333333333333'/>
     </auth>
     <target dev='xvda' bus='xen'/>
   </disk>
   ...

# virsh define xml/testvm3.xml
-> Domain testvm3 defined from xml/testvm3.xml
# virsh start --console testvm3
error: Failed to start domain testvm3
error: internal error: libxenlight failed to create new domain 'testvm3'


So "somthing" goes wrong:

# cat /var/log/libvirt/libxl/libxl-driver.log
-> ...
2018-05-20 15:28:15.270+0000: libxl:
libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult
logfile /var/log/xen/bootloader.7.log
2018-05-20 15:28:15.270+0000: libxl:
libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [26640]
exited with error status 1
2018-05-20 15:28:15.271+0000: libxl:
libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3

# cat /var/log/xen/bootloader.7.log
->
Traceback (most recent call last):
 File "/usr/lib64/xen/bin/pygrub", line 896, in <module>
   part_offs = get_partition_offsets(file)
 File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets
   image_type = identify_disk_image(file)
 File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image
   fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory:
'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789'


So, as far as I "read" the logs, Xen does not find the RBD-device, but I
have no clue, how I can solve this :-( Any idea how it might work?


Thanks a lot,
--
kind regards,

thg

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
thg
Reply | Threaded
Open this post in threaded view
|

Re: Xen with Ceph over libvirt not working ...

thg
Hi Flo,

> we hacked something like 5 years ago:
> https://github.com/FlorianHeigl/xen-ceph-rbd
> Maybe it still works.
> It's without a libvirt layer, libvirt integration is another story.

thanks a lot for this, for sure I will have a closer look into it.

Anyway, the "official" supported way via libvirt would be the prefered
one for a production system.


So any other hints or advises to that?
--

kind regards,

thg

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
thg
Reply | Threaded
Open this post in threaded view
|

Re: Xen with Ceph over libvirt not working ...

thg
Hi everybody out there,

I didn't got yet any answer that helped me, to solve my problem.

Is there nobody out there, who uses CentOS with Xen and Ceph as
distributed storage? Or is it "just" working for you?

Actually I think, that the configuration might be fine, thus it seems,
that CEPH-auth works and the VM-XML-config is interpreted correct by
libvirt. But the next step fails, when libvirt should pass the
blockdevice to Xen?

Could it be possible, that there is an error in the pygrub-script?

Traceback (most recent call last):
  File "/usr/lib64/xen/bin/pygrub", line 896, in <module>
    part_offs = get_partition_offsets(file)
  File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets
    image_type = identify_disk_image(file)
  File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image
    fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory:


An other idea, there are two qemu-img on the system:

# /usr/lib64/xen/bin/qemu-img -V
qemu-img version 2.7.0 ...

and

# /usr/bin/qemu-img --help
qemu-img version 1.5.3 ...
plus the symlink /usr/bin/qemu-img-xen -> qemu-img

Which one is used by Xen, when starting a PV-VM via libvirt?

It seems, that only the second version is compiled with RBD-support,
thus the 2.7.0 is not working:

# /usr/bin/qemu-img info "rbd:devel-pool/testvm3.rbd:id=libvirt...
->
image: rbd:devel-pool/testvm3.rbd:id=libvirt...
file format: raw
virtual size: 16G (17179869184 bytes)
disk size: unavailable

# /usr/lib64/xen/bin/qemu-img info "rbd:devel-pool/testvm3.rbd:id=libvirt...
->
qemu-img: Could not open 'rbd:devel-pool/testvm3.rbd:id=libvirt...
\:6789': Unknown protocol 'rbd'
--

kind regards,

thg

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users