Xen PV Guest won't boot latest kernel- OSError: [Errno 28] No space left on device

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Xen PV Guest won't boot latest kernel- OSError: [Errno 28] No space left on device

Michael Barkdoll
Hello, I have an Ubuntu 14.04 running Xen and a VM running Ubuntu 16.04 LTS that I'm having issues with it booting after upgrading the kernel for meltdown and spectre vulnerability.

````
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
````

Ubuntu 14.04 xen host reboot failure log:
````
$ tailf /var/log/xen/servername.log
Domain 24 needs to be cleaned up: destroying the domain
Done. Rebooting now
libxl: error: libxl_bootloader.c:628:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.27.log
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [-1] exited with error status 1
libxl: error: libxl_create.c:1024:domcreate_rebuild_done: cannot (re-)build domain: -3
````


````
cat /var/log/xen/bootloader.27.log
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /grub/grub.cfg    pyGRUB  version 0.6
 ┌────────────────────────────────────────────────────────────────────────┐
 │ Ubuntu                                                                 │
 │ Ubuntu, with Linux 4.4.0-109-generic                                   │
 │ Ubuntu, with Linux 4.4.0-109-generic (recovery mode)                   │
 │ Ubuntu, with Linux 4.4.0-108-generic                                   │
 │ Ubuntu, with Linux 4.4.0-108-generic (recovery mode)                   │
 │ Ubuntu, with Linux 4.4.0-104-generic                                   │
 │ Ubuntu, with Linux 4.4.0-104-generic (recovery mode)                   │
 │ Ubuntu, with Linux 4.4.0-15-generic                                    │
 └────────────────────────────────────────────────────────────────────────┘
Use the ^ and ┴ keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, 'a' to modify the kernel arguments
before booting, or 'c' for a command line.

Traceback (most recent call last):
  File "/usr/lib/xen-4.4/bin/pygrub", line 905, in <module>
    os.write(fd, ostring)
OSError: [Errno 28] No space left on device
````

I was able to get the vm to boot only by rapidly attempt to access the menu options and boot the old kernel (4.4.0-15-generic). Disk space and inodes seems fine on the host Ubuntu 14.04LTS and Ubuntu 16.04LTS Guest.

Ubuntu 14.04 Host:
````
root@host:/var/log/xen# df -hFilesystem      Size  Used Avail Use% Mounted on
udev            197M   12K  197M   1% /dev
tmpfs            42M  1.1M   41M   3% /run
/dev/dm-0       104G   49G   50G  50% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            208M     0  208M   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sda1       236M   40M  184M  18% /boot
root@host:/var/log/xen# df -hi
Filesystem     Inodes IUsed IFree IUse% Mounted on
udev              50K   721   49K    2% /dev
tmpfs             52K   849   52K    2% /run
/dev/dm-0        6.6M   86K  6.5M    2% /
none              52K     2   52K    1% /sys/fs/cgroup
none              52K     5   52K    1% /run/lock
none              52K     1   52K    1% /run/shm
none              52K     2   52K    1% /run/user
/dev/sda1         61K   298   61K    1% /boot
root@host:/var/log/xen#
````

Ubuntu 16.04 PV Guest:
````
root@guest:~# df -hFilesystem                            Size  Used Avail Use% Mounted on
udev                                  950M     0  950M   0% /dev
tmpfs                                 200M  3.1M  197M   2% /run
/dev/mapper/guest--vg-root              47G  4.3G   40G  10% /
tmpfs                                 997M     0  997M   0% /dev/shm
tmpfs                                 5.0M     0  5.0M   0% /run/lock
tmpfs                                 997M     0  997M   0% /sys/fs/cgroup
/dev/xvda1                            472M  192M  256M  43% /boot
tmpfs                                 200M     0  200M   0% /run/user/1000
root@www2:~# df -hi
Filesystem                           Inodes IUsed IFree IUse% Mounted on
udev                                   238K   406  237K    1% /dev
tmpfs                                  250K   491  249K    1% /run
/dev/mapper/guest--vg-root              3.0M  177K  2.8M    6% /
tmpfs                                  250K     1  250K    1% /dev/shm
tmpfs                                  250K     7  250K    1% /run/lock
tmpfs                                  250K    16  250K    1% /sys/fs/cgroup
/dev/xvda1                             122K   315  122K    1% /boot
tmpfs                                  250K     4  250K    1% /run/user/1000
root@guest:~#
````

Here is the guest Ubuntu 16.04 OS kernels:
````
root@guest:~# dpkg -l | grep linux-image-
ii  linux-image-4.4.0-104-generic       4.4.0-104.127                              amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-4.4.0-108-generic       4.4.0-108.131                              amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-4.4.0-109-generic       4.4.0-109.132                              amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-4.4.0-15-generic        4.4.0-15.31                                amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-extra-4.4.0-104-generic 4.4.0-104.127                              amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-extra-4.4.0-108-generic 4.4.0-108.131                              amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-extra-4.4.0-109-generic 4.4.0-109.132                              amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-extra-4.4.0-15-generic  4.4.0-15.31                                amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-generic                 4.4.0.109.114                              amd64        Generic Linux kernel image
root@guest:~#
````

I'm currently able to boot to 4.4.0-15.31

Here is my xen config file that I use to start the vm:
````
# cat /etc/xen/guest.cfg
# /var/lib/xen/images/ubuntu-netboot/xenial16LTS


name = "guest"


#kernel = "/var/lib/xen/images/ubuntu-netboot/xenial16LTS/vmlinuz"
#ramdisk = "/var/lib/xen/images/ubuntu-netboot/xenial16LTS/initrd.gz"
bootloader = "/usr/lib/xen-4.4/bin/pygrub"


memory = 2048
vcpus = 1




disk = [ '/dev/host-disk2-vg/guest-disk,raw,xvda,rw' ]




#
#  Networking
#
vif         = [ 'ip=xxx.xxx.xxx.xxx ,mac=AA:BB:CC:DD:EE:FF' ]


#
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
````

I also tried launching the vm with strace output of
````
strace xl create -c file.cfg 
````

I'm a bit lost as to the cause of this... I was hoping to just upgrade the kernel for meltdown and spetre, but this issue I've never faced before. I think it might be related to pygrub and the new kernel? I don't know for sure though anything I can check to figure this out better? Thanks.


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Xen PV Guest won't boot latest kernel- OSError: [Errno 28] No space left on device

andy smith-10
Hi,

On Thu, Jan 11, 2018 at 11:53:23AM -0600, Michael Barkdoll wrote:
> Traceback (most recent call last):
>   File "/usr/lib/xen-4.4/bin/pygrub", line 905, in <module>
>     os.write(fd, ostring)
> OSError: [Errno 28] No space left on device

I don't know what the problem is, but this suggests it lies within
pygrub. Try running pygrub separately to see what happens, i.e.:

# /usr/lib/xen-4.4/bin/pygrub /dev/host-disk2-vg/guest-disk

pygrub copies the kernel and initramfs out of the guest disk to
somewhere else so it really might be running out of space somewhere
in the host. I forget where it writes it to though, it's been a
while.

Cheers,
Andy

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Xen PV Guest won't boot latest kernel- OSError: [Errno 28] No space left on device

Michael Barkdoll
In reply to this post by Michael Barkdoll
Thanks for the reply Andy.  I tried as you suggested here is the output of the command.  

> I don't know what the problem is, but this suggests it lies within
> pygrub. Try running pygrub separately to see what happens, i.e.:

> # /usr/lib/xen-4.4/bin/pygrub /dev/host-disk2-vg/guest-disk

````
$ /usr/lib/xen-4.4/bin/pygrub /dev/vps03-disk2-vg/www2-disk
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /grub/grub.cfg
WARNING:root:grub2's saved_entry/next_entry not supported
WARNING:root:Unknown directive else
WARNING:root:...

Traceback (most recent call last):
  File "/usr/lib/xen-4.4/bin/pygrub", line 873, in <module>
    output_directory, not_really)
  File "/usr/lib/xen-4.4/bin/pygrub", line 718, in copy_from_image
    dir=output_directory)
  File "/usr/lib/python2.7/tempfile.py", line 308, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib/python2.7/tempfile.py", line 239, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
OSError: [Errno 2] No such file or directory: '/var/run/xend/boot/boot_kernel.jedvP5'
````

Directory /var/run/xend/boot/ don't exist in the host Xen Ubuntu14.04 of the Guest Ubuntu 16.04.  Guest VM still fails to reboot after a kernel upgrade, unless I manually select the old kernel.


On Thu, Jan 11, 2018 at 11:53 AM, Michael Barkdoll <[hidden email]> wrote:
Hello, I have an Ubuntu 14.04 running Xen and a VM running Ubuntu 16.04 LTS that I'm having issues with it booting after upgrading the kernel for meltdown and spectre vulnerability.

````
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
````

Ubuntu 14.04 xen host reboot failure log:
````
$ tailf /var/log/xen/servername.log
Domain 24 needs to be cleaned up: destroying the domain
Done. Rebooting now
libxl: error: libxl_bootloader.c:628:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.27.log
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [-1] exited with error status 1
libxl: error: libxl_create.c:1024:domcreate_rebuild_done: cannot (re-)build domain: -3
````


````
cat /var/log/xen/bootloader.27.log
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /grub/grub.cfg    pyGRUB  version 0.6
 ┌────────────────────────────────────────────────────────────────────────┐
 │ Ubuntu                                                                 │
 │ Ubuntu, with Linux 4.4.0-109-generic                                   │
 │ Ubuntu, with Linux 4.4.0-109-generic (recovery mode)                   │
 │ Ubuntu, with Linux 4.4.0-108-generic                                   │
 │ Ubuntu, with Linux 4.4.0-108-generic (recovery mode)                   │
 │ Ubuntu, with Linux 4.4.0-104-generic                                   │
 │ Ubuntu, with Linux 4.4.0-104-generic (recovery mode)                   │
 │ Ubuntu, with Linux 4.4.0-15-generic                                    │
 └────────────────────────────────────────────────────────────────────────┘
Use the ^ and ┴ keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, 'a' to modify the kernel arguments
before booting, or 'c' for a command line.

Traceback (most recent call last):
  File "/usr/lib/xen-4.4/bin/pygrub", line 905, in <module>
    os.write(fd, ostring)
OSError: [Errno 28] No space left on device
````

I was able to get the vm to boot only by rapidly attempt to access the menu options and boot the old kernel (4.4.0-15-generic). Disk space and inodes seems fine on the host Ubuntu 14.04LTS and Ubuntu 16.04LTS Guest.

Ubuntu 14.04 Host:
````
root@host:/var/log/xen# df -hFilesystem      Size  Used Avail Use% Mounted on
udev            197M   12K  197M   1% /dev
tmpfs            42M  1.1M   41M   3% /run
/dev/dm-0       104G   49G   50G  50% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            208M     0  208M   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sda1       236M   40M  184M  18% /boot
root@host:/var/log/xen# df -hi
Filesystem     Inodes IUsed IFree IUse% Mounted on
udev              50K   721   49K    2% /dev
tmpfs             52K   849   52K    2% /run
/dev/dm-0        6.6M   86K  6.5M    2% /
none              52K     2   52K    1% /sys/fs/cgroup
none              52K     5   52K    1% /run/lock
none              52K     1   52K    1% /run/shm
none              52K     2   52K    1% /run/user
/dev/sda1         61K   298   61K    1% /boot
root@host:/var/log/xen#
````

Ubuntu 16.04 PV Guest:
````
root@guest:~# df -hFilesystem                            Size  Used Avail Use% Mounted on
udev                                  950M     0  950M   0% /dev
tmpfs                                 200M  3.1M  197M   2% /run
/dev/mapper/guest--vg-root              47G  4.3G   40G  10% /
tmpfs                                 997M     0  997M   0% /dev/shm
tmpfs                                 5.0M     0  5.0M   0% /run/lock
tmpfs                                 997M     0  997M   0% /sys/fs/cgroup
/dev/xvda1                            472M  192M  256M  43% /boot
tmpfs                                 200M     0  200M   0% /run/user/1000
root@www2:~# df -hi
Filesystem                           Inodes IUsed IFree IUse% Mounted on
udev                                   238K   406  237K    1% /dev
tmpfs                                  250K   491  249K    1% /run
/dev/mapper/guest--vg-root              3.0M  177K  2.8M    6% /
tmpfs                                  250K     1  250K    1% /dev/shm
tmpfs                                  250K     7  250K    1% /run/lock
tmpfs                                  250K    16  250K    1% /sys/fs/cgroup
/dev/xvda1                             122K   315  122K    1% /boot
tmpfs                                  250K     4  250K    1% /run/user/1000
root@guest:~#
````

Here is the guest Ubuntu 16.04 OS kernels:
````
root@guest:~# dpkg -l | grep linux-image-
ii  linux-image-4.4.0-104-generic       4.4.0-104.127                              amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-4.4.0-108-generic       4.4.0-108.131                              amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-4.4.0-109-generic       4.4.0-109.132                              amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-4.4.0-15-generic        4.4.0-15.31                                amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-extra-4.4.0-104-generic 4.4.0-104.127                              amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-extra-4.4.0-108-generic 4.4.0-108.131                              amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-extra-4.4.0-109-generic 4.4.0-109.132                              amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-extra-4.4.0-15-generic  4.4.0-15.31                                amd64        Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-generic                 4.4.0.109.114                              amd64        Generic Linux kernel image
root@guest:~#
````

I'm currently able to boot to 4.4.0-15.31

Here is my xen config file that I use to start the vm:
````
# cat /etc/xen/guest.cfg
# /var/lib/xen/images/ubuntu-netboot/xenial16LTS


name = "guest"


#kernel = "/var/lib/xen/images/ubuntu-netboot/xenial16LTS/vmlinuz"
#ramdisk = "/var/lib/xen/images/ubuntu-netboot/xenial16LTS/initrd.gz"
bootloader = "/usr/lib/xen-4.4/bin/pygrub"


memory = 2048
vcpus = 1




disk = [ '/dev/host-disk2-vg/guest-disk,raw,xvda,rw' ]




#
#  Networking
#
vif         = [ 'ip=xxx.xxx.xxx.xxx ,mac=AA:BB:CC:DD:EE:FF' ]


#
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
````

I also tried launching the vm with strace output of
````
strace xl create -c file.cfg 
````

I'm a bit lost as to the cause of this... I was hoping to just upgrade the kernel for meltdown and spetre, but this issue I've never faced before. I think it might be related to pygrub and the new kernel? I don't know for sure though anything I can check to figure this out better? Thanks.



_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users