reboot driver domains with auto-reconnect?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

reboot driver domains with auto-reconnect?

Jason Cooper
All,

I've been tinkering with a miniature VM setup under Xen.  Currently, I
have Linux 4.14.29 based on allnoconfig, and a <6MB initrd booting
successfully (These include xl devd).  There are three primary classes
of VMs: switch, gateway/firewall, and vlan switch.  I then use these to
deploy my network and attach regular, full-OS VMs.

My current dilemma is that I would like to be able to reboot these mini
VMs for kernel updates, etc.  When this happens, I'd like the previously
connected VMs to reconnect to the new instance.

Quite a bit of digging has given me pieces and fragments of solutions,
none of which seem solid.  xenstore-chmod seems promising, but I'm not
certain the best location to run the re-attach code.  Should the
rebooted VM search for dangling connections to the old domid and have
permission to reconnect them all?  Or, should each VM detect the loss of
link, search for the new VM, and reconnect?  Which implies having
permissions to do so.

Has anybody else implemented this concept?  If so, how did you solve it?

thx,

Jason.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: reboot driver domains with auto-reconnect?

Jason Cooper
reply-to-self update.  ;-)

On Mon, Apr 16, 2018 at 03:58:59PM +0000, Jason Cooper wrote:
> My current dilemma is that I would like to be able to reboot these mini
> VMs for kernel updates, etc.  When this happens, I'd like the previously
> connected VMs to reconnect to the new instance.

Here's the script, to be executed on Dom0, that attempts to do the
reboot of a driver domain, and then re-attach the previously attached
vifs.  I'm only concerned about vifs for now.  vbds will come later.

The reboot portion sucks, I despise random sleeps.  I'd like to trigger
on a xenbus variable to indicate the VM is ready to receive attach
events.

Also, the client VM to the driver-domain needs hotplug enabled, which
makes sense.

Occassionally, I'll get a stale vif in the client VM.  Usually this is
when I was debugging something else, and just needed to reboot the
driver-domain.  I'm hoping those won't occur once I nail down a proper
reboot sequence.  I'll see if I can catch the 'timeout waiting for vif
to detach' error message, which maybe related.

------------------>8-----------------------------------------------
#!/bin/bash

if [ $# -ne 1 ]; then
        echo >&2 "Usage: ${0##*/} domain"
        exit 1
fi

DOM="$1"

# get the domain id
DOMID="`xl domid $DOM`"
[[ "$DOMID" =~ (^[0-9]+$) ]] || exit 1

tmp="`mktemp`"

# loop through frontends
while read frontend <&4; do
        while read vif <&5; do
                if [ "x$vif" = "x" ]; then
                        # stale frontend
                        echo >&2 "WARN: stale frontend ($frontend), removing"
                        xenstore-rm /local/domain/$DOMID/backend/vif/$frontend
                        continue
                fi

                # store info for afterwards
                front="`xl domname $frontend`"
                bridge="`xenstore-read /local/domain/$DOMID/backend/vif/$frontend/$vif/bridge`"
                echo "$front bridge=$bridge backend=$DOM" >>"$tmp"

                # remove the vif
                xl -f network-detach $front $vif
        done 5< <(xenstore-list /local/domain/$DOMID/backend/vif/$frontend)
done 4< <(xenstore-list /local/domain/$DOMID/backend/vif)

# reboot the domain
xl destroy $DOM || exit 2
sleep 2
xl create -c $DOM || exit 3

if ! [ -e "$tmp" ]; then
        exit 0
fi

# reattach everything
while read ln <&4; do
        echo >&2 "re-attach [$ln]"
        xl network-attach $ln || exit 4
done 4< <(cat $tmp)

rm -f $tmp
------------------>8-----------------------------------------------

thx,

Jason.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: reboot driver domains with auto-reconnect?

Florian Heigl
Hi Jason, just saw this by chance
(gmail didn't put the mail into the right folder :)

> Am 19.04.2018 um 16:19 schrieb Jason Cooper <[hidden email]>:
>
> reply-to-self update.  ;-)
>
> On Mon, Apr 16, 2018 at 03:58:59PM +0000, Jason Cooper wrote:
>> My current dilemma is that I would like to be able to reboot these mini
>> VMs for kernel updates, etc.  When this happens, I'd like the previously
>> connected VMs to reconnect to the new instance.
>
> Here's the script, to be executed on Dom0, that attempts to do the
> reboot of a driver domain, and then re-attach the previously attached
> vifs.  I'm only concerned about vifs for now.  vbds will come later.

Awesome job!

This issue is exactly what made me give up on driver domains.
Hopefully this will go to upstream and default behaviour??!

Flo

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: reboot driver domains with auto-reconnect?

Jason Cooper
In reply to this post by Jason Cooper
On Thu, Apr 19, 2018 at 02:19:25PM +0000, Jason Cooper wrote:
> On Mon, Apr 16, 2018 at 03:58:59PM +0000, Jason Cooper wrote:
> > My current dilemma is that I would like to be able to reboot these mini
> > VMs for kernel updates, etc.  When this happens, I'd like the previously
> > connected VMs to reconnect to the new instance.
>
> Here's the script, to be executed on Dom0, that attempts to do the
> reboot of a driver domain, and then re-attach the previously attached
> vifs.  I'm only concerned about vifs for now.  vbds will come later.

v2 attached.

> The reboot portion sucks, I despise random sleeps.  I'd like to trigger
> on a xenbus variable to indicate the VM is ready to receive attach
> events.

still haven't tackled this yet.

> Also, the client VM to the driver-domain needs hotplug enabled, which
> makes sense.
>
> Occassionally, I'll get a stale vif in the client VM.  Usually this is
> when I was debugging something else, and just needed to reboot the
> driver-domain.  I'm hoping those won't occur once I nail down a proper
> reboot sequence.  I'll see if I can catch the 'timeout waiting for vif
> to detach' error message, which maybe related.

A couple of notes:

 - Upon calling 'xl reboot ...' or 'xl shutdown ...' the guest VM kernel
   attempts to execute /sbin/reboot or /sbin/poweroff.  If you did the
   same dumb thing I did, and put all the busybox symlinks in /bin/, you
   need to add symlinks for /sbin/{reboot,poweroff}.

 - during develop/test/debug-repeat cycles, I accidentally created extra
   vifs.  So instead of just v234.0 <-> eth0, I'd also have vif234.1 <->
   eth1.  It's *not* sufficient to do

        xenstore-rm /local/domain/234/device/vif/1

   You must also do

        xenstore-rm /local/domain/418/backend/vif/234/1

   Where 418 is the current driver domain and 234 is the guest.

 - For some reason 'xl shutdown -w DOM', while it does wait, doesn't
   wait long enough.  You'll see the sleep following the command because
   otherwise 'xl create ...' would error out.  This probably needs some
   more digging.

 - configuring hotplug in the guest is critical if you'd like the
   re-connected interface to be auto-magically setup as it was before

        echo '/bin/mdev' >/proc/sys/kernel/hotplug

   And edit /etc/mdev.conf for the network interfaces to trigger
   scripts.

-------------------------->8------------------------------------------
#!/bin/bash

if [ $# -ne 1 ]; then
        echo >&2 "Usage: ${0##*/} domain"
        exit 1
fi

DOM="$1"

# get the domain id
DOMID="`xl domid $DOM`"
[[ "$DOMID" =~ (^[0-9]+$) ]] || exit 1

tmp="`mktemp`"

# loop through frontends
while read frontend <&4; do
        while read vif <&5; do
                if [ "x$vif" = "x" ]; then
                        # stale frontend
                        echo >&2 "WARN: stale frontend ($frontend), removing"
                        xenstore-rm /local/domain/$DOMID/backend/vif/$frontend
                        continue
                fi

                # store info for afterwards
                front="`xl domname $frontend`"
                bridge="`xenstore-read /local/domain/$DOMID/backend/vif/$frontend/$vif/bridge`"
                if [ "x$front" != "x" ] && [[ "$bridge" =~ (br[0-9][0-9]*) ]]; then
                        echo "$front bridge=$bridge backend=$DOM" >>"$tmp"

                        # remove the vif
                        echo >&2 "Removing $vif from $front"
                        xl -f network-detach $front $vif
                fi
        done 5< <(xenstore-list /local/domain/$DOMID/backend/vif/$frontend)
done 4< <(xenstore-list /local/domain/$DOMID/backend/vif)

# reboot the domain
xl shutdown -w $DOM || exit 2
sleep 1
xl create -c $DOM || exit 3

if [ "`cat $tmp | wc -c`" -eq 0 ]; then
        rm -f $tmp
        exit 0
fi

# reattach everything
echo
while read ln <&4; do
        echo >&2 "re-attach [$ln]"
        xl network-attach $ln || exit 4
done 4< <(cat $tmp)

rm -f $tmp

exit 0
-------------------------->8------------------------------------------

thx,

Jason.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users