Restarting xend - how?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Restarting xend - how?

Jeff Noxon
I've played with xen 2.0.5, 2.0-testing (May 16 daily build), and 2.0.6.
I have 11 domU's running on my test box.

If xend stops (or if I stop it), I am unable to restart it without
rebooting.  xfrd starts and that's it.  That leads to some difficult
management situations.  Is there a work-around?

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Restarting xend - how?

Jeff Noxon
xend died again during some more testing.

When I attempt to restart it, I can see this in the logs.  Perhaps it means
something to someone:

xend.log:

[2005-05-24 12:39:47 xend] INFO (SrvDaemon:610) Xend Daemon started

xend-debug.log:

network start bridge=xen-br0 netdev=eth0 antispoof=no
Traceback (most recent call last):
  File "/usr/sbin/xend", line 121, in ?
    sys.exit(main())
  File "/usr/sbin/xend", line 107, in main
    return daemon.start()
  File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 525, in start
    self.run()
  File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 615, in run
    SrvServer.create(bridge=1)
  File "/usr/lib/python/xen/xend/server/SrvServer.py", line 47, in create
    xend = SrvRoot()
  File "/usr/lib/python/xen/xend/server/SrvRoot.py", line 29, in __init__
    self.get(name)
  File "/usr/lib/python/xen/xend/server/SrvDir.py", line 69, in get
    val = val.getobj()
  File "/usr/lib/python/xen/xend/server/SrvDir.py", line 39, in getobj
    self.obj = klassobj()
  File "/usr/lib/python/xen/xend/server/SrvDomainDir.py", line 25, in __init__
    self.xd = XendDomain.instance()
  File "/usr/lib/python/xen/xend/XendDomain.py", line 798, in instance
    inst = XendDomain()
  File "/usr/lib/python/xen/xend/XendDomain.py", line 65, in __init__
    self.initial_refresh()
  File "/usr/lib/python/xen/xend/XendDomain.py", line 153, in initial_refresh
    d_dom = self._new_domain(config, doms[domid])
  File "/usr/lib/python/xen/xend/XendDomain.py", line 188, in _new_domain
    deferred = XendDomainInfo.vm_recreate(savedinfo, info)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 218, in vm_recreate
    d = vm.construct(config)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 453, in construct
    self.construct_image()
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 480, in construct_image
    image_handler(self, image)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 1065, in vm_image_linux
    vm.create_domain("linux", kernel, ramdisk, cmdline)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 757, in create_domain
    self.create_channel()
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 782, in create_channel
    remote_port=remote)
  File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 662, in createDomChannel
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 83, in domChannel
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 253, in __init__
    remote_port=remote_port)
  File "/usr/lib/python/xen/xend/server/channel.py", line 137, in createPort
    remote_port=int(remote_port))
xen.lowlevel.xu.PortError: Failed to map domain control interface

Regards,

Jeff

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Restarting xend - how?

Michael Warner
In reply to this post by Jeff Noxon
Hey jeff,

Are you stoping all your domains beforehand. I found I can't get xend to
restart while domains are running. So try doing

/etc/init.d/xendomains stop
/etc/init.d/xend restart
/etc/init.d/xendomains start

Thats how I''ve always done it. Although if xend crashed for some reason
you may be stuck unless you can still ssh into each machine and shutdown
manually.

-Mike

Jeff Noxon wrote:

>I've played with xen 2.0.5, 2.0-testing (May 16 daily build), and 2.0.6.
>I have 11 domU's running on my test box.
>
>If xend stops (or if I stop it), I am unable to restart it without
>rebooting.  xfrd starts and that's it.  That leads to some difficult
>management situations.  Is there a work-around?
>
>_______________________________________________
>Xen-users mailing list
>[hidden email]
>http://lists.xensource.com/xen-users
>  
>


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Restarting xend - how?

Jeff Noxon
Unfortunately, xend becomes unresponsive occasionally.  I can't have
the foresight to stop my domains to prevent this problem.  Also, the
domains need to be running!

I have just discovered that if I remove /var/lib/xend-db/*, xend can be
started again.  I also removed a lockfile.

However, I have only had limited success manipulating domains at that
point.  'xm shutdown' did not work, but destroy did, sort-of.  The domain
lingers around forever with 0 memory allocated to it.  When I start up
my domains again, I get something like this:

# xm list
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      251    0  r----    116.8
Domain-1           1        0    1  -----      5.6
Domain-10         10        0    1  -----     11.5
Domain-11         11        0    0  -----      2.7
Domain-2           2        0    1  -----      3.7
Domain-3           3        0    0  -----      3.4
Domain-4           4        0    1  -----     20.9
Domain-5           5        0    0  -----      3.0
Domain-6           6        0    1  -----      2.9
Domain-7           7        0    0  -----      2.2
Domain-8           8        0    1  -----      3.1
Domain-9           9        0    0  -----      4.5
working-1         12       47    1  -b---      4.6    9612
working-2         13       63    0  -b---      3.0    9613
working-3         14       47    1  -b---      2.8    9614
working-4         15      127    0  -b---     10.5    9615
working-5         16       47    1  -b---      2.4    9616
working-6         17       47    0  -b---      2.2    9617
working-7         18       47    1  -b---      1.9    9618
working-8         19       47    0  -b---      2.5    9619
working-9         20       63    1  -b---      4.1    9620
working-10        21      127    0  -b---     11.3    9621
working-11        22       47    1  -b---      2.3    9622

Also, /etc/init.d/xendomains does not work anymore.  So I still need to
reboot to make things normal again.

Regards,

Jeff

On Tue, May 24, 2005 at 01:39:44PM -0600, Michael Warner wrote:

> Hey jeff,
>
> Are you stoping all your domains beforehand. I found I can't get xend to
> restart while domains are running. So try doing
>
> /etc/init.d/xendomains stop
> /etc/init.d/xend restart
> /etc/init.d/xendomains start
>
> Thats how I''ve always done it. Although if xend crashed for some reason
> you may be stuck unless you can still ssh into each machine and shutdown
> manually.
>
> -Mike

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: Restarting xend - how?

Anthony Liguori
Jeff Noxon wrote:

>I have just discovered that if I remove /var/lib/xend-db/*, xend can be
>started again.  I also removed a lockfile.
>
>However, I have only had limited success manipulating domains at that
>point.  'xm shutdown' did not work, but destroy did, sort-of.  The domain
>lingers around forever with 0 memory allocated to it.  When I start up
>my domains again, I get something like this:
>  
>
What you're seeing is a well-known problem.  Xend can corrupt its own
saved state and can not handle recovering that state.  If it's going to
keep doing this, it's going to need some sort of read log to prevent
this from happening in the future.

Blowing away that state will allow Xend to restart but alot of that
information is needed (since it contains what devices are attached to
the domain).

If Xend does not know what devices are attached, it doesn't know what to
remove on destroy.  The hypervisor won't let a domain completely go away
until all of it's resources are removed and this leads to the "zombie"
domains that you're seeing.

Xend is being refactored heavily and a new device database is being
implemented which should eliminate a lot of these problems.  However,
this is all a work in progress and there isn't all that much that you
can do right now to prevent this.

VM-Tools is a bit more reliable but has less features (and has some
issues with the latest unstable).  A new version is going to be released
shortly.

VM-Tools targets both 2.0 and 3.0 whereas the new Xend code so far is
going to require 3.0.

Regards,

Anthony Liguori


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users