Live-migrated domain crashes

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Live-migrated domain crashes

Vesso Novov
Hi all,

I wonder if anyone has had any experience with the
live migration capabilites of Xen.
I've managed to install and start Xen 2.0.6 on two
Linux hosts, both running Mandrake 9.1 (I'm losing my
CDROM when I boot up with this 2.6.11.10 kernel
though).
I've also managed to create a domU on each using the
ttylinux as described in the user doc.
However, when I try live-migrating it from hostA to
hostB the operation appers to be successfull as the
new domU appers on hostB's 'xm list' output, but
crashes when I try 'xm console ExampleDomain'.
An if I try live-migrating it form hostB to hostA it
only briefly appers on the hostA's 'xm list' output
with status code 'p' before crashing.

This si the console output after I migrated
ExampleDomain from hostA to hostB and tried loging in.
I've also attached the log files from both hosts:


>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[hostB]# xm list
Name              Id  Mem(MB)  CPU  State  Time(s)
Console
Domain-0           0      827    0  r----   2712.3
ExampleDomain      5       64    0  -b---      0.0  
9605
[hostB]# xm console ExampleDomain
************ REMOTE CONSOLE: CTRL-] TO QUIT ********
invalid operand: 0000 [#1]
PREEMPT
Modules linked in:
CPU:    0
EIP:    0061:[<c0220b62>]    Not tainted VLI
EFLAGS: 00010292   (2.6.11.10-xenU)
EIP is at __make_request+0x52/0x510
eax: c0332dc4   ebx: 00000001   ecx: c10ac6f0   edx:
c00c5758
esi: c00c5758   edi: 00000000   ebp: c10aba60   esp:
c0023ca8
ds: 007b   es: 007b   ss: 0069
Process kjournald (pid: 573, threadinfo=c0022000
task=c0376570)
Stack: c00c5758 c0023ce0 00000003 00000000 0000201c
00000002 00000002 00000000
       c00c5758 c0355a00 c012e3d0 c10aba60 c0221497
c00c5758 c10aba60 c10bb550
       c0376570 00000000 c0376570 c012e3d0 c0023d18
c0023d18 c035c46c c035c474
Call Trace:
 [<c012e3d0>] autoremove_wake_function+0x0/0x60
 [<c0221497>] generic_make_request+0x157/0x1f0
 [<c012e3d0>] autoremove_wake_function+0x0/0x60
 [<c012e3d0>] autoremove_wake_function+0x0/0x60
 [<c0138b43>] mempool_alloc+0x73/0x1b0
 [<c012e3d0>] autoremove_wake_function+0x0/0x60
 [<c0221592>] submit_bio+0x62/0x100
 [<c015a437>] bio_alloc+0xe7/0x1e0
 [<c0156840>] end_buffer_write_sync+0x0/0x80
 [<c0159d65>] submit_bh+0xd5/0x120
 [<c0159e1d>] ll_rw_block+0x6d/0x90
 [<c01ce6c1>] journal_commit_transaction+0x1041/0x1260
 [<c016e4f4>] d_rehash+0x54/0x80
 [<c017785e>] simple_lookup+0x3e/0x50
 [<c01057b8>] ctrl_if_send_message_noblock+0xc8/0x120
 [<c01d0bff>] kjournald+0xdf/0x270
 [<c012e3d0>] autoremove_wake_function+0x0/0x60
 [<c012e3d0>] autoremove_wake_function+0x0/0x60
 [<c0109202>] ret_from_fork+0x6/0x1c
 [<c01d0b00>] commit_timeout+0x0/0x10
 [<c01d0b20>] kjournald+0x0/0x270
 [<c0107275>] kernel_thread_helper+0x5/0x10
Code: 16 8d 04 40 8b 44 81 04 c1 e8 09 89 44 24 14 8b
5a 10 8d 44 24 38 89 44 24 04 89 34 24 83 e3 01 e8 54
21 f2 ff 8b 86 ec 00 00 00 <0f> 0d 08 90 8b 44 24 38
8b 68 10 83 e5 04 74 0e f6 86 e9 00 00
 <1>invalid operand: 0000 [#2]
PREEMPT
Modules linked in:
CPU:    0
EIP:    0061:[<c0220b62>]    Not tainted VLI
EFLAGS: 00010292   (2.6.11.10-xenU)
EIP is at __make_request+0x52/0x510
eax: c0332dc4   ebx: 00000001   ecx: c10ac750   edx:
c00c5758
esi: c00c5758   edi: 00000000   ebp: c10ab1a0   esp:
c037dc14
ds: 007b   es: 007b   ss: 0069
Process pdflush (pid: 34, threadinfo=c037c000
task=c10bb550)
Stack: c00c5758 c037dc4c c0220dc6 c00c5758 0000201e
00000002 00000002 00004042
       c00c5758 c0355a00 c012e3d0 c10ab1a0 c0221497
c00c5758 c10ab1a0 00000000
       00000030 00000000 c10bb550 c012e3d0 c037dc84
c037dc84 c035c46c c035c474
Call Trace:
 [<c0220dc6>] __make_request+0x2b6/0x510
 [<c012e3d0>] autoremove_wake_function+0x0/0x60
 [<c0221497>] generic_make_request+0x157/0x1f0
 [<c012e3d0>] autoremove_wake_function+0x0/0x60
 [<c012e3d0>] autoremove_wake_function+0x0/0x60
 [<c0138b43>] mempool_alloc+0x73/0x1b0
 [<c012e3d0>] autoremove_wake_function+0x0/0x60
 [<c0221592>] submit_bio+0x62/0x100
 [<c015a437>] bio_alloc+0xe7/0x1e0
 [<c0157190>] end_buffer_async_write+0x0/0x150
 [<c0159d65>] submit_bh+0xd5/0x120
 [<c0158550>] __block_write_full_page+0x170/0x340
 [<c013e3e9>] kmem_cache_alloc+0x69/0x70
 [<c0159bd5>] block_write_full_page+0xd5/0x100
 [<c01bbcb0>] ext3_get_block+0x0/0xa0
 [<c01bc901>] ext3_ordered_writepage+0xd1/0x1b0
 [<c01bbcb0>] ext3_get_block+0x0/0xa0
 [<c01bc7f0>] bget_one+0x0/0x10
 [<c017a4b3>] mpage_writepages+0x263/0x3b0
 [<c015ce40>] blkdev_writepage+0x0/0x30
 [<c01bc830>] ext3_ordered_writepage+0x0/0x1b0
 [<c013b832>] do_writepages+0x42/0x50
 [<c0178991>] __sync_single_inode+0x71/0x210
 [<c0178b97>] __writeback_single_inode+0x67/0x160
 [<c010797a>] __switch_to+0x2a/0x380
 [<c0178e27>] sync_sb_inodes+0x197/0x2a0
 [<c0179014>] writeback_inodes+0xe4/0x100
 [<c013b5f6>] wb_kupdate+0x96/0x110
 [<c013c188>] __pdflush+0xf8/0x220
 [<c013c2b0>] pdflush+0x0/0x30
 [<c013c2d6>] pdflush+0x26/0x30
 [<c013b560>] wb_kupdate+0x0/0x110
 [<c013c2b0>] pdflush+0x0/0x30
 [<c012deaa>] kthread+0xaa/0xb0
 [<c012de00>] kthread+0x0/0xb0
 [<c0107275>] kernel_thread_helper+0x5/0x10
Code: 16 8d 04 40 8b 44 81 04 c1 e8 09 89 44 24 14 8b
5a 10 8d 44 24 38 89 44 24 04 89 34 24 83 e3 01 e8 54
21 f2 ff 8b 86 ec 00 00 00 <0f> 0d 08 90 8b 44 24 38
8b 68 10 83 e5 04 74 0e f6 86 e9 00 00

ttylinux 4.2
Linux 2.6.11.10-xenU on i686 arch

tiny.local login: root
<1>invalid operand: 0000 [#3]
PREEMPT
Modules linked in:
CPU:    0
EIP:    0061:[<c016f7af>]    Not tainted VLI

************ REMOTE CONSOLE EXITED *****************
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<


               
___________________________________________________________
How much free photo storage do you get? Store your holiday
snaps for FREE with Yahoo! Photos http://uk.photos.yahoo.com
network start bridge=xen-br0 netdev=eth0 antispoof=no
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c033bb84
 Init. ramdisk: c033c000->c033c000
 Phys-Mach map: c033c000->c034c000
 Page tables:   c034c000->c034e000
 Start info:    c034e000->c034f000
 Boot stack:    c034f000->c0350000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
/usr/lib/python/xen/xend/server/controller.py:183: DeprecationWarning: Deferred.setTimeout is deprecated.  Look for timeout support specific to the API you are using instead.
  deferred.setTimeout(self.timeout)
vif-bridge up vif=vif1.0 domain=ExampleDomain mac=aa:00:00:6f:68:cd bridge=xen-br0
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 130971, 'rx_shmem_frame': 130970, 'handle': 0}
/usr/lib/python/xen/xend/XendMigrate.py:261: DeprecationWarning: Deferred.setTimeout is deprecated.  Look for timeout support specific to the API you are using instead.
  d.setTimeout(self.timeout)
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif1.0 domain=ExampleDomain mac=aa:00:00:6f:68:cd bridge=xen-br0
_op_migrate_cb> <xen.xend.XendMigrate.XendMigrateInfo instance at 0x850ed44> <POST /xend/domain/ExampleDomain HTTP/1.1>
_op_migrate_cb> url= http://giotto.doc.ic.ac.uk:8002/xend/domain/5

[2005-05-27 11:42:54 xend] INFO (SrvDaemon:610) Xend Daemon started
[2005-05-27 11:42:56 xend] INFO (XendRoot:112) EVENT> xend.start 0
[2005-05-27 11:47:38 xend] DEBUG (XendDomainInfo:720) init_domain> Created domain=1 name=ExampleDomain memory=64
[2005-05-27 11:47:38 xend] INFO (console:94) Created console id=11 domain=1 port=9601
[2005-05-27 11:47:39 xend] DEBUG (XendDomainInfo:1130) Creating vbd dom=1 uname=file:/data/vesso/ttylinux-xen
[2005-05-27 11:47:39 xend] DEBUG (blkif:155) Connecting blkif <BlkifBackendInterface 1 0>
[2005-05-27 11:47:39 xend] DEBUG (XendDomainInfo:1107) Creating vif dom=1 vif=0 mac=aa:00:00:6f:68:cd
[2005-05-27 11:47:39 xend] INFO (XendRoot:112) EVENT> xend.console.create [11, 1, 9601]
[2005-05-27 11:47:39 xend] INFO (XendRoot:112) EVENT> xend.domain.create ['ExampleDomain', '1']
[2005-05-27 11:47:39 xend] INFO (XendRoot:112) EVENT> xend.domain.unpause ['ExampleDomain', '1']
[2005-05-27 11:47:40 xend] DEBUG (blkif:203) Connecting blkif to event channel <BlkifBackendInterface 1 0> ports=12:3
[2005-05-27 11:48:09 xend] INFO (console:44) Console connected 11 127.0.0.1 1092
[2005-05-27 11:48:09 xend] INFO (XendRoot:112) EVENT> xend.console.connect [11, '127.0.0.1', 1092]
[2005-05-27 11:48:38 xend] INFO (console:58) Console disconnected 11 127.0.0.1 1092
[2005-05-27 11:48:38 xend] INFO (XendRoot:112) EVENT> xend.console.disconnect [11, '127.0.0.1', 1092]
[2005-05-27 11:52:02 xend] INFO (XendMigrate:323) Migrate BEGIN: ['migrate', ['id', '1'], ['state', 'begin'], ['live', 1], ['resource', 0], ['src', ['host', 'mir'], ['domain', '1']], ['dst', ['host', 'giotto.doc.ic.ac.uk']]]
[2005-05-27 11:52:02 xend] INFO (XendRoot:112) EVENT> xend.domain.migrate ['ExampleDomain', '1', 'begin', ['migrate', ['id', '1'], ['state', 'begin'], ['live', 1], ['resource', 0], ['src', ['host', 'mir'], ['domain', '1']], ['dst', ['host', 'giotto.doc.ic.ac.uk']]]]
[2005-05-27 11:52:08 xend] DEBUG (XendDomain:487) domain_restart_schedule> 1 suspend 1
[2005-05-27 11:52:08 xend] INFO (XendRoot:112) EVENT> xend.domain.shutdown ['ExampleDomain', '1', 'suspend']
[2005-05-27 11:52:08 xend] DEBUG (XendDomain:244) XendDomain>reap> domain died name=ExampleDomain id=1
[2005-05-27 11:52:08 xend] DEBUG (XendDomain:247) XendDomain>reap> shutdown id=1 reason=suspend
[2005-05-27 11:52:08 xend] INFO (XendRoot:112) EVENT> xend.virq 4
[2005-05-27 11:52:08 xend] INFO (XendRoot:112) EVENT> xend.domain.suspended ['ExampleDomain', '1']
[2005-05-27 11:52:08 xend] INFO (XendDomain:564) Destroying domain: name=ExampleDomain
[2005-05-27 11:52:08 xend] DEBUG (XendDomainInfo:665) Destroying vifs for domain 1
[2005-05-27 11:52:08 xend] DEBUG (netif:305) Destroying vif domain=1 vif=0
[2005-05-27 11:52:08 xend] DEBUG (XendDomainInfo:674) Destroying vbds for domain 1
[2005-05-27 11:52:08 xend] DEBUG (blkif:552) Destroying blkif domain=1
[2005-05-27 11:52:08 xend] DEBUG (blkif:408) Destroying vbd domain=1 idx=0
[2005-05-27 11:52:08 xend] DEBUG (XendDomainInfo:634) Closing console, domain 1
[2005-05-27 11:52:08 xend] DEBUG (XendDomainInfo:622) Closing channel to domain 1
[2005-05-27 11:52:08 xend] INFO (XendRoot:112) EVENT> xend.domain.destroy ['ExampleDomain', '1']
[2005-05-27 11:52:08 xend] DEBUG (blkif:363) Unbinding vbd (type file) from /dev/loop0
[2005-05-27 11:52:08 xend] INFO (XendDomain:564) Destroying domain: name=ExampleDomain
[2005-05-27 11:52:08 xend] DEBUG (XendDomainInfo:634) Closing console, domain 1
[2005-05-27 11:52:08 xend] ERROR (XendDomainInfo:627) Domain destroy failed: ExampleDomain
Traceback (most recent call last):
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 625, in destroy_domain
    return xc.domain_destroy(dom=self.dom)
error: (3, 'No such process')
[2005-05-27 11:52:08 xend] INFO (XendRoot:112) EVENT> xend.domain.destroy ['ExampleDomain', '1']
[2005-05-27 11:52:08 xend] INFO (XendMigrate:345) Migrate OK: ['migrate', ['id', '1'], ['state', 'ok'], ['live', 1], ['resource', 0], ['src', ['host', 'mir'], ['domain', '1']], ['dst', ['host', 'giotto.doc.ic.ac.uk'], ['domain', 5]]]
[2005-05-27 11:52:08 xend] INFO (XendRoot:112) EVENT> xend.domain.died ['ExampleDomain', '1']
[2005-05-27 11:52:08 xend] INFO (XendRoot:112) EVENT> xend.domain.migrate ['ExampleDomain', '1', 'ok', ['migrate', ['id', '1'], ['state', 'ok'], ['live', 1], ['resource', 0], ['src', ['host', 'mir'], ['domain', '1']], ['dst', ['host', 'giotto.doc.ic.ac.uk'], ['domain', 5]]]]

network start bridge=xen-br0 netdev=eth0 antispoof=no
/usr/lib/python/xen/xend/server/controller.py:183: DeprecationWarning: Deferred.setTimeout is deprecated.  Look for timeout support specific to the API you are using instead.
  deferred.setTimeout(self.timeout)
vif-bridge up vif=vif5.0 domain=ExampleDomain mac=aa:00:00:6f:68:cd bridge=xen-br0
recv_fe_driver_status> {'status': 1}

recv_fe_interface_connect {'tx_shmem_frame': 4135, 'rx_shmem_frame': 4134, 'handle': 0}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=ExampleDomain mac=aa:00:00:6f:68:cd bridge=xen-br0

[2005-05-27 11:46:39 xend] INFO (SrvDaemon:610) Xend Daemon started
[2005-05-27 11:46:39 xend] INFO (XendRoot:112) EVENT> xend.start 0
[2005-05-27 11:52:02 xend] INFO (XendRoot:112) EVENT> xend.domain.create ['Domain-5', '5']
[2005-05-27 11:52:02 xend] DEBUG (XendDomain:384) domain_configure> id=5 config=['vm', ['name', 'ExampleDomain'], ['memory', '64'], ['cpu', '-1'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['ip', ':1.2.3.4::::eth0:dhcp'], ['root', '/dev/sda1 ro'], ['args', '4']]], ['device', ['vbd', ['uname', 'file:/data/vesso/ttylinux-xen'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vif', ['mac', 'aa:00:00:6f:68:cd']]]]
[2005-05-27 11:52:02 xend] INFO (console:94) Created console id=13 domain=5 port=9605
[2005-05-27 11:52:02 xend] DEBUG (XendDomainInfo:1130) Creating vbd dom=5 uname=file:/data/vesso/ttylinux-xen
[2005-05-27 11:52:02 xend] DEBUG (blkif:155) Connecting blkif <BlkifBackendInterface 5 0>
[2005-05-27 11:52:02 xend] DEBUG (XendDomainInfo:1107) Creating vif dom=5 vif=0 mac=aa:00:00:6f:68:cd
[2005-05-27 11:52:02 xend] INFO (XendRoot:112) EVENT> xend.console.create [13, 5, 9605]
[2005-05-27 11:52:02 xend] INFO (XendRoot:112) EVENT> xend.domain.create ['ExampleDomain', '5']
[2005-05-27 11:52:08 xend] DEBUG (blkif:203) Connecting blkif to event channel <BlkifBackendInterface 5 0> ports=15:4
[2005-05-27 11:53:41 xend] INFO (console:44) Console connected 13 127.0.0.1 37962
[2005-05-27 11:53:41 xend] INFO (XendRoot:112) EVENT> xend.console.connect [13, '127.0.0.1', 37962]
[2005-05-27 11:53:53 xend] DEBUG (XendDomain:244) XendDomain>reap> domain died name=ExampleDomain id=5
[2005-05-27 11:53:53 xend] INFO (XendDomain:564) Destroying domain: name=ExampleDomain
[2005-05-27 11:53:53 xend] DEBUG (XendDomainInfo:665) Destroying vifs for domain 5
[2005-05-27 11:53:53 xend] DEBUG (netif:305) Destroying vif domain=5 vif=0
[2005-05-27 11:53:53 xend] DEBUG (XendDomainInfo:674) Destroying vbds for domain 5
[2005-05-27 11:53:53 xend] DEBUG (blkif:552) Destroying blkif domain=5
[2005-05-27 11:53:53 xend] DEBUG (blkif:408) Destroying vbd domain=5 idx=0
[2005-05-27 11:53:53 xend] DEBUG (XendDomainInfo:634) Closing console, domain 5
[2005-05-27 11:53:53 xend] DEBUG (XendDomainInfo:622) Closing channel to domain 5
[2005-05-27 11:53:53 xend] INFO (XendRoot:112) EVENT> xend.virq 4
[2005-05-27 11:53:53 xend] DEBUG (blkif:363) Unbinding vbd (type file) from /dev/loop0
[2005-05-27 11:53:53 xend] INFO (console:58) Console disconnected 13 127.0.0.1 37962
[2005-05-27 11:53:53 xend] INFO (XendRoot:112) EVENT> xend.domain.exit ['ExampleDomain', '5', 'crash']
[2005-05-27 11:53:53 xend] INFO (XendRoot:112) EVENT> xend.domain.destroy ['ExampleDomain', '5']
[2005-05-27 11:53:53 xend] INFO (XendRoot:112) EVENT> xend.console.disconnect [13, '127.0.0.1', 37962]
[2005-05-27 11:53:54 xend] INFO (XendRoot:112) EVENT> xend.domain.died ['ExampleDomain', '5']

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

RE: Live-migrated domain crashes

Ian Pratt

> An if I try live-migrating it form hostB to hostA it only
> briefly appers on the hostA's 'xm list' output with status
> code 'p' before crashing.

Sounds like youre migrating the domain onto a different file system. You
need to be using NFS, iSCSI, GNBD etc.

Ian

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users