best practices in using shared storage for XEN Virtual Machines and auto-failover?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

best practices in using shared storage for XEN Virtual Machines and auto-failover?

Rudi Ahlers-2
Hi all,

Can anyone pleas tell me what would be best practice to use shared
storage with virtual machines, especially when it involved high
availability / automated failover between 2 XEN servers?

i.e. if I setup 2x identical XEN servers, each with say 16GB RAM, 4x
1GB NIC's, etc. Then I need the xen domU's to auto failover between
the 2 servers if either goes down (hardware failure / overload /
kernel updates / etc).

What is the best way to connect a NAS / SAN to these 2 servers for
this kind of setup to work flawlessly? The NAS can export iSCSI, NFS,
SMB, etc. I'm sure I could even use ATAOE if needed

--
Kind Regards
Rudi Ahlers
SoftDux

Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

RE: best practices in using shared storage for XEN VirtualMachines and auto-failover?

admin-366
It is easier to use NFS than iSCSI for shared storage target.  If you use
iSCSI, you will need to use a cluster aware file system to avoid data
corruption.  If your storage target can do both, just use NFS.

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Rudi Ahlers
Sent: Thursday, October 14, 2010 6:25 AM
To: xen-users
Subject: [Xen-users] best practices in using shared storage for XEN
VirtualMachines and auto-failover?

Hi all,

Can anyone pleas tell me what would be best practice to use shared
storage with virtual machines, especially when it involved high
availability / automated failover between 2 XEN servers?

i.e. if I setup 2x identical XEN servers, each with say 16GB RAM, 4x
1GB NIC's, etc. Then I need the xen domU's to auto failover between
the 2 servers if either goes down (hardware failure / overload /
kernel updates / etc).

What is the best way to connect a NAS / SAN to these 2 servers for
this kind of setup to work flawlessly? The NAS can export iSCSI, NFS,
SMB, etc. I'm sure I could even use ATAOE if needed

--
Kind Regards
Rudi Ahlers
SoftDux

Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users



_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

RE: best practices in using shared storage for XEN VirtualMachines and auto-failover?

Jeff Sturm
In reply to this post by Rudi Ahlers-2
> -----Original Message-----
> From: [hidden email] [mailto:xen-users-
> [hidden email]] On Behalf Of Rudi Ahlers
> Sent: Thursday, October 14, 2010 7:25 AM
> To: xen-users
> Subject: [Xen-users] best practices in using shared storage for XEN
VirtualMachines
> and auto-failover?
>
> Hi all,
>
> Can anyone pleas tell me what would be best practice to use shared
storage with
> virtual machines, especially when it involved high availability /
automated failover
> between 2 XEN servers?

With 2 servers, I hear good things about DRBD, if you don't want to go
the SAN route.  If you have a SAN make sure it is sufficiently
redundant--i.e. two (or more) power supplies, redundant Ethernet, spare
controllers, etc.  And of course RAID 10 or similar RAID level to guard
against single-drive failure.

> i.e. if I setup 2x identical XEN servers, each with say 16GB RAM, 4x
1GB NIC's, etc.
> Then I need the xen domU's to auto failover between the 2 servers if
either goes down
> (hardware failure / overload / kernel updates / etc).

Pay close attention to power and networking.  With 4 NICs available per
host, I'd go for a bonded pair for general network traffic, and a
multipath pair for I/O.  Use at least two switches.  If you get it right
you should be able to lose one switch or one power circuit and maintain
connectivity to your critical hosts.

In my experience with high availability, the #1 mistake I see is
overthinking the esoteric failure modes and missing the simple stuff.
The #2 mistake is inadequate monitoring to detect single device
failures.  I've seen a lot of mistakes that are simple to correct:

- Plugging a bonded Ethernet pair into the same switch.
- Connecting dual power supplies to the same PDU.
- Oversubscribing a power circuit.  When a power supply fails, power
draw on the remaining supply will increase--make sure this increase
doesn't overload and trip a breaker.
- Ignoring a drive failure until the 2nd drive fails.

You can use any of a variety of clustering tools, like heartbeat, to
automate the domU failover.  Make sure you can't get into split-brain
mode, where a domU can start on two nodes at once--that would quickly
corrupt a shared filesystem.  With any shared storage configuration,
node fencing is generally an essential requirement.

> What is the best way to connect a NAS / SAN to these 2 servers for
this kind of setup
> to work flawlessly? The NAS can export iSCSI, NFS, SMB, etc. I'm sure
I could even
> use ATAOE if needed

For my money I'd go with iSCSI (or AoE), partition my block storage and
export whole block devices as disk images for the domU guests.  If your
SAN can't easily partition your storage, consider a clustered logical
volume manager like CLVM on RHCS.

-Jeff



_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

RE: best practices in using shared storage for XENVirtualMachines and auto-failover?

Jeff Sturm
In reply to this post by admin-366
> -----Original Message-----
> From: [hidden email] [mailto:xen-users-
> [hidden email]] On Behalf Of [hidden email]
> Sent: Thursday, October 14, 2010 8:37 AM
> To: 'xen-users'
> Subject: RE: [Xen-users] best practices in using shared storage for
XENVirtualMachines
> and auto-failover?
>
> It is easier to use NFS than iSCSI for shared storage target.  If you
use iSCSI, you will
> need to use a cluster aware file system to avoid data corruption.  If
your storage
> target can do both, just use NFS.

You don't need a clustered filesystem.  You don't need a shared
filesystem at all.  Just partition your shared block storage (with a
clustered logical volume manager if needed) and use a logical volume for
each domU image.

-Jeff



_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: best practices in using shared storage for XENVirtualMachines and auto-failover?

Freddie Cash-8
On Thu, Oct 14, 2010 at 6:45 AM, Jeff Sturm <[hidden email]> wrote:

>> -----Original Message-----
>> From: [hidden email] [mailto:xen-users-
>> [hidden email]] On Behalf Of [hidden email]
>> Sent: Thursday, October 14, 2010 8:37 AM
>> To: 'xen-users'
>> Subject: RE: [Xen-users] best practices in using shared storage for
> XENVirtualMachines
>> and auto-failover?
>>
>> It is easier to use NFS than iSCSI for shared storage target.  If you
> use iSCSI, you will
>> need to use a cluster aware file system to avoid data corruption.  If
> your storage
>> target can do both, just use NFS.
>
> You don't need a clustered filesystem.  You don't need a shared
> filesystem at all.  Just partition your shared block storage (with a
> clustered logical volume manager if needed) and use a logical volume for
> each domU image.

Trying to get this straight in my head.  :)

Would you run the iSCSI initiator in the host and attach the block
device to the guest?
Or would you run the iSCSI initiator in the guest?

How does the migration from one host machine to another work with iSCSI?

--
Freddie Cash
[hidden email]

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

RE: best practices in using shared storage forXENVirtualMachines and auto-failover?

Jeff Sturm
> -----Original Message-----

> From: [hidden email] [mailto:xen-users-
> [hidden email]] On Behalf Of Freddie Cash
> Sent: Thursday, October 14, 2010 11:38 AM
> To: xen-users
> Subject: Re: [Xen-users] best practices in using shared storage forXENVirtualMachines
> and auto-failover?
>
> > You don't need a clustered filesystem.  You don't need a shared
> > filesystem at all.  Just partition your shared block storage (with a
> > clustered logical volume manager if needed) and use a logical volume
> > for each domU image.
>
> Would you run the iSCSI initiator in the host and attach the block device to the guest?
> Or would you run the iSCSI initiator in the guest?
I like to do the former.  That makes the guest easier--you don't have to figure out things like modifying initrd to do boot-from-SAN.

> How does the migration from one host machine to another work with iSCSI?

No different than migration with any other shared storage.  Just make sure each dom0 has a working initiator, and is logged into the appropriate targets.  If your SAN has ACLs, you'll need to allow access from every possible dom0.

-Jeff


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: best practices in using shared storage forXENVirtualMachines and auto-failover?

Freddie Cash-8
On Thu, Oct 14, 2010 at 8:59 AM, Jeff Sturm <[hidden email]> wrote:

>> -----Original Message-----
>> From: [hidden email] [mailto:xen-users-
>> [hidden email]] On Behalf Of Freddie Cash
>> Sent: Thursday, October 14, 2010 11:38 AM
>> To: xen-users
>> Subject: Re: [Xen-users] best practices in using shared storage forXENVirtualMachines
>> and auto-failover?
>>
>> > You don't need a clustered filesystem.  You don't need a shared
>> > filesystem at all.  Just partition your shared block storage (with a
>> > clustered logical volume manager if needed) and use a logical volume
>> > for each domU image.
>>
>> Would you run the iSCSI initiator in the host and attach the block device to the guest?
>> Or would you run the iSCSI initiator in the guest?
>
> I like to do the former.  That makes the guest easier--you don't have to figure out things like modifying initrd to do boot-from-SAN.
>
>> How does the migration from one host machine to another work with iSCSI?
>
> No different than migration with any other shared storage.  Just make sure each dom0 has a working initiator, and is logged into the appropriate targets.  If your SAN has ACLs, you'll need to allow access from every possible dom0.

So how does the migration work?  This is the part that is still
tripping me up.  :)

You have HostA configured with the iSCSI initiator, with VM1 using the
block device for storage, running all happily.

Then you tell VM1 to migrate to HostB.

Does HostB already have the iSCSI initiator running?  Does it get
started when the migration starts?  What happens to the initiator on
HostA?

I understand how it works with an NFS share and a disk image file.
It's the whole "exported block device on the network" bit that's
messing me up.

--
Freddie Cash
[hidden email]

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: best practices in using shared storage forXENVirtualMachines and auto-failover?

John Madden-4
> Then you tell VM1 to migrate to HostB.
>
> Does HostB already have the iSCSI initiator running?  Does it get
> started when the migration starts?  What happens to the initiator on
> HostA?

Yes, both dom0's should be connected to the storage and be able to see
it at all times.  You have to fence or otherwise "uhm, guarantee" that
only one dom0 will be writing to the block devices at any point in time.
  cLVM won't do this for you but other components of the cluster
services will.

> I understand how it works with an NFS share and a disk image file.
> It's the whole "exported block device on the network" bit that's
> messing me up.

Try to not distinguish too much between the two in this case.  You can't
go writing to a block device from two different hosts at the same time
without a cluster filesystem, but even live migration doesn't allow such
writes anyway.

John




--
John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana
[hidden email]

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: best practices in using shared storage for XEN Virtual Machines and auto-failover?

khris4
In reply to this post by Rudi Ahlers-2
iscsi and drbd would be a great way to go. That is the setup i use at my location
Reply | Threaded
Open this post in threaded view
|

Re: best practices in using shared storage for XENVirtualMachines and auto-failover?

Rudi Ahlers-2
In reply to this post by Freddie Cash-8
On Thu, Oct 14, 2010 at 5:37 PM, Freddie Cash <[hidden email]> wrote:

> Trying to get this straight in my head.  :)
>
> Would you run the iSCSI initiator in the host and attach the block
> device to the guest?
> Or would you run the iSCSI initiator in the guest?
>
> How does the migration from one host machine to another work with iSCSI?
>
> --
> Freddie Cash
> [hidden email]
>
> _______________________________________________


This is actually what I want to know as well.

I know the different technologies (AOE / iSCSI / NFS / etc) all have
different pro's & cons, but how do I actually use them optimally?

AOE seems to have less network overhead since it runs on the network
layer and not on TCP or IP, so this is probably a good option to go
for. I'm thinking of setting up a few NAS servers with CentOS + CLVM,
or maybe even a few NAS servers with Gluster and then export the file
systems from them to the XEN servers.

But, do I export it to the dom0, or the domU's directly? That I don't know.

--
Kind Regards
Rudi Ahlers
SoftDux

Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: best practices in using shared storage for XEN VirtualMachines and auto-failover?

Rudi Ahlers-2
In reply to this post by Jeff Sturm
On Thu, Oct 14, 2010 at 3:42 PM, Jeff Sturm <[hidden email]> wrote:

>> -----Original Message-----
>> From: [hidden email] [mailto:xen-users-
>> [hidden email]] On Behalf Of Rudi Ahlers
>> Sent: Thursday, October 14, 2010 7:25 AM
>> To: xen-users
>> Subject: [Xen-users] best practices in using shared storage for XEN
> VirtualMachines
>> and auto-failover?
>>
>> Hi all,
>>
>> Can anyone pleas tell me what would be best practice to use shared
> storage with
>> virtual machines, especially when it involved high availability /
> automated failover
>> between 2 XEN servers?
>
> With 2 servers, I hear good things about DRBD, if you don't want to go
> the SAN route.  If you have a SAN make sure it is sufficiently
> redundant--i.e. two (or more) power supplies, redundant Ethernet, spare
> controllers, etc.  And of course RAID 10 or similar RAID level to guard
> against single-drive failure.

I am planning on setting up a SAN with a few Gluster / CLVM servers -
just need to decide which one first, but I'm going to attemp high
availability + load balancing + ease-of-upgrade-with-no-downtime. Each
server will run RAID10 (maybe RAID6?)


> Pay close attention to power and networking.  With 4 NICs available per
> host, I'd go for a bonded pair for general network traffic, and a
> multipath pair for I/O.  Use at least two switches.  If you get it right
> you should be able to lose one switch or one power circuit and maintain
> connectivity to your critical hosts.

So would you bond eth0 & eth1, and then eth2 & eth3 together? But then
connect the bonded eth0+1 one one switch, and eth2+3 on another switch
for failover? Or would you have eth0 & eth2 on one switch, and eth1 &
eth3 on the other? Is this actually possible? I presume the 2 switches
should also be connected together (preferably via fiber?) and then
setup Spanning Tree? Or should I seperate the 2 networks,and connect
them indivually to the internet?

>
> In my experience with high availability, the #1 mistake I see is
> overthinking the esoteric failure modes and missing the simple stuff.
> The #2 mistake is inadequate monitoring to detect single device
> failures.  I've seen a lot of mistakes that are simple to correct:
>
> - Plugging a bonded Ethernet pair into the same switch.
> - Connecting dual power supplies to the same PDU.
> - Oversubscribing a power circuit.  When a power supply fails, power
> draw on the remaining supply will increase--make sure this increase
> doesn't overload and trip a breaker.
> - Ignoring a drive failure until the 2nd drive fails.
>
> You can use any of a variety of clustering tools, like heartbeat, to
> automate the domU failover.  Make sure you can't get into split-brain
> mode, where a domU can start on two nodes at once--that would quickly
> corrupt a shared filesystem.  With any shared storage configuration,
> node fencing is generally an essential requirement.
>
>> What is the best way to connect a NAS / SAN to these 2 servers for
> this kind of setup
>> to work flawlessly? The NAS can export iSCSI, NFS, SMB, etc. I'm sure
> I could even
>> use ATAOE if needed
>
> For my money I'd go with iSCSI (or AoE), partition my block storage and
> export whole block devices as disk images for the domU guests.  If your
> SAN can't easily partition your storage, consider a clustered logical
> volume manager like CLVM on RHCS.
>
> -Jeff
>

I am considering CLVM, or Gluster - just need to play with them and
decide which one I prefer :)

>
>
> _______________________________________________
> Xen-users mailing list
> [hidden email]
> http://lists.xensource.com/xen-users
>



--
Kind Regards
Rudi Ahlers
SoftDux

Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

RE: best practices in using shared storage forXENVirtualMachines and auto-failover?

admin-366
In reply to this post by Rudi Ahlers-2
You would export the NFS so the dom0 can connect to it to store data.  The
domU would not need to access the NFS share (or iSCSI target or what ever
storage you used).  As far as the domU are concerned, the targets that the
dom0 has mounted would simply appear to be local drives.

> _______________________________________________


This is actually what I want to know as well.

I know the different technologies (AOE / iSCSI / NFS / etc) all have
different pro's & cons, but how do I actually use them optimally?

AOE seems to have less network overhead since it runs on the network
layer and not on TCP or IP, so this is probably a good option to go
for. I'm thinking of setting up a few NAS servers with CentOS + CLVM,
or maybe even a few NAS servers with Gluster and then export the file
systems from them to the XEN servers.

But, do I export it to the dom0, or the domU's directly? That I don't know.

--
Kind Regards
Rudi Ahlers
SoftDux

Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users



_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users
Reply | Threaded
Open this post in threaded view
|

RE: best practices in using shared storage for XEN VirtualMachines and auto-failover?

Jeff Sturm
In reply to this post by Rudi Ahlers-2
> -----Original Message-----
> From: Rudi Ahlers [mailto:[hidden email]]
> Sent: Friday, October 15, 2010 5:11 AM
> To: Jeff Sturm
> Cc: xen-users
> Subject: Re: [Xen-users] best practices in using shared storage for XEN
> VirtualMachines and auto-failover?
>
> On Thu, Oct 14, 2010 at 3:42 PM, Jeff Sturm <[hidden email]> wrote:
> > With 2 servers, I hear good things about DRBD, if you don't want to go
> > the SAN route.  If you have a SAN make sure it is sufficiently
> > redundant--i.e. two (or more) power supplies, redundant Ethernet,
> > spare controllers, etc.  And of course RAID 10 or similar RAID level
> > to guard against single-drive failure.
>
> I am planning on setting up a SAN with a few Gluster / CLVM servers - just need to
> decide which one first, but I'm going to attemp high availability + load balancing +
> ease-of-upgrade-with-no-downtime. Each server will run RAID10 (maybe RAID6?)

Sounds like a fun project.  Never tried Gluster, myself.

> So would you bond eth0 & eth1, and then eth2 & eth3 together? But then connect the
> bonded eth0+1 one one switch, and eth2+3 on another switch for failover? Or would
> you have eth0 & eth2 on one switch, and eth1 &
> eth3 on the other? Is this actually possible?

We generally do the latter.  It's certainly possible, with active-backup bonding mode for IP traffic on eth0/1, and multipath for SAN traffic on eth2/3.

The switches are not required to cooperate in this mode (unlike active-active or LACP), but should remain connected to one another in case two hosts have their active links on different switches. You'll see that happen if you unplug a cable on the active link on any host--the backup link will become active for that host only.

> I presume the 2 switches should also be
> connected together (preferably via fiber?) and then setup Spanning Tree? Or should I
> seperate the 2 networks,and connect them indivually to the internet?

You wouldn't use spanning tree with bonding--use one or the other.   I've had some luck with Linux bridging and spanning tree, but the network delays in learning mode may be intolerable.  (Linux bridging has no support for RSTP, yet.)

Try it yourself.  I'd encourage you to run your own tests by physically unplugging cables and powering down switches before you go into production.

-Jeff



_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xensource.com/xen-users