Ethernet MTU

[Sylvain COUTANT]

Hi,

Is there any plan to raise supported ethernet mtu above 1500 ? Currently, packets are dropped above this point (tested against xen-3.0.2-testing). Even 802.1q tags require explicitly lowering the mtu to 1496 to work.


BR,

--
Sylvain


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Molle Bestefich]

Sylvain Coutant wrote:
> Is there any plan to raise supported ethernet mtu above 1500 ?
> Currently, packets are dropped above this point.
> Even 802.1q tags require explicitly lowering the mtu to 1496 to work.

I thought the IP layer was supposed to fragment packets if they do not
fit in the underlying ethernet's MTU.

(What's the point about fragmentation, otherwise?)

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[James Harper]

If this is a problem, it must be a new one - I've trunked a virtual
interface from dom0 to domU before without any hassles (except for a
memory leak which kills dom0 after a few days)

James

> -----Original Message-----
> From: [hidden email] [mailto:xen-devel-
> [hidden email]] On Behalf Of Sylvain Coutant
> Sent: Wednesday, 16 August 2006 20:14
> To: 'xen-devel'
> Cc: [hidden email]
> Subject: [Xen-devel] Ethernet MTU
>
> Hi,
>
> Is there any plan to raise supported ethernet mtu above 1500 ?

Currently,
> packets are dropped above this point (tested against
xen-3.0.2-testing).

> Even 802.1q tags require explicitly lowering the mtu to 1496 to work.
>
>
> BR,
>
> --
> Sylvain
>
>
> _______________________________________________
> Xen-devel mailing list
> [hidden email]
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[James Harper]

> > Even 802.1q tags require explicitly lowering the mtu to 1496 to
work.
>
> I thought the IP layer was supposed to fragment packets if they do not
> fit in the underlying ethernet's MTU.
>
> (What's the point about fragmentation, otherwise?)

802.1q is an ethernet protocol (vlan tagging for trunking multiple
vlan's over the same physical interface). It lives under IP.

James

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Molle Bestefich]

James Harper wrote:
> If this is a problem, it must be a new one - I've trunked a virtual
> interface from dom0 to domU before without any hassles (except for a
> memory leak which kills dom0 after a few days)

Fiddling with the MTU sizes has been mentioned as a workaround for
VLAN problems on xen-users countless times, AFAIR.

Quoting Cisco:

"The IP protocol was designed for use on a wide variety of
transmission links. Although the maximum length of an IP datagram is
64K, most transmission links enforce a smaller maximum packet length
limit, called a MTU. The value of the MTU depends on the type of the
transmission link. The design of IP accommodates MTU differences by
allowing routers to fragment IP datagrams as necessary. The receiving
station is responsible for reassembling the fragments back into the
original full size IP datagram.

IP fragmentation involves breaking a datagram into a number of pieces
that can be reassembled later. The IP source, destination,
identification, total length, and fragment offset fields, along with
the "more fragments" and "don't fragment" flags in the IP header, are
used for IP fragmentation and reassembly. For more information about
the mechanics of IP fragmentation and reassembly, please see RFC 791:
http://www.ietf.org/rfc/rfc791.txt
"

... just in case anyone wanted to read half a RFC on the mailing list :-)

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Molle Bestefich]

James Harper wrote:
> 802.1q lives under IP.

That's right.

But the VLAN exposes a virtual interface.
That virtual interface should automatically have an MTU size of the
underlying interface minus 4 bytes (for the VLAN tag).

And then IP fragmentation, working on top of that virtual Ethernet
interface, should perform fragmentation correctly.

Sorry if I was unclear.

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Sylvain COUTANT]

> James Harper wrote:
> > 802.1q lives under IP.
>
> That's right.
>
> But the VLAN exposes a virtual interface.
> That virtual interface should automatically have an MTU size of the
> underlying interface minus 4 bytes (for the VLAN tag).

It should. It does not. It's usually not a problem as networking equipment handle frames up to 1520-1540 bytes without much trouble, at least equipments able to manage tagged frames.


> And then IP fragmentation, working on top of that virtual Ethernet
> interface, should perform fragmentation correctly.

IP fragmentation handling is for routers when they connect two networks with different mtu sizes. This is not the case here, the IP stack generates frames that are as large as the mtu makes it possible.


Spamming xen-devel with non-Xen related network education and RFCs was not my intent, I was just wondering about jumbo frames support into Xen networking layer ...


BR,

--
Sylvain


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[James Harper]

> James Harper wrote:
> > 802.1q lives under IP.
>
> That's right.
>
> But the VLAN exposes a virtual interface.
> That virtual interface should automatically have an MTU size of the
> underlying interface minus 4 bytes (for the VLAN tag).
>
> And then IP fragmentation, working on top of that virtual Ethernet
> interface, should perform fragmentation correctly.
>
> Sorry if I was unclear.

That only works if all of the participating notes share the same
effective MTU. Consider this broken implementation:

A1----B1:B2-----C1:C2----D1

A and D are hosts
B and C are L2 routers (eg Linux bridges or Ethernet switches)

A1 is a normal Ethernet interface with an mtu of 1500
B1 is a normal Ethernet interface with an mtu of 1500
B2 is an 802.1q tagged interface with an mtu of 1496
C1 is an 802.1q tagged interface with an mtu of 1496
C2 is a normal Ethernet interface with an mtu of 1500
D1 is a normal Ethernet interface with an mtu of 1500

B1 is bridged to B2 vlan 4
C1 vlan 4 is bridged to C2

If A sends a packet with an MTU of 1500 out on the A1 interface, it is
received by B on B1, but then can't be forwarded via B2 because it won't
fit. There is no concept of fragmentation at the layer 2 level, that's a
layer 3 (IP) thing, so the packet is dropped (and presumably B would log
an error of some sort).

For 802.1q, Linux normally does some trickery where the MTU is bumped up
by 4 when a tag is involved, but as far as userspace is concerned the
MTU is always 1500 whether it's a native or tagged packet. I believe
there are still some hardware drivers (or maybe hardware
implementations) that are broken in this respect (1504 byte packets are
dropped), but otherwise it works well.

If you ever have to manually fiddle with MTU's then something is
horribly broken.

And yes, dammit, I've just realised too that i've been doing a reply-all
and crossposting. I guess I'm stuck with it now - I haven't subscribed
to xen-users. Sorry.

James

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Molle Bestefich]

Sylvain Coutant wrote:
> > But the VLAN exposes a virtual interface.
> > That virtual interface should automatically have an MTU size of the
> > underlying interface minus 4 bytes (for the VLAN tag).
>
> It should. It does not. It's usually not a problem

That's wrong.  Counted by the number of people on xen-users having
problems with VLANs and MTUs, it is very much a problem.


> > And then IP fragmentation, working on top of that virtual Ethernet
> > interface, should perform fragmentation correctly.
>
> IP fragmentation handling is for routers when they connect two networks
> with different mtu sizes. This is not the case here, the IP stack generates
> frames that are as large as the mtu makes it possible.

Really?  If you pass off 100kB of TCP data to the IP layer, will it
get split in correctly sized IP packets, but the fragment flag won't
be set?

Sounds odd, and I think you're wrong.
But I'd like to know for sure.

(The discussion is relevant either way, since a Xen domain is often
used as a router.)


> Spamming xen-devel with non-Xen related network education and RFCs
> was not my intent,

You cross-posted with xen-users.  I think the information is relevant
for a couple of people there, at least.


> I was just wondering about jumbo frames support into Xen networking layer

Fair nuff, hope someone can answer that?

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Molle Bestefich]

James Harper wrote:

> Consider this broken implementation:
>
> A1----B1:B2-----C1:C2----D1
>
> A and D are hosts
> B and C are L2 routers (eg Linux bridges or Ethernet switches)
>
> A1 is a normal Ethernet interface with an mtu of 1500
> B1 is a normal Ethernet interface with an mtu of 1500
> B2 is an 802.1q tagged interface with an mtu of 1496
> C1 is an 802.1q tagged interface with an mtu of 1496
> C2 is a normal Ethernet interface with an mtu of 1500
> D1 is a normal Ethernet interface with an mtu of 1500
>
> B1 is bridged to B2 vlan 4
> C1 vlan 4 is bridged to C2
>
> If A sends a packet with an MTU of 1500 out on the A1 interface, it is
> received by B on B1, but then can't be forwarded via B2 because it won't
> fit. There is no concept of fragmentation at the layer 2 level, that's a
> layer 3 (IP) thing, so the packet is dropped (and presumably B would log
> an error of some sort).

Darn, you're absolutely right.

Besides from solution A which you say (I don't think that's the case,
but OK) is already in place:
A) Bridge discards frame, logs error message.

Another compelling solution would be:

B) Use another type of switch/bridge, call it "intelligent switch" or
"VLAN switch" or whatever, which can inspect into the IP layer of
frames carrying IP packets and fragment them accordingly.

Does that exist?

A much much better solution than A (slightly worse than C perhaps),
while still keeping it simple, would be:

C) Fix the bridging code in Linux to reject interfaces which does not
match MTU-wise with the interfaces already in the bridge.

Solution C ensures that the administrator knows that there's a problem
beforehand, not when/if it randomly occurs.  Much better than A.

In the case of Xen, for C to be efficient, Xen should automatically
force the MTU of one end of the virtual wire to be the same as the MTU
on the interface at the other end, otherwise the point is moot, since
the bridge would think that all is OK with it's end of the virtual
wire having eg. MTU 1496, but it could still get oversized frames
because the other end has 1500.

(For real wire situations, solution A would still need to be in place,
since we can't synchronize those automatically..)


> For 802.1q, Linux normally does some trickery where the MTU is bumped up
> by 4 when a tag is involved, but as far as userspace is concerned the
> MTU is always 1500 whether it's a native or tagged packet.

So when configuring a VLAN on an interface, you say that Linux
automatically adds 4 to the MTU of the underlying physical interface?

Since Linux has no way of synchronizing the MTU with the interface at
the other end of the wire, that's really stupid.  It should just lower
the MTU of the virtual interface by 4 bytes instead.


> If you ever have to manually fiddle with MTU's then something is
> horribly broken.

Something definitely is...
I'd say probably more than one thing...


> And yes, dammit, I've just realised too that i've been doing a reply-all
> and crossposting. I guess I'm stuck with it now - I haven't subscribed
> to xen-users. Sorry.

I contemplated stripping xen-devel, but that seemed wrong too.

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Adrian Chadd]

On Wed, Aug 16, 2006, Sylvain Coutant wrote:
> Hi,
>
> Is there any plan to raise supported ethernet mtu above 1500 ? Currently, packets are dropped above this point (tested against xen-3.0.2-testing). Even 802.1q tags require explicitly lowering the mtu to 1496 to work.

Keeping this -just- on xen-devel -

Which interface are you talking about being > 1500 bytes?

Are you trunking a VLAN into a server via domU and then using
the bridging code to split these vlans up across different
vif's?

Or are you bringing in a VLAN trunk into a domU somehow and
running vlans inside the domU?

(Snip discussion about IP, fragmentation and the like. Its useful
to understand -why- there's a problem but not to understand
what you're trying to do; and thusly what needs fixing.)




Adrian


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Sylvain COUTANT]

> > Is there any plan to raise supported ethernet mtu above 1500 ?
> Currently, packets are dropped above this point (tested against xen-3.0.2-
> testing). Even 802.1q tags require explicitly lowering the mtu to 1496 to
> work.
>
> Keeping this -just- on xen-devel -
>
> Which interface are you talking about being > 1500 bytes?

My primary point is to support jumbo frames (9000 bytes mtu). My second point is to trunk domU to an external switch.

1/ I am able to set up a linux bridge with 9000 bytes mtu without trouble as long as it does not involve any Xen interface. And unable to set the mtu on vifs to something larger than 1500 bytes. I would like to configure my vifs to be compliant in mtu with the Ethernet network they are connected to which is full jumbo.


2/ I tried several scenario such as managing tags in dom0's (and setup as many bridges as vlans) or trunking vlan frames to domUs. In this later case, I need to bring down the mtu to 1496 for the frames to be forwarded across the bridge. Whatever code is rejecting the frames, it could be avoided if Xen's vifs were able to support larger mtus.


BR,

--
Sylvain COUTANT


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Herbert Xu]

Sylvain Coutant <[hidden email]> wrote:
>
> 1/ I am able to set up a linux bridge with 9000 bytes mtu without trouble as long as it does not involve any Xen interface. And unable to set the mtu on vifs to something larger than 1500 bytes. I would like to configure my vifs to be compliant in mtu with the Ethernet network they are connected to which is full jumbo.

You should be able to raise the MTU beyond 1500 with xen-unstable.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[hidden email]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Sylvain COUTANT]

> You should be able to raise the MTU beyond 1500 with xen-unstable.

This is the information I was just looking at. Thanks.

BTW, what is the current max mtu ?


BR.

--
Sylvain.


_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Herbert Xu]

On Wed, Aug 16, 2006 at 04:16:46PM +0200, Sylvain Coutant wrote:
>
> BTW, what is the current max mtu ?

65535 - 14 = 65521.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[hidden email]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Daniele Palumbo]

I have some news...

On Wednesday 16 August 2006 22:53, Dominique Rousseau wrote:
> After some tries, it looks like the vif code don't handle frames
> with size larger than 1500.

true:
http://lists.xensource.com/archives/html/xen-devel/2006-08/msg00944.html

> Looking at the code, there is the following snippet that seems to be
> the probleme (in drivers/xen/netback/netback.c) :
>
>                if (unlikely(txreq.size < ETH_HLEN) ||
>                    unlikely(txreq.size > ETH_FRAME_LEN)) {
>                        DPRINTK("Bad packet size: %d\n", txreq.size);
>                        make_tx_response(netif, txreq.id, NETIF_RSP_ERROR);
>                        netif_put(netif);
>                        continue;
>                }

./include/linux/if_vlan.h:#define VLAN_ETH_FRAME_LEN    1518    /* Max. octets
in frame sans FCS */
./include/linux/if_ether.h:#define ETH_FRAME_LEN        1514            /*
Max. octets in frame sans FCS */

./include/linux/if_vlan.h:#define VLAN_ETH_HLEN 18              /* Total
octets in header.       */
./include/linux/if_ether.h:#define ETH_HLEN     14              /* Total
octets in header.       */

netfront_vlan.patch:
---
# Node ID a5ee1cb525bbe8954dc3332ceec951e09a378b68
# parent: c097485037f7417b77db34da198b62a9c9481dc5
Make MTU rx check in netfront more permissive to allow for 8021q vlan tagging.

Signed-off-by: Keir Fraser <[hidden email]>

--- a/linux-2.6-xen-sparse/drivers/xen/netfront/netfront.c      Wed Mar 15
12:41:58 2006 +0100
+++ b/linux-2.6-xen-sparse/drivers/xen/netfront/netfront.c      Wed Mar 15
12:52:33 2006 +0100
@@ -845,11 +845,11 @@ static int netif_poll(struct net_device
        }

        while ((skb = __skb_dequeue(&rxq)) != NULL) {
-               if (skb->len > (dev->mtu + ETH_HLEN)) {
+               if (skb->len > (dev->mtu + ETH_HLEN + 4)) {
                        if (net_ratelimit())
                                printk(KERN_INFO "Received packet too big
for "
                                       "MTU (%d > %d)\n",
-                                      skb->len - ETH_HLEN, dev->mtu);
+                                      skb->len - ETH_HLEN - 4, dev->mtu);
                        skb->len  = 0;
                        skb->tail = skb->data;
                        init_skb_shinfo(skb);
---

looking at all that things, maybe the solutions is to change ETH_FRAME_LEN in
VLAN_ETH_FRAME_LEN...
or to remove that check (already done in unstable).

i'll try tomorrow.

bye
d.

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel

[Daniele Palumbo]

On Wednesday 13 September 2006 16:55, Daniele Palumbo wrote:
I made a working patch (tested by me, icmp and tcp).
This work *WITH* netfront_vlan.patch.

simply, the patch will use VLAN_ETH_FRAME_LEN (1518 bytes) instead of
ETH_FRAME_LEN (1514 bytes).
this patch will involve (for clear code) common.h (including vlans headers)
and netback.c (using #define in vlans headers ;), both in driver netback
directory.

this is something like "quick and dirthy", what do you think about it?

HTH
bye
d.

> netfront_vlan.patch:
> ---
> # Node ID a5ee1cb525bbe8954dc3332ceec951e09a378b68
> # parent: c097485037f7417b77db34da198b62a9c9481dc5
> Make MTU rx check in netfront more permissive to allow for 8021q vlan
> tagging.
>
> Signed-off-by: Keir Fraser <[hidden email]>
>
> --- a/linux-2.6-xen-sparse/drivers/xen/netfront/netfront.c      Wed Mar 15
> 12:41:58 2006 +0100
> +++ b/linux-2.6-xen-sparse/drivers/xen/netfront/netfront.c      Wed Mar 15
> 12:52:33 2006 +0100
> @@ -845,11 +845,11 @@ static int netif_poll(struct net_device
>         }
>
>         while ((skb = __skb_dequeue(&rxq)) != NULL) {
> -               if (skb->len > (dev->mtu + ETH_HLEN)) {
> +               if (skb->len > (dev->mtu + ETH_HLEN + 4)) {
>                         if (net_ratelimit())
>                                 printk(KERN_INFO "Received packet too big
> for "
>                                        "MTU (%d > %d)\n",
> -                                      skb->len - ETH_HLEN, dev->mtu);
> +                                      skb->len - ETH_HLEN - 4, dev->mtu);
>                         skb->len  = 0;
>                         skb->tail = skb->data;
>                         init_skb_shinfo(skb);
> ---

--
PGP Key-ID: 0xF482D454
--
to boldly go where no man has gone before.

_______________________________________________
Xen-devel mailing list
[hidden email]
http://lists.xensource.com/xen-devel