issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh

classic Classic list List threaded Threaded
8 messages Options
spi
Reply | Threaded
Open this post in threaded view
|

issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh

spi
Hi all

I filed this issue with the Debian user-list as well but as I think it is not Debian related I file it here as well.


I have issues with the on domU startup automatically generated
antispoofing rules by

/etc/xen/scripts/vif-bridge and
/etc/xen/scripts/vif-common.sh

Both are part of the Debian xen-utils-common package
(4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 installed on Debian 9.4).

A domU test01 has two virtual interfaces - vif-test01-INT and
vif-test01-TEST, both are connected to separate bridges named brINT and
brTEST. The brINT is just an internal bridge without connectivity to an
outside network to just connect all domUs and the dom0. The IP
addressfor the vif-test01-INT interface is 192.168.240.68.


The automatically generated rules per domU are:

1    ACCEPT     all  --  anywhere             anywhere            
PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged
2    ACCEPT     udp  --  anywhere             anywhere            
PHYSDEV match --physdev-in vif-test01-INT --physdev-is-bridged udp
spt:bootpc dpt:bootps
3    ACCEPT     all  --  anywhere             anywhere            
PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged
4    ACCEPT     all  --  192.168.240.68       anywhere            
PHYSDEV match --physdev-in vif-test01-INT --physdev-is-bridged
5    ACCEPT     all  --  anywhere             anywhere            
PHYSDEV match --physdev-out vif-test01-TEST --physdev-is-bridged
6    ACCEPT     udp  --  anywhere             anywhere            
PHYSDEV match --physdev-in vif-test01-TEST --physdev-is-bridged udp
spt:bootpc dpt:bootps
7    ACCEPT     all  --  anywhere             anywhere            
PHYSDEV match --physdev-out vif-test01-TEST --physdev-is-bridged
8    ACCEPT     all  --  test01               anywhere            
PHYSDEV match --physdev-in vif-test01-TEST --physdev-is-bridged
...
33   REJECT     all  --  anywhere             anywhere            
reject-with icmp-port-unreachable

From what I see is that the rules 1/3 and 5/7 are doubled.

The next issue is that antispoofing rules just don't work. If I change
the ip adress of the vif-test01-INT interface to something like
192.168.240.168 IP packets between test01 and other domUs are still
forwarded.


If I manually change the iptables rules to something like (in this
example just for the brINT connected interface):

-A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
-p all ! -s 192.168.240.68 -j DROP
-A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
-p all ! -d 192.168.240.68 -j DROP
-A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
-p all -j ACCEPT
-A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
-p all -j ACCEPT
...
-A FORWARD -j REJECT --reject-with icmp-port-unreachable

then antispoofing works and IP packets with IP addresses different then
192.168.240.68 get dropped.


Can somebody confirm this is an issue? Or do I just not understand how
the antispoofing rules work on a virtual bridge?

Is there a way to diable generation of antispoofing rules automatically
on domU startup? I could configure a different vif.default.script in
xl.conf and write a wrapper script, but it might be easier to just
disable it and load iptables rules manually.


-- Cheers, Sebastian EMail: [hidden email]

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh

Hans van Kranenburg-2
Hi Sebastian,

On 03/24/2018 04:03 PM, [hidden email] wrote:
> Hi all
>
> I filed this issue with the Debian user-list as well but as I think it is not Debian related I file it here as well.

What you can do best (for the Debian part of it) is file a bug against
src:xen in Debian to start a discussion.

The behaviour will likely not change any more for Xen 4.8 in Stretch,
but this is one of the things that should be improved for Xen 4.1X in
Buster. (I'm one of the people working on that.)

> I have issues with the on domU startup automatically generated
> antispoofing rules by
>
> /etc/xen/scripts/vif-bridge and
> /etc/xen/scripts/vif-common.sh

I myself use openvswitch instead of bridges, because it's so much more
convenient with vlans and in that situation all of the rules do not make
sense anyway, since since openvswitch traffic to domUs does not hit
iptables.

> Both are part of the Debian xen-utils-common package
> (4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 installed on Debian 9.4).
>
> A domU test01 has two virtual interfaces - vif-test01-INT and
> vif-test01-TEST, both are connected to separate bridges named brINT and
> brTEST. The brINT is just an internal bridge without connectivity to an
> outside network to just connect all domUs and the dom0. The IP
> addressfor the vif-test01-INT interface is 192.168.240.68.
>
>
> The automatically generated rules per domU are:
>
> 1    ACCEPT     all  --  anywhere             anywhere            
> PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged
> [...]

Because of the above, I have actually never studied this behaviour in
detail, I've just always patched it out, postponing looking for a proper
fix. (My network at work only contains domUs which are managed by
ourselves, noone else has shell/root access on them).

> From what I see is that the rules 1/3 and 5/7 are doubled.
>
> The next issue is that antispoofing rules just don't work. If I change
> the ip adress of the vif-test01-INT interface to something like
> 192.168.240.168 IP packets between test01 and other domUs are still
> forwarded.
>
> If I manually change the iptables rules to something like (in this
> example just for the brINT connected interface):
>
> -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
> -p all ! -s 192.168.240.68 -j DROP
> -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
> -p all ! -d 192.168.240.68 -j DROP
> -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
> -p all -j ACCEPT
> -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
> -p all -j ACCEPT
> ...
> -A FORWARD -j REJECT --reject-with icmp-port-unreachable
>
> then antispoofing works and IP packets with IP addresses different then
> 192.168.240.68 get dropped.
>
> Can somebody confirm this is an issue? Or do I just not understand how
> the antispoofing rules work on a virtual bridge?

The general feeling I have is that this area indeed needs some work to
clean it up and make it better. TBH, I have not looked back yet in xen
version control to see what the history of these scripts and the
iptables changes is.

Taking one step back, what are the things we want to do with whatever
kind of rules in the dom0?
* filtering irrelevant L2 unicast flooding
* whitelisting mac addresses
* whitelisting ip addresses
* ?

For bridges, some of these things can be done with iptables (if you also
set the right sysctl options to actually make bridged traffic hit
iptables...), but for filtering specific mac address / ip address etc
you need to know about them in the dom0. For openvswitch, it can
probably also be done in a different way. Then, what about properly
cleaning up things on domU shutdown, what about live migration, other
things...

So, there's not really a good out of the box solution for this now
afaics. We should find out what's there now, and why it was done like
that in the past and then see how it can be improved.

Your experience, opinion, feedback etc. matters. Thanks for starting
this thread. Best thing is to do the improvements "upstream" (here)
instead of in Debian.

> Is there a way to diable generation of antispoofing rules automatically
> on domU startup? I could configure a different vif.default.script in
> xl.conf and write a wrapper script, but it might be easier to just
> disable it and load iptables rules manually.

Remove/comment-out the handle_iptable line.

Hans

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
spi
Reply | Threaded
Open this post in threaded view
|

Re: issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh

spi
Hi Hans

Thanks for your reply.

>>
>> I filed this issue with the Debian user-list as well but as I think it is not Debian related I file it here as well.
>
> What you can do best (for the Debian part of it) is file a bug against
> src:xen in Debian to start a discussion.

Will do.

>
> The behaviour will likely not change any more for Xen 4.8 in Stretch,
> but this is one of the things that should be improved for Xen 4.1X in
> Buster. (I'm one of the people working on that.)
>

No issue with that. I just think something is not working here and
wasn't really sure, it was my understanding or really an issue.

>> I have issues with the on domU startup automatically generated
>> antispoofing rules by
>>
>> /etc/xen/scripts/vif-bridge and
>> /etc/xen/scripts/vif-common.sh
>
> I myself use openvswitch instead of bridges, because it's so much more
> convenient with vlans and in that situation all of the rules do not make
> sense anyway, since since openvswitch traffic to domUs does not hit
> iptables.
>

I am not familiar with openvswitch but like the idea of bridges as I can
filter traffic with iptables.


>> Both are part of the Debian xen-utils-common package
>> (4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 installed on Debian 9.4).
>>
>> A domU test01 has two virtual interfaces - vif-test01-INT and
>> vif-test01-TEST, both are connected to separate bridges named brINT and
>> brTEST. The brINT is just an internal bridge without connectivity to an
>> outside network to just connect all domUs and the dom0. The IP
>> addressfor the vif-test01-INT interface is 192.168.240.68.
>>
>>
>> The automatically generated rules per domU are:
>>
>> 1    ACCEPT     all  --  anywhere             anywhere            
>> PHYSDEV match --physdev-out vif-test01-INT --physdev-is-bridged
>> [...]
>
> Because of the above, I have actually never studied this behaviour in
> detail, I've just always patched it out, postponing looking for a proper
> fix. (My network at work only contains domUs which are managed by
> ourselves, noone else has shell/root access on them).
>

That's the case with me as well. All domUs are managed by trusted people
- me ;-) The issue though is another one. Some of the domUs are
connected to the Internet and host services. The risk here is that if
they get hacked an intruder got access to other internal machines as
well by manipulating domU's IP address. Think for instance of NFS
authentication based on IP addresses. One could argue here that a proper
NFS authentication is needed here and that's completely true, but NFSv4
with Kerberos tickets for domU file services sounds like a nightmare.
However dom0 and domU bring a perfect way for antispoofing rules - we
know and trust all the interfaces as they are managed by the dom0 and we
can apply filter on these. In a physical environment it is not that easy
to obtain. Binding IP addresses to MAC addresses doesn't address this
issue as MAC addresses easily can be spoofed too. So the equivalent in
the physical world would be one-to-one physical connections between
hosts and a firewall to bind IP addresses to real physical interfaces.
That's a huge benefit of virtualization. Antispoofing much easier to get.

>> From what I see is that the rules 1/3 and 5/7 are doubled.
>>
>> The next issue is that antispoofing rules just don't work. If I change
>> the ip adress of the vif-test01-INT interface to something like
>> 192.168.240.168 IP packets between test01 and other domUs are still
>> forwarded.
>>
>> If I manually change the iptables rules to something like (in this
>> example just for the brINT connected interface):
>>
>> -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
>> -p all ! -s 192.168.240.68 -j DROP
>> -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
>> -p all ! -d 192.168.240.68 -j DROP
>> -A FORWARD -m physdev --physdev-is-bridged --physdev-in vif-test01-INT
>> -p all -j ACCEPT
>> -A FORWARD -m physdev --physdev-is-bridged --physdev-out vif-test01-INT
>> -p all -j ACCEPT
>> ...
>> -A FORWARD -j REJECT --reject-with icmp-port-unreachable
>>
>> then antispoofing works and IP packets with IP addresses different then
>> 192.168.240.68 get dropped.
>>
>> Can somebody confirm this is an issue? Or do I just not understand how
>> the antispoofing rules work on a virtual bridge?
>
> The general feeling I have is that this area indeed needs some work to
> clean it up and make it better. TBH, I have not looked back yet in xen
> version control to see what the history of these scripts and the
> iptables changes is.
>

Honestly I think that that approach is a really good one if one uses
bridges. The issue though is that these automatically generated filters
don't work (and yes, I set the sysctl options...).

If looking for the generated filters they look good to me, they just
don't work. So it might be an iptables/bridges issue which is not really
XEN related. I am happy to file another report some place else but would
ask somebody to confirm that's really an issue which is not XEN related.

> Taking one step back, what are the things we want to do with whatever
> kind of rules in the dom0?
> * filtering irrelevant L2 unicast flooding
> * whitelisting mac addresses
> * whitelisting ip addresses
> * ?
>

These three are definitely important. For me antispoofing is important
for the reason I laid out earlier, but that's what you probably mean by
whitelisting. Although whitelisting mac addresses is important in the
end (most of the) services rely on IP that's why protecting IP addresses
is so crucial.

> For bridges, some of these things can be done with iptables (if you also
> set the right sysctl options to actually make bridged traffic hit
> iptables...), but for filtering specific mac address / ip address etc
> you need to know about them in the dom0. For openvswitch, it can
> probably also be done in a different way. Then, what about properly
> cleaning up things on domU shutdown, what about live migration, other
> things...
>

As said for bridges this approach works to set up and delete filters on
domU startup and shutdown as the to be assigned IP address is configured
on dom0 level in the domU config file. That IP address is being used for
the iptables filters and is outside the manageable scope of the domU.

> So, there's not really a good out of the box solution for this now
> afaics. We should find out what's there now, and why it was done like
> that in the past and then see how it can be improved.
>
> Your experience, opinion, feedback etc. matters. Thanks for starting
> this thread. Best thing is to do the improvements "upstream" (here)
> instead of in Debian.
>
>> Is there a way to diable generation of antispoofing rules automatically
>> on domU startup? I could configure a different vif.default.script in
>> xl.conf and write a wrapper script, but it might be easier to just
>> disable it and load iptables rules manually.
>
> Remove/comment-out the handle_iptable line.

That was exactly my approach to handle these filters and create them
directly/manually with iptables. I think now I'll change the scripts to
create the iptables rules automatically but change the rules to the ones
I suggested which seem to work in my environment.

Cheers,
Sebastian

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh

Wiebe Cazemier
In reply to this post by spi
> From: [hidden email]
> To: [hidden email]
> Sent: Saturday, 24 March, 2018 16:03:03
> Subject: [Xen-users] issue with iptables antispoofing rules in xen4.8 generetab
> by vif-bridge and vif-common.sh
>
> Hi all
>
> I filed this issue with the Debian user-list as well but as I think it is not
> Debian related I file it here as well.
>
> I have issues with the on domU startup automatically generated
> antispoofing rules by
>
> /etc/xen/scripts/vif-bridge and
> /etc/xen/scripts/vif-common.sh

On a side-note, the recommended way of configuring the network is doing it manually (i.e. defining the bridge in your OS configuration files). The issues with the script are numerous. For one, you can't do (the equivalent of) '/etc/init.d/networking restart', because then the Xen script is not run. Or you iptables state will fail because network devices aren't there yet. Etc.

See: https://wiki.xenproject.org/wiki/Xen_Networking#Setting_up_bridged_networking

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh

Hans van Kranenburg-2
On 03/26/2018 07:58 PM, Wiebe Cazemier wrote:

>> From: [hidden email]
>> To: [hidden email]
>> Sent: Saturday, 24 March, 2018 16:03:03
>> Subject: [Xen-users] issue with iptables antispoofing rules in xen4.8 generetab
>> by vif-bridge and vif-common.sh
>>
>> Hi all
>>
>> I filed this issue with the Debian user-list as well but as I think it is not
>> Debian related I file it here as well.
>>
>> I have issues with the on domU startup automatically generated
>> antispoofing rules by
>>
>> /etc/xen/scripts/vif-bridge and
>> /etc/xen/scripts/vif-common.sh
>
> On a side-note, the recommended way of configuring the network is doing it manually (i.e. defining the bridge in your OS configuration files). The issues with the script are numerous. For one, you can't do (the equivalent of) '/etc/init.d/networking restart', because then the Xen script is not run. Or you iptables state will fail because network devices aren't there yet. Etc.
>
> See: https://wiki.xenproject.org/wiki/Xen_Networking#Setting_up_bridged_networking

Are you confusing the old 'network-bridge' with 'vif-bridge'?

vif-bridge only attaches the dom0 side of the vif to the bridge.

Hans


_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh

Wiebe Cazemier
----- Original Message -----

> From: "Hans van Kranenburg" <[hidden email]>
> To: "Wiebe Cazemier" <[hidden email]>, [hidden email]
> Cc: [hidden email]
> Sent: Monday, 26 March, 2018 20:11:23
> Subject: Re: [Xen-users] issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh
>>
>> On a side-note, the recommended way of configuring the network is doing it
>> manually (i.e. defining the bridge in your OS configuration files). The issues
>> with the script are numerous. For one, you can't do (the equivalent of)
>> '/etc/init.d/networking restart', because then the Xen script is not run. Or
>> you iptables state will fail because network devices aren't there yet. Etc.
>>
>> See:
>> https://wiki.xenproject.org/wiki/Xen_Networking#Setting_up_bridged_networking
>
> Are you confusing the old 'network-bridge' with 'vif-bridge'?
>
> vif-bridge only attaches the dom0 side of the vif to the bridge.
>
> Hans

Oh, ha, apparently I am. Never mind :)

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
spi
Reply | Threaded
Open this post in threaded view
|

Re: issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh

spi
In reply to this post by Wiebe Cazemier
>> I filed this issue with the Debian user-list as well but as I think it is not
>> Debian related I file it here as well.
>>
>> I have issues with the on domU startup automatically generated
>> antispoofing rules by
>>
>> /etc/xen/scripts/vif-bridge and
>> /etc/xen/scripts/vif-common.sh
>
> On a side-note, the recommended way of configuring the network is doing it manually (i.e. defining the bridge in your OS configuration files). The issues with the script are numerous. For one, you can't do (the equivalent of) '/etc/init.d/networking restart', because then the Xen script is not run. Or you iptables state will fail because network devices aren't there yet. Etc.
>
> See: https://wiki.xenproject.org/wiki/Xen_Networking#Setting_up_bridged_networking
>
That's exactly how I configure my bridges - manually in Debian's
/etc/neworking/interfaces file. But point taken, it seems to make more
sense to configure iptables rules manually.

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
spi
Reply | Threaded
Open this post in threaded view
|

Re: issue with iptables antispoofing rules in xen4.8 generetab by vif-bridge and vif-common.sh

spi
In reply to this post by Wiebe Cazemier
>>>
>>> On a side-note, the recommended way of configuring the network is doing it
>>> manually (i.e. defining the bridge in your OS configuration files). The issues
>>> with the script are numerous. For one, you can't do (the equivalent of)
>>> '/etc/init.d/networking restart', because then the Xen script is not run. Or
>>> you iptables state will fail because network devices aren't there yet. Etc.
>>>
>>> See:
>>> https://wiki.xenproject.org/wiki/Xen_Networking#Setting_up_bridged_networking
>>
>> Are you confusing the old 'network-bridge' with 'vif-bridge'?
>>
>> vif-bridge only attaches the dom0 side of the vif to the bridge.
>>
>> Hans
>
> Oh, ha, apparently I am. Never mind :)
>
Lucky me, I was about to confess that I have no clue waht you were
talking about ;-)

But still - can anyone confirm the iptables rules as automatically
generated are not working?

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users