Linux: How do I force a specific network interface to be used?

Question

This could be considered a continuation of this earlier SO question.

Ideally, I'd like to jail a process into only using a certain interface, no matter what. It will be making TCP connections, sending UDP datagrams, and listening for UDP broadcasts. Currently, what I am doing is:

1. Determine the IP of the interface to use.
2. Create an IP policy rule to route all packets coming from the interface to that IP
3. Create another IP policy rule to route all packets coming from that IP to that interface
4. Set up a default routing table for each rule

Now, this works, mostly, but the client process has to also be willing to play along. That is, it needs to bind to the specific IP of the interface it wants to use, and I think I need to set SO_BINDTODEVICE as well. (However, I keep reading conflicting information about whether SO_BINDTODEVICE actually works when using TCP or UDP.) Luckily, the client application is Python, and I can extend the socket class to do all of this transparently. But I'm not sure it is a complete solution, especially with regard to receiving broadcasts.

My questions are:

1. Does SO_BINDTODEVICE do what I want here? Or is it only effective for raw sockets? Someone commented that, "SO_BINDTODEVICE on a socket doesn't guarantee the socket will only receive packets that arrived on that physical interface's wire/antenna." If this is indeed true, then what does SO_BINDTODEVICE do?

2. Is there a way to do this such that the local IP doesn't have to be unique? This wouldn't be a problem other than the fact that the DHCP server on one interface may allocate it an IP that is in use by another interface, thus confusing the routing table.

3. How do I receive broadcasts only from a specific interface? Binding to a specific IP seems to make it ignore broadcasts, which makes sense, but isn't quite what I'm looking for.

I'm running on Ubuntu 8.04 w/ Linux kernel 2.6.26. Being able to access the same subnet on two different networks through two different interfaces simultaneously is a non-negotiable requirement, thus making it (mostly) immune to "don't do that." :)

Answer 1

As for my general question, there appear to be a few ways to do it:

• The complicated way that involves the routing table changes and cooperation from each process. This is the way I described above. One advantage it has it it works from userspace. I have put some additional notes on it and answered my specific questions below.

• Write a custom kernel model that ignores the routing table completely, if SO_BINDTODEVICE is set. The client process is still required to call setsockopt(SOL_SOCKET, SO_BINDTODEVICE, dev), however. This option is definitely not for the faint of heart.

• Virtualize the process. This probably isn't appropriate for a lot of people, and it will brings own set of headaches, mostly with configuration. But it is worth a mention.

Options 1 and 2 require processes to opt-in to work as we'd like them to. This can be partially alleviated by creating a dynamic library that hijacks the socket() call to create the socket and then immediately bind it to the device before returning the descriptor. This is described in more detail here.

After doing some research and lots of Googling, I can draw a few conclusions on how the Linux kernel 2.6.26 behaves. Note that these are probably all implementation-specific behaviors, and perhaps even kernel-specific. Test your own platform before deciding to implement features based upon my single point of data.

1. SO_BINDTODEVICE does indeed do what it says, at least for UDP.

2. Unique IPs for each interface seems to be necessary, since we're using the routing tables. A custom kernel module could bypass this restriction.

3. To receive broadcasts on a specific interface, bind to the device first using SO_BINDTODEVICE, and then bind to the broadcast address with the usual bind() call. The device binding needs to be done before anything else. Then, the socket will receive only broadcasts that arrive on that interface.

I tested it by first creating a socket. Then I bound it to a specific interface using setsockopt(SOL_SOCKET, SO_BINDTODEVICE, dev). Finally, I bound it to the broadcast address. From another computer, I sent a broadcast that would be received through a nonbound interface. The device-bound socket did not receive this broadcast, which makes sense. Remove the setsockopt(SOL_SOCKET, SO_BINDTODEVICE, dev) call and the broadcast will be received.

It should also be mentioned that you can use setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) here as well. Note that the semantics of SO_REUSEADDR change with broadcast addresses. Specifically, it is legal to have two sockets bound to the broadcast address and the same port on the same machine if they both have SO_REUSEADDR set.

UPDATE: SO_BINDTODEVICE with broadcasts seems to be fraught with peril, specifically, reception of broadcast frames. I was observing broadcast frames received on one interface, but disappearing on the other at the same time. It appears that they are effected by the local routing table, but are immune to IP policy rules. However, I'm not 100% certain about this and am only mentioning it as a point of investigation if you wish to continue it. All this to say: use at your own risk. In the interest of time, I opened a raw socket on the interface and parsed the Ethernet and IP headers myself.

Answer 2

Not a direct answer to your question, but just an FYI. As you mentioned above, this solution may be too much work for what you need/want to do.

I personally like the idea of creating a network stack hook kernel module that will allow me to do this. This way I have full control over multicast and unitcast frames going and coming from userspace. You'd have to use something like netlink sockets to send/receive data to and from your driver and userspace application, but it works very well and is very fast.

You also get to hook into any level of the stack this way... Ethernet, or IP. Thus having full control over what you send/receive.

Here's an example article that talks about hooking into the netfilter stack.
Note: this article hooks into the IP stack, and it's also old. I know that the APIs have changed, but a lot of this article still applies practically and theoretically. If you wanted to hook into the bridging layer, you would use the a similar mechanism, but specify

BR_LOCAL_IN instead of NF_IP_LOCAL_IN

Note: This is very similar to opening a raw socket on the interface. You'll have to build your frames yourself.

Answer 3

After a hard-fought weekend, I'm pleased to present a solution that addresses most of what I've previously discussed with almost zero hassle.

There is a sysctl called net.ipv4.conf.all.rp_filter that can be set to 0 to disable source validation:

    rp_filter - INTEGER
         2 - do source validation by reversed path, as specified in RFC1812
             Recommended option for single homed hosts and stub network
             routers. Could cause troubles for complicated (not loop free)
             networks running a slow unreliable protocol (sort of RIP),
             or using static routes.

         1 - (DEFAULT) Weaker form of RP filtering: drop all the packets
             that look as sourced at a directly connected interface, but
             were input from another interface.

         0 - No source validation.

This can also be set on a per interface basis using /proc/sys/net/ipv4/conf/<interface>/rp_filter.

As one poster explained it, it makes IP routing "less deterministic" in the sense that packets coming from one subnet aren't guaranteed to always go out the same interface. In this instance, this is exactly what it is needed. Please do additional research to determine if this is really what you want.

Broadcasts are still problematic for reasons I do not understand, but I am finally satisfied with this issue and I hope it helps others.