I guess this is the part where I admit that I'm really not all that good at networking. I tend to make silly mistakes and misunderstand basic concepts. Thankfully Luke is quite a bit better at it than I am.
So anyway. We were trying to figure out the cause of the error message "ip_conntrack: table full, dropping packet." I mean, obviously it comes from iptables' connection tracking, and tells us that it's dropping a packet because the table is full. Turn to proc:
# cat /proc/net/ip_conntrack | grep "^udp" | wc -l
31250
Uh-huh. Long story short, it was pretty obviously a hammered DNS server, running in a domU.
The problem can be "solved" by adding a rule to disable connection tracking for UDP packets to the "raw" table.
# iptables -t raw -I PREROUTING -p udp -j NOTRACK
At this point we broke local DNS resolution. Changing NOTRACK to ACCEPT fixed it again. Luke figured out the problem in short order, which is that the request goes out on a high-numbered port, the answer comes back, there's nothing to associate the two. It was really a goddamn epiphany. So we've got a way of solving our problem, but it's not exactly clean.
For right now, we're just increasing the limit on tracked connections until we figure out an iptables rule that allows us to turn off tracking for the FORWARD chain.
Ideologically speaking, we're not really interested in firewalling the clients' machines. Protecting our network, sure, but anything beyond that is your problem. Run whatever filters and firewalls you like. We're not really interested in connection tracking, and will be turning it off as soon as we figure out a way of doing so without disruption.
So anyway. We were trying to figure out the cause of the error message "ip_conntrack: table full, dropping packet." I mean, obviously it comes from iptables' connection tracking, and tells us that it's dropping a packet because the table is full. Turn to proc:
# cat /proc/net/ip_conntrack | grep "^udp" | wc -l
31250
Uh-huh. Long story short, it was pretty obviously a hammered DNS server, running in a domU.
The problem can be "solved" by adding a rule to disable connection tracking for UDP packets to the "raw" table.
# iptables -t raw -I PREROUTING -p udp -j NOTRACK
At this point we broke local DNS resolution. Changing NOTRACK to ACCEPT fixed it again. Luke figured out the problem in short order, which is that the request goes out on a high-numbered port, the answer comes back, there's nothing to associate the two. It was really a goddamn epiphany. So we've got a way of solving our problem, but it's not exactly clean.
For right now, we're just increasing the limit on tracked connections until we figure out an iptables rule that allows us to turn off tracking for the FORWARD chain.
Ideologically speaking, we're not really interested in firewalling the clients' machines. Protecting our network, sure, but anything beyond that is your problem. Run whatever filters and firewalls you like. We're not really interested in connection tracking, and will be turning it off as soon as we figure out a way of doing so without disruption.