One single IP address won't forward but all the rest do

I had a user report that one node couldn’t access the voip server over the ipsec vpn, but all the rest would. Perplexing. Not only could any other node on the remote network connect to the server, but I could ping the node that wouldn’t talk from that server.

The edge router is a Ubiquity EdgeRouter Lite which runs linux. From the shell, I ran a tcpdump and, sure enough, the node that failed to connect was getting ICMP Destination Unreachable for that one server at that one UDP port sourced from the router itself.

I’m loathe to resort to the “Microsoft Method” for fixing problems as made famous by Chris O’Dowd in the IT Crowd: “Have you tried turning it off and on again?”

After some time I finally found the problem. I’m still not sure where that problem came from, but my guess is that before openswan started, the node tried to establish a connection with the voip server. That started a connection in conntrack that created this entry:

udp     17 3574 src=10.23.0.122 dst=10.99.0.254 sport=5060 dport=5060   
   packets=118486 bytes=59987941  [UNREPLIED] src=10.99.0.254   
   dst=1.2.3.4 sport=5060 dport=1025 packets=0 bytes=0 mark=0   
   secmark=0 use=2

As long as that entry remained, every attempt to connect to the same host was routed out the gateway address instead of going through the vpn.

To repair the problem I simply ran

conntrack -F

Since there were no masquerade connections that I cared whether or not they remained established that was sufficient. A less aggressive approach would be to just delete the one line that was causing the problem.