I came across a strange little networking problem earlier today.
The client in question had needed to be rebooted earlier in the day. The server had had no changes made to it. The problem only started occurring after the reboot of the client.
We had an NFS directory on the server that the client’s subnet had access to. However when we tried to mount the directory (via the automounter) it failed with a permission denied error. I should point out at this stage that the client is multihomed, that it it has several network interfaces onto different subnets and only one of these was allowed to access the share.
A little snooping showed that while the requests were mostly going out from the correct interface, some packets were coming out on an interface for another subnet. This shouldn’t occur since there is only one route packets can take out and its through the default router on the main interfaces subnet. It turns out however that we have 3 default routers on this machine – all active. This is likley a remnant of how the machine was installed, we copy in a defaultrouter file with multiple entries and since hosts usually only have one interface the rest get ignored. In the case of this system though the interface had been manually added over time and the routes were not activated, until the reboot then they all became active.
This meant that the system had 3 interfaces to contact the server on, but that the server would only authorise one of these interfaces. Since the interfaces had routes to the server the portmapper (rpcbind) was happily mapping ports on interfaces that we’d rather it didn’t.
Delete the routes, and delete the defaultrouter entries, restart rpcbind, and all works again.