Mastering NFS Protocol Versions and analyze required Traffic
When firewalls are in place between NFS clients and servers, it can be complex when providing NFS version 3 and lower.
Each NFS version introduces new features and changes in how the client and server communicate, the ports used, and how security and performance are handled. Whether it’s the reliance on RPC calls and dynamic ports in NFSv2/v3 or the shift to a more stateful and firewall-friendly architecture in NFSv4, knowing the underlying mechanics is crucial for effective troubleshooting, network design, and performance optimization.
This post will help to understand NFS traffic by breaking down the core communication patterns, ports, and differences across the protocol versions, providing clarity for administrators, engineers, and anyone working with NFS in their environments.
List Supported NFS Versions via rpcinfo
Below we will see to how to determine the supported and enabled NFS versions on the NFS server and on clients who wants to mount NFS exports (shares) from.
On the Client to determine the supported NFS versions of a remote NFS Server
On the client, query the NFS server by using the rpcinfo command to see which NFS versions it supports:
Because the NFS protocol was designed to use Remote Procedure Call (RPC) as the foundation, we can just query the rpc portmapper (on tcp/udp port 111, also on Windows NFS Server) to list all versions, protocols and ports the NFS server will support and listening on.
Remote Procedure Call (RPC) were used for the NFS protocol to abstract the network layer and therefore simplifying access of files on remote servers as if they were local by using local procedure calls (open, read, write, close) instead of also managing low-level network details
rpcinfo -p <server_ip remote NFS Server> This requires traffic to the rpc portmapper listening on tcp/udp port 111 on the NFS server is allowed.
So below we can see that the NFS server Matrix-Files.matrixpost-lab.net finally supports NFS version 2, 3 & 4.
NFS version 4 does not use UDP. If you mount a file system with the proto=udp option, then NFS version 3 is used instead of version 4.
On the Server
To determine directly on the NFS server the supported NFS versions, run:
# rpcinfo -p | grep nfs
You should see NFS v2, v3, and v4 services:
# rpcinfo -p
Finally the NFS services (v3 and v2) generally require 5 services be running and accessible through a firewall, more further down.
On Windows NFS Server we can use the following PowerShell cmdlet.
PS> Get-NfsServerConfiguration
By using the Set-NfsServerConfiguration cmdlet we can control which versions are enabled or not.
PS> Set-NfsServerConfiguration -EnableNfsv2 $False -EnableNfsv4 $True -EnableNfsv3 $True
Source: https://learn.microsoft.com/en-us/powershell/module/nfs/set-nfsserverconfiguration
More about how to set up an NFS Server in Windows Server 2022 you will find in my following post.
Mounting NFS shares with a specific protocol version
If supported and enabled on the server and client, clients will mount the NFS export by using NFSv4 by default if not explicitly set using the proto=<2,3,4> flag below.
To mount an NFSv3 share explicitly by using the vers=3 flag.
# mount -t nfs -o vers=3 <server_ip>:/export/path /mnt # mount -t nfs -o vers=3,proto=udp 10.0.0.151:/NFS-Exports/share01 /nfs-data/ -t, --types <list> limit the set of filesystem types -o, --options <list> comma-separated list of mount options -o nfsvers=<1,2,3,4> or just -o vers=<1,2,3,4> -o proto=<tcp,udp> or just -o <tcp,udp>
Nfs-Mountd Service
mountd is the NFS mount daemon. It responds to client mount requests for NFSv2 and NFSv3 (not required for NFSv4).
NFSv4 does not use mountd. Mount requests goes directly to the NFS daemon listening on TCP port 2049.
The NFS mount daemon provides the necessary metadata and access control mechanisms before a client can mount an NFS export. It responds to client requests sent via the MOUNT protocol (an RPC service).
It checks the server’s export configuration (usually /etc/exports on Linux) to verify which clients are allowed to mount specific exports. Once access is verified, mountd provides the client with an NFS file handle for the requested directory. The client uses this handle for subsequent file operations.
It ensures the client is allowed to mount the export with the specified options (e.g., read-only, read-write, root squashing).
root_squash prevents root user connected remotely from having root privileges and assigns them the user ID for the user nfsnobody. This effectively “squashes” the power of the remote root user to the lowest local user, preventing unauthorized alteration of files on the remote server.
The NFS mount daemon can also inform clients of available exports (showmount -e server). More about the showmount command further down.
Determine if mountd is running.
# systemctl status nfs-mountd.service
# ps aux | grep mountd
By querying the rpc portmapper on the NFS server we can see that the mountd service is listening on tcp/udp port 20048.
SLES 12 and higher will use here the static port 20048 (specified in /etc/services).
# rpcinfo -p
NFS-Server Service
# systemctl status nfs-server.service
2049 (TCP/UDP) fixed port for NFS operations.
NFSversion 4 does not use UDP as mentioned. If you mount a file system with the proto=udp option, then NFS version 3 is used instead of version 4.
Capture and analyze NFS Traffic
To see what happens on the network layer when a client is mounting an NFS export (share) and accessing it, I will capture the traffic on a Linux (SLES15 SP5) NFS server as shown below.
More about capturing and analyzing network traffic on Linux by using TShark, the command-line version of Wireshark, you will find in my following post.
Here I will start to capture traffic on the NFS server by setting a capture filter (-f flag) to only capture all traffic from the NFS client with the IP address 10.0.0.130.
# tshark -i eth0 -f "host 10.0.0.130" -w /opt/capture01.pcapng -i <interface>, --interface <interface> name or idx of interface (def: first non-loopback) -f <capture filter> packet filter in libpcap filter syntax -w <outfile|-> write packets to a pcapng-format file named "outfile"
Next I will mount an NFS export (share) on the NFS client (IP 10.0.0.130) we capture the inbound/outbound traffic on the NFS server (IP 10.0.0.151).
I will mount the export here by using explicitly the NFS protocol version 3 and the UDP protocol.
# mount -t nfs -o vers=3,proto=udp 10.0.0.151:/NFS-Exports/share01 /nfs-data/
Below I will analyze the captured file from the NFS server by using WireShark.
First the client (IP 10.0.0.130) will query the listening UDP port (UDP as set in the mount command above) of the NFS Server (IP 10.0.0.151) by using the portmapper daemon running and listening on tcp/udp port 111 (here by using the udp socket (protocol:port)) on the NFS server.
The portmapper daemon on the NFS Server responding with the default UDP Port 2049.
Next the client will query the listening UDP port for the mount daemon on the NFS Server.
The portmapper daemon on the NFS Server responding with the UDP Port 20048 on which the mount daemon is listening for incoming request.
On SLES12 and higher mountd will use the static port 20048 (specified in /etc/services).
Source: https://www.suse.com/support/kb/doc/?id=000016649
NetApp ONTAP9 is using here btw. tcp/udp port 635 for mountd.
Modify ports used for NFSv3 services
https://docs.netapp.com/us-en/ontap/nfs-admin/modify-ports-used-nfsv3-services-task.html
When we run the rpcinfo -p command on the NFS server, it will confirm that the mount daemon for NFSv1, NFSv2 and NFSv3 is listening on UDP Port 20048, also on TCP Port 20048 but previously we forced the client to use UDP by setting the vers=3,proto=udp flags.
# rpcinfo -p
From now on when the client accesses the mounted NFS exports (shares), only UPD port 2049 is used on the NFS server. But still by using Remote Procedure Calls (RPC) for local procedure calls (open, read, write, close).
Below you will see the equal command in NFSv4 to mount the NFS export with the v4 Prodecure: COMPOUND (1).
There’s no separate MOUNT call like in previous version, you just ask for file handles and attributes directly using the NFSv4 COMPOUND procedure.
Mounted NFS export (share) on the client.
So in case you will mount an NFS export by using NFSv3 or NFSv2 which is using the portmapper and mountd service, you need to allow more traffic besides TCP port 2049 like just used in NFSv4.
In case the rpc portmapper listening on tcp/udp port 111 cannot be reached by the NFS client you want to mount the export, you will run into the follwing errors, shown when using the -v –verbose flag.
If supported on the client and NFS server, the mount command without setting the protocol version explicitly, will mount the NFS export (share) by using NFSv4 and therefore on TCP port 2049, which we allowed on the server.
So when you want to provide also the NFSv3 protocol or lower on the server and have restricted access to the NFS server by using a firewall, we also need to allow further traffic.
NFS services (v3 and v2) generally require 5 services be running and accessible through a firewall:
- portmapper -> Portmapper runs and listens on both NFS clients and on NFS servers. It already runs on a static port, 111 for both TCP and UDP. Firewalls should be configured to allow incoming packets to this port on both TCP and UDP. This firewall exception is needed for incoming packets to both NFS Server systems and NFS client systems.
- nfsd -> The NFS Daemon runs only on NFS Servers (not on clients). It already runs on a static port, 2049 for both TCP and UDP. Firewalls should be configured to allow incoming packets to this port on both TCP and UDP. This firewall exception is only needed for packets incoming to a NFS Server.
- mountd -> Mount daemon runs only on NFS Servers.
- lockd -> Lockd run both on NFS Client and NFS Servers. Static ports can be set and then firewall exceptions will be needed for incoming TCP and UDP packets to those ports on both NFS Clients and NFS Servers.
- statd -> Statd run both on NFS Client and NFS Servers. Static ports can be set and then firewall exceptions will be needed for incoming TCP and UDP packets to those ports on both NFS Clients and NFS Servers.
We will see these 5 services and its listening ports also when running the rpcinfo -p command.
The rpcinfo -p command is used to query the RPC portmapper (or rpcbind) service and list all registered RPC services (Remote Procedure Call services) along with their program numbers, versions, protocols, and port numbers.
nlockmgr -> A user-facing RPC service entry point that allows RPC clients (like NFS clients) to interact with lockd in the kernel. Think of it as a frontend RPC interface exposed by rpcbind and listed via rpcinfo -p. It forwards RPC calls to the lockd kernel module to process file locking requests. lockd is a kernel-level daemon that implements file locking for NFSv2 and NFSv3.
status -> This is the NFS status monitor service, known as statd (for the service name in systemd on SLES). The RPC service name shown as status is what clients use to contact the service via RPC.
statd is the NFS status monitor daemon that manages lock state recovery in case of failures (for example, when a server crashes). The Linux kernel has internal hooks for this (nfsd relies on statd to coordinate locks).
The mountd daemon will use the static port 20048 (specified in /etc/services) on SLES 12 and higher.
On SLES11 and older mountd does not use the value in /etc/services. Therefore, to set a static port on those older distributions, edit /etc/sysconfig/nfs and find and set MOUNTD_PORT=”<port number>“.
Lockd and Statd run both on NFS Client and NFS Servers. Static ports can be set and then firewall exceptions will be needed for incoming TCP and UDP packets to those ports on both NFS Clients and NFS Servers.
For SLES 11 SP2 and higher, including SLES 12 and 15 we can configure them in /etc/sysconfig/nfs, find and set the parameters as desired:
- STATD_PORT=
- LOCKD_TCPPORT=
- LOCKD_UDPPORT=
NOTE: Initially, SLES 12 did not supply or use these parameters in /etc/sysconfig/nfs. However, the code to make use of them has been added in maintenance updates. Update the packages “nfs-client” and “nfs-kernel-server” to version 1.3.0-9.1 (released in October 2015).
More you will find here for SUSE Linux Enterprise and its static ports for NFS https://www.suse.com/support/kb/doc/?id=000016649.
For NetApp here https://docs.netapp.com/us-en/ontap/nfs-admin/modify-ports-used-nfsv3-services-task.html or https://kb.netapp.com/on-prem/ontap/da/NAS/NAS-KBs/Which_Network_File_System_NFS_TCP_and_NFS_UDP_ports_are_used_on_the_storage_system.
Determine NFS Exports (Shares) remote from Clients
To see a list of exports and its policies (aka export rules) from a specific NFS server, we can use the showmount command below.
Export policies contain one or more export rules that process each client access request. The result of the process determines whether the client is denied or granted access and what level of access.
You run the showmount -e command on the NFS client.
# showmount -e
-e stands for exports. It asks the NFS server’s mountd daemon for a list of currently available shared directories (exports).
showmount is a client-side tool querying an NFSv2 or NFSv3 server. It won’t work for NFSv4 servers, as NFSv4 doesn’t use mountd for exporting filesystems.
Both of my NFS server below will still support and have enabled the NFSv2 and NFSv3 protocol besides NFSv4 and therefore the showmount -e will work.
Below I will first query my Windows NFS server for all exports and then my Linux (SLES15) NFS server.
# showmount -e <NFS Server>
The NFS server responds with the list of exports (defined in /etc/exports or equivalent).
As my client here (SLES15-SP-Testing01) has the IP address 10.0.0.130 assigned to, I can mount the /NFS-Exports/share01 export here because its IP address is defined in the export policy.
So when trying to mount the /NFS-Exports/share02 export which have not assigned the clients IP address in its export rules to access, I will run into an access denied by server as shown below.
When we capture the traffic on the NFS server when a client is executing the showmount -e <NFS Server FQDN or IP> command, we can see the following traffic by the way.
First we need to start the capturing on the NFS server.
Now we can query the NFS server for the exports list by using the showmount -e <NFS Server FQDN or IP> command.
After first query the portmapper on tcp port 111 (frame 1), the mountd service on the NFS server is contacted to request the export list. (below frame 14 highlighted below).
As mentioned above, the showmount -e command just works when NFSv2 or NFSv3 is enabled on the NFS server and therefore the mountd service is running and listening for incoming requests, NFSv4 does not use mountd.
EXPORT is the 5th procedure in the MOUNT protocol (as defined by RPC standards).
Procedure 5: EXPORT returns the list of exported directories and their access restrictions (e.g., which clients can mount them).
The mountd service finally responses with the exports in frame 16 below.
More about NFS you will find in my blog by using the following link https://blog.matrixpost.net/?category=nfs.
Links
Setting static ports for NFS on Linux
https://www.suse.com/support/kb/doc/?id=000016649NetApp ONTAP9 – Modify ports used for NFSv3 services
https://docs.netapp.com/us-en/ontap/nfs-admin/modify-ports-used-nfsv3-services-task.htmlCommon NFS Mount Options
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/storage_administration_guide/s1-nfs-client-config-options
