Linux diskless client nfs throughput

Question

I have a gentoo diskless client that mounts using nfs and it seems like the throughput of nfs is slow. iperf show the network able to push 770 Mb/s and hdparm shows the sata disk on the server at 90 Mb/s, but doing a dd across the network only shows up as 10 Mb/s (is about 40 Mb/s when the same test is done on the server to a third machine).

Odd thing is that the pxeboot mounts nfsver2. Not sure if this is a problem or not, but other mounts after boot mount as nfsver3. Really seems like it is a config problem with nfs somewhere, but I have not found anything that makes a difference. Doing rsize=8k,wsize=8k, noatime, async. Anyone have any tips to look for or is this 10 Mb/s really the throughput I should expect for a diskless client?

Answer 1

I assume this is gigabit network we're talking about; 100Mbit networks have a maximum of about 10 megabytes per second.

NFS is not very efficient for some things, but bulk transfers should still be able to be done at the speed of the wire. How fast can you transfer files, over the same network, via for example ftp or http?

Maybe the switch is slowing things down, have you tried connecting the boxes directly to one another using a cable?

Answer 2

Interesting what the fix was. When you mount a pxelinux nfs root, you have to set the nfs parameters in the pxelinux.cfg file (not in fstab). In setting the nfs root partition to nfsver3, tcp I got good performance gains. However, what really made the difference was removing the rsize and wsize parameters in both the pxelinux config file and fstab and letting nfs figure out what the block size should be. This let the nfs read/write perform to what I expected it to be. Here is my config and some metrics if it helps anyone else.

one ~ # nfsstat -m
/ from /dev/root
 Flags: rw,vers=3,rsize=4096,wsize=4096,namlen=255,hard,nointr,nolock,proto=tcp,
timeo=600,retrans=2,sec=sys,addr=192.168.0.5

/store/tv from 192.168.99.99:/store/tv
 Flags: rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,nointr,noloc
k,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.99.99,mountvers=3,mountp
roto=tcp,addr=192.168.99.99

one ~ # time dd if=/dev/null of=/store/tv/test bs=16k count=10000
10000+0 records in
10000+0 records out
163840000 bytes (164 MB) copied, 2.79913 s, 58.5 MB/s

real    0m3.048s
user    0m0.033s
sys     0m1.983s

one pxelinux.cfg # time dd if=/store/tv/test of=/dev/null bs=16k
154614+1 records in
154614+1 records out
2533208064 bytes (2.5 GB) copied, 56.4591 s, 44.9 MB/s

real    0m56.509s
user    0m0.163s
sys     0m26.802s