NFS regression in RC3

Help and Support Raspberry Pi
1

I’m running RC3 on Raspberry Pi2 with nfs mounted disks and the speed have dropped after latest upgrade. I’m getting less than 1Mbyte/S.

I get 94Mbit with iperf in both directions and if I do a dd with bs=16K I get good speed (6-7Mbyte/S). My fstab is as you suggest with x-systemd…

192.168.10.24:/mnt/stripe/serier /mnt/Serier nfs ro,x-systemd.automount,noauto,soft,udp,noatime,nfsvers=3 0 0

I have done a pcap and can see that when I copy a file with a normal cp command I get nfs packets out of order. I did a reboot between the 2 tests to clear all buffers. I have also soubles checked the ntp and I’m using the same and ntp is synched.

How can I upload the pcap files? I made a screenshot of the highlights and the 16K transfer is perfect with a READ call and a READ reply in perfect order

Copy

DD with 16K buffer

BR

Johan

2

I don’t see any out of order IP fragments in the logs you are showing - all of the offset fields of the fragments are monotonically increasing. Out of order reception of packets is not an issue anyway - TCP and UDP are specifically designed to be able to cope with this because it frequently happens across the internet. Fragments are reassembled into the correct order and only then is the reassembled packet acknowledged, once the window size is reached.

What I do see is a retransmission which will be caused by a lost fragment, which could be caused by many things, most notably packet loss. You don’t specify an rsize or wsize in your mount option and I’m not sure off hand what the defaults are for udp nfs 3, but it appears from the fragment offsets that it is 16K.

Have you tried dropping rsize and wsize to 8192 and/or trying tcp ? In some testing I did quite a while ago (on raspbmc on a Pi 1) I found that an rsize and rsize of 8192 gave the best performance without being overly sensitive to packet loss. (If you make them too big you have too many fragments and a much higher risk of a lost fragment causing the entire IP packet to be resent)

It’s possible that some of the kernel defaults for nfs have changed between RC2 and RC3 although I’m not aware of any changes off hand.

3

Hi DBMandrake

I did my test om an 3Gig mv file but for the simplicity I redid alla tests with a smaller (45K) textfile. The results are the same. Is there a way to upload pcap files?

I’ve tried tcp to no avail and also with both 8 och 32K mount options. nfs v3 is defaulting to 16K on udp and to my surprise to 132K on TCP. I’m connected with gig on all my devices but the rbpi2

UDP mount
192.168.10.24:/mnt/stripe/serier on /mnt/Serier type nfs (ro,noatime,vers=3,rsize=16384,wsize=16384,namlen=255,soft,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=192.168.10.24,mountvers=3,mountport=816,mountproto=udp,local_lock=none,addr=192.168.10.24)

TCP mount
192.168.10.24:/mnt/stripe/serier on /mnt/Serier type nfs (ro,noatime,vers=3,rsize=131072,wsize=131072,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.10.24,mountvers=3,mountport=816,mountproto=tcp,local_lock=none,addr=192.168.10.24)

I redid the tests and as soon as I do a dd with default bs (512B) transfer rates drops to <1Mbit

This is with 8K mount options and I didn’t let it run all the way. A few seconds shows where we are heading.

dd if=/mnt/Serier/CSI/Season\ 01/CSI.Crime.Scene.Investigation.S01E05.Friends.and.Lovers.mkv iflag=direct | pv | dd of=/dev/null

^C96MiB 0:00:06 [ 869KiB/s] [ <=> ]
5657088 bytes (5.7 MB) copied5657088 bytes (5.7 MB) copied, 6.51414 s, 868 kB/s
, 6.51365 s, 868 kB/s

dd if=/mnt/Serier/CSI/Season\ 01/CSI.Crime.Scene.Investigation.S01E05.Friends.and.Lovers.mkv iflag=direct bs=8K | pv | dd of=/dev/null

^C.5MiB 0:00:05 [5.33MiB/s] [ <=> ]
29622272 bytes (30 MB) copied, 5.13737 s, 5.8 MB/s

dd if=/mnt/Serier/CSI/Season\ 01/CSI.Crime.Scene.Investigation.S01E05.Friends.and.Lovers.mkv iflag=direct bs=16K | pv | dd of=/dev/null

^C.9MiB 0:00:09 [7.44MiB/s] [ <=> ]
75923456 bytes (76 MB) copied, 9.89484 s, 7.7 MB/s

4

Hi again,

Here’s what’s happening with a cp, he’s using 16K windows but does 2 READ Call with only 2 mS inbetween (packet 17 & 18) which gets retransmitted in packet 51.

5

Pcap files are generally huge, so we don’t have somewhere for you to upload them to.

A retransmission in 2ms does seem wrong though.

CC @sam_nazarko

Do you also see poor performance if you use Kodi’s built in NFS client instead of a kernel mount ? A bit harder to test of course but you could try finding the highest bitrate Jellyfish test clip you can stream over NFS without pauses using the Kodi NFS client, and then compare it to the same videos streaming via the kernel NFS mount. (Don’t compare it directly with a file copy test) Generally the kernel mount should perform 10-20% better than the built in NFS client.

6

I tried from a Intel Nuc with Kodi 14 (kodibuntu) and it working fine. He does indeed do 3 calls in fast succession but handles it correctly. I’m using freenas 9.3 on my nas server. I can sustain 8-9Mbyte/S regardless of mount size

7

OSMC ships its own libnfs and this has not changed since RC2. Do you see the same regression with the built in client?

S

8

Hi Sam,

I’m suspecting my NAS. There was an update on freenas 9.3 last week and there is a possibility that’s the real cause of these problems. I’m not home right now but will do a rollback and post my findings.

I have now tried on 4 different rbpi2, all from Alpha to RC1 and they are showing the same behaviour, dd with no bs set yields terrible throughput.

Thanks for your top notch support and software

9

All was related to my Freenas. After restarting the nfs service speeds are now normal.

It was really strange that speed from Intel boxes were alright.

Keep up the good work and I’m looking forward to the Intel version

1 Like