Slow NFS: "Kernelmismatch" between Vero 5 and NFS-server

btak13 · 2 September 2026 05:54

Hi!

This is just a heads up since I solved my problem through downgrading my kernel on the NFS server from 7.2.2 to 7.1.11. Hopefully somebody with the same issues finds this post.

I have mounted my music and video directories via NFS on my Vero 5 through /etc/fstab entries:
192.168.X.Y:/Server/Videos /ServernameVideos nfs noauto,x-systemd.automount 0 0
192.168.X.Y:/Server/Photos /Servername/Photos nfs noauto,x-systemd.automount 0 0
192.168.X.Y:/Server/Musik/FLAC /Servername/Musik/FLAC nfs noauto,x-systemd.automount 0 0
192.168.X.Y:/Server/Musik/MP3 /Servername/Musik/MP3 nfs noauto,x-systemd.automount 0 0
192.168.X.Y:/Server/Musik/Neu /Servername/Musik/Neu nfs noauto,x-systemd.automount 0 0

Yesterday my NFS server running on Arch Linux did an update to his kernel from 7.1.11 to 7.2. Then the weirdness begann… The Vero 5 simply stalled on video playback. Cause was a transferrate of about 30 kb/s for this /Servername/Vidoes NFS mount tested with dd. Strangely all other NFS mounts showed the normal 90 to 220 MB/s in my network, only the video share was slow! Music and pictures playback was still working on the Vero.

On another Linux crate (Gentoo Linux) with also kernel 7.2.2 like the server the Video NFS mount was also good with 90 MB/s (around 100 MB/s is the peak of my HDDs), so NFS on the server side seemed fine. After unsuccessfully fiddling with the mount parameters I decided to go back to kernel version 7.1.11-arch1-1 on the NFS server. And instantly the Vero 5 was back to normal NFS rates like before!

I suspect a strange incompatibility between the Vero 5 using kernel 4.9.269-93-osmc and Arch 7.2.2 in the NFS stack. Didn’t dig deeper into this, but nfstat, nfsiostat nor logs on both computers showed any hint on the culprit.

Ciao,

Martin

sam_nazarko · 2 September 2026 06:05

Hi

I know of some file attribute changes in 7.2, but not much in detail.

What does exportfs show on 7.1 vs 7.2?
There may be some clues there.

Sam

btak13 · 2 September 2026 07:05

exportfs -av output is identical on both kernel variants. /etc/exports on the server is still unaltered.

What puzzled me and can be a hint:
Doing an ls -al on the Vero immediately after accessing a file with dd showed questionmarks for all file attributes except filesize. Repeating the command after 2-3 seconds showed the expected values.

Two other thoughts:
The Vero 5 uses IPv4 only but the network is dual stack. I restricted the Gentoo computer to IPv4 for the tests to have equal circumstances.

The shares on the NFS server are on two HDDs. Video is on one HDD /ServernameVideos, pictures and music on the other. Both HDDs are the same brand and type.

Martin

sam_nazarko · 3 September 2026 05:10

Thanks.

The question marks in ls -al look like the useful clue here. That’s stat() not completing, so I’d suspect attributes are being invalidated and each revalidation is taking seconds, rather than this being a throughput problem.

If you’re willing to put 7.2 back on briefly, the first thing to try is dd with iflag=direct on the Vero. That bypasses the client page cache. Fast with it and slow without would tell us this is cache invalidation.

Two other things worth posting: the fs type and mount options for both disks from /proc/mounts on the server, and findmnt on the Vero for the slow mount and a working one. Your fstab doesn’t pin vers or rsize/wsize, so they may not have negotiated the same.

My hunch is the NFSv4 change attribute. nfsd only backs that with i_version where the filesystem supports it, and 4.9 predates change_attr_type, so any movement in the value reads as the file having changed. That would fit one disk being affected and not the other.

Also, 220 MB/s is faster than your disks, so some of those figures came out of page cache. Worth using iflag=direct, or sync; echo 3 > /proc/sys/vm/drop_caches, when comparing.

Sam

btak13 · 3 September 2026 17:33

Well, then try again!
Sorry for the german errors. I translate the main cause in the text.

Server kernel is now 7.2.2-arch1-1 #1 SMP PREEMPT_DYNAMIC

Ups, there is a difference between the disks on the server: ext4 and xfs
/dev/sdb1 /Server ext4 rw,noatime,nodiratime 0 0
/dev/sdc1 /Server/Videos xfs rw,noatime,nodiratime,inode64,logbufs=8,logbsize=32k,noquota 0 0
The video disk was added a month later, so I used my standard xfs filesystem. But I am puzzled why sdb is ext4 since xfs is my favorite since decades (SGI Irix carry over)!

Mounts on Vero5, Videos fails, Photos works:
192.168.x.y:/Server/Videos on /Servername/Videos type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.x.z,local_lock=none,addr=192.168.x.y)
192.168.x.y:/Server/Photos on /Servername/Photos type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.x.z,local_lock=none,addr=192.168.x.y)

Recreating the error:
dd if=‘Official World Record! Fantastic Classical Music Medley played by a Train-aBNHmUT3GPg.mkv’ of=/dev/null status=progress interrupted with ctrl-c
205312 Bytes (205 kB, 200 KiB) kopiert, 6 s, 33,6 kB/s^C
409+0 Datensätze ein
408+0 Datensätze aus
208896 Bytes (209 kB, 204 KiB) kopiert, 6,21944 s, 33,6 kB/s

nfsiostats during a slow copy:
192.168.x.y:/Server/Videos mounted on /Servername/Videos:

       ops/s       rpc bklog
      15.842           0.000

read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
3.050 13.091 4.293 0 (0.0%) 0.838 0.964
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%) 0.000 0.000

Immediate ls -al fails with I/O error
ls: Zugriff auf ‘Official World Record! Fantastic Classical Music Medley played by a Train-aBNHmUT3GPg.mkv’ nicht möglich: Eingabe-/Ausgabefehler
-??? ? ? ? ? ? ‘Official World Record! Fantastic Classical Music Medley played by a Train-aBNHmUT3GPg.mkv’

dd with iflag fails with I/O error:
dd if=‘Official World Record! Fantastic Classical Music Medley played by a Train-aBNHmUT3GPg.mkv’ of=/dev/null iflag=direct status=progress
dd: Fehler beim Lesen von ‘Official World Record! Fantastic Classical Music Medley played by a Train-aBNHmUT3GPg.mkv’: Eingabe-/Ausgabefehler
0+0 Datensätze ein
0+0 Datensätze aus
0 Bytes kopiert, 0,00510472 s, 0,0 kB/s

Doing a “sync ; echo 3 > /proc/sys/vm/drop_caches” and
dd if=‘Official World Record! Fantastic Classical Music Medley played by a Train-aBNHmUT3GPg.mkv’ of=/dev/null status=progress
fails with I/O error.

If I copy the file to /Servername/Photos on the ext4 formatted disk copy will do 100-117 MB/s disc limited and a second time with 117 MB/s network limited (Vero 5 1Gb/s). Sidenote: With kernel 7.1.11 I got only 90 MB/s from the Videos disk, xfs seems to brake a little.
The NFS-server has 12 GB RAM and system uses about 2 GB (clamav is a memory hog!). So up to 10 GB files are served out of the memory cache after the first call. Backbone of my network is 10 Gb, the NFS-Server is the only node with a 10 Gb ethernet card. 220 MB/s is my main computer with a 2.5 Gb ethernet card.

So maybe xfs in combination with kernel 7.2 is the culprit? I quit for now and will think about it further. Maybe it’s a kernel 7.2 bug…

JimKnopf · 4 September 2026 09:53

I carefully read your whole story and that is also my first idea.

JimKnopf · 4 September 2026 11:04

Only 4k block/buffer size while reading?

btak13 · 8 September 2026 06:49

Thanks for the hint.

With kernel 7.1.11 on the server and the same operation nfsiostats reads:

192.168.116.3:/Server/Videos mounted on /Jupiter/Videos:

       ops/s       rpc bklog
      24.282           0.000

read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
23.622 24160.257 1022.795 0 (0.0%) 184.224 184.467
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%) 0.000 0.000

Couldn’t find a 4k cache in the kernel config at first glance. Also my main computer with kernel 7.2.3 still has no problems with the same nfs-share.

I changed the filesystem from xfs to ext4 for the share and the Vero5 is happily working with the NFS-server at kernel 7.2.3 now.

My conclusion:
Somewhere in the combination Vero5 with kernel 4.9, NFS-server kernel 7.2.2/3 and the filesystem xfs for the shared partition is the culprit. But where I have still no clue. Obviously ext4 is a safer choice for NFS-shares.

Let’s book this behaviour as general weirdness.

Bye,

Martin