I have a vero4k+ that randomly freezes (during idle, but also during playback), with errors like this:
Mar 2 00:17:27 vero4k kernel: [ 3525.547042@0] skbuff: skb_over_panic: text:ffffffc001527288 len:2690 put:2690 head:ffffffc059dd5000 data:ffffffc059dd5042 tail:0xac4 end:0x680 dev:eth0
Mar 2 00:17:27 vero4k kernel: [ 3525.555113@0] BUG: failure at net/core/skbuff.c:100/skb_panic()!
Mar 2 00:17:27 vero4k kernel: [ 3525.555126@0] Kernel panic - not syncing: BUG!
and again an hour later:
Mar 2 01:17:27 vero4k kernel: [ 3568.295441@0] skbuff: skb_over_panic: text:ffffffc001527288 len:2690 put:2690 head:ffffffc00031e000 data:ffffffc00031e042 tail:0xac4 end:0x680 dev:eth0
Mar 2 01:17:27 vero4k kernel: [ 3568.303656@0] BUG: failure at net/core/skbuff.c:100/skb_panic()!
Mar 2 01:17:27 vero4k kernel: [ 3568.303697@0] Kernel panic - not syncing: BUG!
This happens quite frequently. A couple of times during playback of a movie. It also appears to be daily the case when it sits idle (I always find it frozen).
To find the error messages, today I installed rsyslog and enabled forwarding of all logs to a remote system. Before that I was not able to see what is wrong.
grab-logs -A -P | sed \
-e "s| dir '.*' due | dir 'MEDIAHIDDEN' due |g" -e "s| dir '.*' as | dir 'MEDIAHIDDEN' as |g" \
-e "s| directory '.*' does | directory 'MEDIAHIDDEN' does |g" \
-e "s| was found in dir .*$| was found in dir MEDIAHIDDEN|g" \
-e "s| for item '.*', it | for item 'MEDIAHIDDEN', it |g" \
-e "s| title search for '.*'$| title search for 'MEDIAHIDDEN'|g" \
-e "s| GetMovieId \(.*\), | GetMovieId \(MEDIAHIDDEN\), |g" \
-e "s| Searching for '.*' using | Searching for 'MEDIAHIDDEN' using|g"
I need to specify vers=3 or it does not mount.
Then the noatime and nodiratime are just to avoid date updates on reads.
It did not change much though:
# mount | grep /data
systemd-1 on /data type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
10.11.12.1:/data on /data type nfs (rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.11.12.1,mountvers=3,mountport=45489,mountproto=udp,local_lock=none,addr=10.11.12.1)
I think the main change is that it switched from tcp to udp.
@sam_nazarko I don’t use WiFi. This device plays 4k videos, which I guess would not be ideal for Wifi.
Regarding flow control:
# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate: on
RX: off
TX: off
RX negotiated: on
TX negotiated: on
So, it is off. This vero4k+ is connected to a gigabit ethernet port of a DIR860L router. I don’t see any options to enable flow control on this router.
@dillthedog the NFS server is a gentoo box. Nothing fancy about it. I already have 5 RPis running with NFS-root from the same NFS server. 2 of them are running OSMC.
This vero4k+ is not using NFS-root (it is plain vanilla, as it was shipped).
I always had random freezes on this vero4k+. Since the day I received it. But now they way too frequent. I cannot finish a movie without a freeze. When I received it, it was freezing once per week, so I thought this would be “normal” for a new device that is not probably that stable yet.
Nothing changed in the network since I received it (August 2018).
Well, I think it is common for OSMC devices to freeze from time to time. I guess it is about kodi…
My non-kodi RPIs never freeze. I have 3 PRIs 1B (yes, the oldest ones) playing 1080p movies (with omxplayer) around the clock for several years now (I run 3 private TV channels at my home). They never crashed. Not even once! But the OSMC ones (which are a lot more powerful: RPIs 3B+)… freezes do happen from time to time…
The retry count looks suspicious there and you should be getting much higher speeds (in the order of 900Mbps). Is this a real Ethernet setup (i.e. no powerline adapters)?
This is interesting…
Do you use them daily? I think it is subject to the frequency of kodi use.
Yes, it is just a tiny CAT5E cable with length of 30cm to a DIR860L. Then this DIR860L is connected to a Linksys SRW2024P switch, on which the NFS server is connected.
I tried a PC which is connected directly to SRW2024P:
Yes, every day. I have high uptime (some units on for 30+ days)
Unfortunately Kodi doesn’t handle shares going away gracefully; so if the NFS connection drops, Kodi will freeze for a period of time.
So are you experiencing iperf issues with other devices on the network too?
There were some hardware issues with some initial Vero 4K + devices which caused poor performance in the TX direction only. Usually these users would not be able to get speeds above 2-3Mbps. The RX direction is never affected.
Well, I don’t think so… There are many devices running against this NFS server, and a few of them are streaming videos all the time. I don’t think the retries are the issue.
I did another test: I removed the vero4k+ from its position and attached it directly to the main switch with a new cable. Same thing. Poor speeds with a few retries. The poor speeds appear in both directions. The receiving direction does not have any retries.
What makes me think this is a h/w issue, is the kernel log. Either the kernel driver is problematic, or this vero4k+ has a broken Ethernet.
I’m almost certain that this is anything but a hardware issue.
Of the few devices that had Ethernet problems, none had a problem with RX direction. If the hardware is faulty, we’d expect to see extremely poor TX performance (in the order of < 100Mbps) but perfect RX performance (900Mbps+).
What’s the MTU size on your network?
Do you see the skbuff BUG_ON without NFS mounted?
Do you see the skbuff BUG_ON with default rsize,wsize?
The retries look like a duplex mismatch to me; but I am not a networking expert.
Please also update to the latest kernel, which as some Ethernet improvements I plan to release shortly.
Add the following line: deb http://apt.osmc.tv stretch-devel main
Run the following commands to update: sudo apt-get update && sudo apt-get dist-upgrade && reboot
Your system should have have received the update.
I also recommend you edit /etc/apt/sources.list again and remove the line that you added after updating. This will return you to the normal update channel.