RPi3 / November update: system hangs / lockup after a few hours (not even ssh works)

After many years of stability with OSMC, after the last update to kodi 18.5, my headless music-only osmc RPi3 started to hang after a few hours without use. OSMC becomes totally unresponsive (I use it via it’s web interface and via Yatse, the android app), and the system itself locks up, I can no longer SSH into the box. A power cycle is needed to restore the system. Kodi logs don’t show anything and the system journal does not use persistent storage, so no idea what’s going on.

I’ve never seen this behavior before. This is on an RPi3. The SD card has about 1.5 years of use and appears OK, there are no issues after a power cycle. The issue seems to happen after a few hours of idle time.

Anybody noticed similar behavior? any suggested workaround?

We haven’t had reports of stability issues on Pi with this update.

I’d check the usual suspects like PSU and SD.

You can enable a persistent journal but I’d be surprised if the situation was software related.

Sam

Thanks for the fast response. I’m cloning the sd card and will give changing it a shot.

Forgot to mention that the RPi3b+ has a hifiberry digi hat on it. I use a low-noise ifi PSU, 2.5A, for about 3 years now (used on different RPi builds over time). I did notice today when trying to troubleshoot and connecting a monitor that the
image
… icon was showing up while the system was booting up - I don’t recall seeing it before but sInce I never use this box with a monitor I can’t tell if that was ‘normal’ or not. The icon goes away after the system boots up.

Side question, any recommendation for modern low-noise PSUs for RPi audio builds? I could try with 3 or 4amps one. And be ready for RPi4.

No it is not normal. That symbol means that the voltage is dropping too low. It is not unusual for a power supply to go out of spec over time like this. You would only be seeing it during boot as that is probably the only time you are putting it under much load. If you started doing something else requiring a lot of CPU i’m sure you would see it pop up again. I suspect that if you replaced the power supply your stability problems would go away.

I followed @sam_nazarko 's comment and changed the SD card. I’m trying Sandisk endurance/c10/v30 this time and the overall responsiveness of the microserver improved dramatically. I guess 1.5 years for the previous card was just too much.

That said, I just noticed OSMC/kodi stopped responding again (both Web Interface and also via yatse/remote control) - after a few hours of having audio playback in pause; the difference now is that SSH continued to work. After logging in, I could see that the Kodi process was consuming 2GB of “VIRT” RAM, instead of the usual 500M (this system has no video, no addons). After restarting the mediacenter service, kodi came back to life. There seems to be something going on, maybe it’s a memory leak in Kodi (no success yet in reliably reproducing this)

Have you tried a different power supply? These raspberry pi’s can get quite unpredictable and do very strange things when they have less than optimal power delivery.

The Pi3 has only 1GB of RAM and some of that is reserved for graphics. Is 2GB a typo – or have you, for example, enabled swap space?

I had the same observation RE memory (as shown by top), turns out that VIRT memory can exceed physical memory even without swap (I did not enable swap), it counts memory-mapped files and other random bits). RSS memory (resident) is the metric that can’t add to more than the physical memory and unfortunately I did not note down how much of it Kodi was consuming.

RE power supply, I didn’t change it yet but I started to look into it. I am using a supposedly decent one (iFi Power, low noise, in use for the last 2 years it so)