Kodi hanging at least once per day since August update

My Vero 4K has been generally very stable for a long time.

Since the August update, I started experiencing frequent hanging of the Kodi interface.

Initially I thought it could be my NTFS-formatted external HD for the media sources. I took that to a Windows machine and there were a couple of minor errors (file descriptor). Put it back and no change.

Next thought: MariaDB. Well, every hang and crash forced a recovery, so I completely rebuilt the database. No change. I moved the database to another system. No change.

When Kodi hangs, I can still ssh to the Vero 4K, and try to stop processes, or force an ugly reboot (reboot -f -h).

This time I thought I should collect logs, so I turned on logging. I could not get any response from grab-logs, and uptime showed load average: 50.04, 50.04, 50.05. So forced a reboot.

I ran grab-logs -A after the reboot, URL is: https://paste.osmc.tv/otigusidux

I would appreciate any guidance.

Thanks!

There is not too much information left in these logs which gives a hint what has happened.
But I see syncthing running as well on this box … could you at least for tests deactivate that sw stack and see whether anything changes?

The only further idea I have is to activate persistent kernel logs in hope to catch any hint pointing to the root cause by that. It is somewhat complex to catch the kernel logs after a crash once it happened, so let us know if you’re interested for instructions.

Sounds like a good idea.

I’ve disabled and uninstalled syncthing - it is useful and has been reliable for the past year, but I’m happy to eliminate it as a potential source of instability. It can go back when everything is stable.

I’m willing to enable persistent kernel logs, if you can share the instructions.

Thank you!

Unfortunately we need kernel messages from previous boots which are disabled by default with OSMC. To activate and provide such information, please, follow the steps below:

  1. login via SSH to the OSMC device, user osmc, password osmc
  2. cd /var/log
  3. sudo mkdir journal
  4. (from now, kernel messages are written to new directories for every boot)
  5. sudo shutdown -r now
  6. now wait for the issue/event which is the problem of this topic
  7. once it happens again and you are forced to reboot the OSMC device or it rebooted automatically, you’ve to identify the right kernel message log:
    7.a) login via SSH and invoke
    sudo journalctl --list-boots --no-pager
    7.b) the lines start with an index id like 0, -1, -2, etc. and contain the date and time when log was started
  8. upload the appropriate kernel log using
    sudo journalctl -k -b <identified index> --no-pager|paste-log
    (replace <identified index> with the real index id, see above)
  9. also, upload the appropriate full log using
    sudo journalctl -b <identified index> --no-pager|paste-log
    (replace <identified index> with the real index id, see above)
  10. provide the returned URLs here
  11. don’t forget to remove the created journal directory otherwise your system’s root file system gets filled
    11.a) login via SSH
    11.b )cd /var/log
    11.c) sudo rm -R -f journal && sudo reboot (repeat this line if you get a ‘cannot remove’ error until it works and your ssh connection gets lost by the reboot)

Thx for your time. We hope to help you, soon.

Alright, all is prepped. It has already failed once today a few hours before making the change, so now it just a game of wait and see.

Thanks again!

ok, please, activate debug logging within the mediacenter as well and do a grab-logs -A together with the kernel log

Done.

Interestingly, aml-vnc crashed when I tried to connect to do this. Restarting the service worked and I was able to navigate the gui (I use the mobile app for control, and have tightvnc in view-only mode)

And back to waiting for a crash :slight_smile:

You have aml-vnc running as a service? That will put quite some stress on the Vero.

I haven’t noticed any appreciable load on the system due to aml-vnc. It seems to have a low memory requiremnt (sits about 5 M when idle), and CPU used in 24 hours is about 6 minutes.

However, point taken. I will remove the service and run it from an ssh session when required.

Well, system has been stable for over a day.

Only changes made were removing syncthing and mariadb.

I guess one or the combination of both of them just didn’t get along with the August update.

Happy to call this issue resolved.

Thanks for the assistance!

Well it might be that the service works different to the console one. Maybe it is actually better that way. I just know that when I have it running I have quite some CPU usage.

When I’m connected, it does impact the system. That is reduced a bit by using is as a display only (not allowing the client to send any inputs), and using the iphone app for remote control

Happy to hear this. Don’t forget to execute step 11. from the instructions above otherwise your file system gets filled over time.

Done.

Thanks for the reminder!

As a follow-up, I tried reinstalling syncthing, and found the hanging came back. I did notice masses of vsftp processes at the same time. I stopped the device which was connecting via ftp, and the problem went away. Stable now for several days.

So maybe not syncthing, or vsftp, but a problem when both very active at the same time.

Drive they both access (as well as Kodi) is ntfs so may try switching to ext4.