[SOLVED] OSMC random freezes afer january 2018 update


#1

I’ve been getting some random freezes with osmc.
It all started after the 2018-01 update.
Before that, I assure the system was 100% stable - Without any issue for many months.
I have not changed anything in the system or installed anything new.

In this system I have installed from osmc interface:

  • transmission,
  • ftp server,
  • samba server.

Apart of that:

  • nginx for serving a https server in a directory on the external hdd (NTFS formatted)
  • aria2c accessible trough a web interface served by nginx in https.
  • transmission is configured to be access trough a https tunnel by nginx

All this has been working for months without an issue and I have not changed the configuration after the last 2018-01 update.

From several kodi.old.logs (the ones that are kept between reboots):
There is nothing unusual.
In fact, the more unusual is the case of today morning: there is now report in the log since 01:20 and the system crashed at 09:30
I was not using the system, and tried a simple samba connection.
Yesterday night, the system crashed when deleting some files in and external hdd via samba connection.
Also Yesterday night, it simply crashed when browsing the kodi interface.
Yesterday afternoon it suddenly crashed when connected trough ssh and editing a script in the home directory.
Many times I simply find the system hanged.
It is completely random and unexpected.

again looking at the kodi.old.logs, there is a common notice between them, and repeated several times:
“AddOnLog: PVR IPTV Simple Client: EPG file path is not configured. EPG not loaded.”
(but it always complained about that)
anyway, I’ve tried disabling it, but the result is the same.

Another common issue between kodi.old.logs:
“01:16:18.966 T:1926253056 ERROR: DBus: Error org.freedesktop.DBus.Error.ServiceUnknown - The name org.freedesktop.UPower was not provided by any .service files”
But again, this been there forever and it is reported only at kodi start up.

My last logs are here:
https://paste.osmc.tv/irazeyuten

I don’t want to complain, as I think osmc releases are awesome (I must buy a vero 4k :wink: ) - my Pi is my home server and media center :slight_smile:
But I start to suspect on kernel panics only happening from the last update.
is there any way of tracing this ??

Thank you


#2

It’s difficult to pinpoint anything but your system is highly configured, so, while there’s nothing wrong with doing so, you’re more likely to run into problems.

That said, I see that you have openswan installed, though this package is no longer available in Debian stretch. In fact, the packages.debian.org site indicates that openswan 1:2.6.37-3 was last available in Debian wheezy (version 7). OSMC is now on Debian stretch, version 9. The currently supported IPsec package on Debian stretch is strongswan.

Although it probably won’t cause a system crash, your UPS service, raspiupshat.service, is also failing and needs to be looked at.


#3

Thank you very much for your comments and time took looking into the logs.

1
The ups service is failing because I’ve removed the ups in order to test if it may be the cause of problem (low current).
The Raspberry pi is now directly connected to and official psu.

2
About open swan, I will investigate. But please note that before January update this was working fine.
Do you think the new kernel version may have an issue with this ?

(I will disable open swan in the meantime).


#4

The update moved OSMC from Debian jessie to Debian stretch and was a major change in many areas. So it’s very possibly related to the update.


#5

Is there anyway of making the logs persistent between reboots (not the kodi logs)?
This would improve the possibility of tracing the moment of a system freeze


#6

#7

I’ve updated the ups service to detect properly when the ups is not there.
Btw, if someone else has the geekworm ups hat, i’ve shared the service and scripts here (just copy all the files to a folder in the pi and “sudo .\install.sh”:
https://drive.google.com/open?id=1lDuTRcKRHd3O-S290wtPHh-jEM2yIDRp

  1. I’ve replaced openswan by strongswan (it took some hours to reconfigure, buy it is work fine allowing connection by l2tp psk.

In summary: the system seems pretty much stable again.
I will keep it on tests some more days before changing the thread title to “solved”.

thank’s for helping.


#8

Well … the freezes are back again.
This time they occured twice yesterday night:

  • Downloading a file with aria2 (download manager) into the external hdd;
  • Downloading the same file with wget to the same location into the external hdd;

I remember that 3 of the times I had freezes was:

  • Copying files between directories trough a samba access in the external hdd;
  • Deleting a directory with many files trough samba access in the external hdd;
  • Watching a movie that was in the external hdd;

Yes … I’ve seen that the 5 issues above have all in common the external hdd" !
The hard disk I’m using is a 2.5" Seagate with 2 TB formatted in NTFS connected to USB and powered by the Raspberry Pi3 itself, which is using an official PSU.

QUESTION: Is it possible that the system hangs completely due to hdd hardware failure or simple file system corruption ?

(Anyway, I’ve now activated persistent logs by creating the var/log/journal/ directory which links to an external USB pen drive - in order to spare the internal SD CARD from wearing and avoiding the usage of the external hdd which has been a common point to the issues)


#9

Yes, that’s a possibility.

A misbehaving disk / corrupted filesystem can cause panics and other unexpected behaviour.

Sam


#10

Meanwhile the system just crashed again !
This time I have persistent logs but … there is nothing new:

It was 09:44.
I had just logged in trough ssh and the terminal was idle.

At the same time I started to use kodi trough the osmc interface in the tv. Entered the programs sub menu and the system just hanged !
No pings, no ssh logins, no samba, no ftp, just freezed plain dead,

The last entries in the log just before the crash:

Mar 05 09:44:45 osmc sudo[17282]: pam_unix(sudo:session): session opened for user root by osmc(uid=0)
Mar 05 09:44:57 osmc sudo[17282]: pam_unix(sudo:session): session closed for user root

And before that just two logins logout I’ve made trough ssh. I was doing literally nothing with the system.

AS I’ve said it was 09:45. In kodi.lod.log (the one active at the time of the crash, there is noting logged since 04:21

02:51:49.807 T:1408234240  NOTICE: AddOnLog: PVR IPTV Simple Client: EPG file path is not configured. EPG not loaded.
04:18:53.991 T:1764745984  NOTICE: Previous line repeats 17 times.
04:18:53.991 T:1764745984   ERROR: OnTvStandby - Unexpected [standby_pc_on_tv_standby] setting value
04:21:50.643 T:1408234240  NOTICE: AddOnLog: PVR IPTV Simple Client: EPG file path is not configured. EPG not loaded. 

Ok thanks. I will connect the disk to windows and do a chkdsk and see if it solves the issue.


#11

Your caddy might report smart data, you could install smartmontools and check smartctl -a on the device.


#12

Thanks again for the suggestion. I’ve run smartctl but no errors are reported:

https://paste.osmc.tv/nadicavize

[EDIT]

I will now connect the disk to windows and do a scan / fix:

Errors reported by windows:

 Found corrupt basic file structure for "<0x3,0x677b>"
        ... repaired online.
    Found corrupt basic file structure for "<0x3,0x6784>"
        ... repaired online.
    Found corrupt basic file structure for "<0x3,0x678c>"
        ... repaired online.
    Found corrupt basic file structure for "<0x3,0x6790>"
        ... repaired online.
    Found corrupt basic file structure for "<0x3,0x679f>"
        ... repaired online.
    Found corrupt basic file structure for "<0x3,0x67b5>"
        ... repaired online.
Found 6 lost files (<0x3,0x677b>, <0x3,0x6784>, <0x3,0x678c>, ...); requesting reconnection to index "$I30" of directory "\??\R:\found.000"
        ... repaired online.

They are now fixed but does not seem to be enough for system hangs.

(I will keep he persistent logs active for some more days in an additional usb pen.)


#13

One other thing to consider is that the disk might be drawing too much power from the Pi for short periods. A powered USB hub should take care of that possibility.


#14

I strongly support the suggestion using a self-powered hub for this external hdd.
Although I use a 5.2 V 2.5 A power supply for my Pi3, the Rasberry did not run stable with devices connected which have had total sum of around 500 mA.
Btw. the package usbutils offers the tool lsusb which lets you read out the MaxPower of the connected usb device:
sudo apt-get install usbutils
lsusb -v


#15

Well … I start to believe that maybe the issue here.

I used a Raspi UPS HAT - nice addition to my rapberry pi. making possible to a clean shutdown at a power failure. But it has only 2.5A output, and NOT while charging the battery - if the battery is charging at the time of usage the power is not enough.
Since I’ve removed the ups hat, the freezing lower a lot.
In fact I’ve only had one in the past week. And it was when copying files inside the external hdd and trying to watch a movie at the same time.
I’m now going to test with a powered usb hub. if this solves the issue, I must start thinking in a 5V@5A PSU or even a 6A. :slight_smile:
(i don’t want to have 2 PSU connected to the Pi)


#16
  • Solved:
    Issue was related to insufficient power to the Pi.
    In my case, using the official raspberry pi PSU is enough for the Pi3 and a 2TB USB HDD - but now enough when conected trough the UPS HAT.

  • Solution:
    Use an additional PSU for a powered USB HUB