I’m running the current release of OSMC on a Raspberry Pi 3B. I’ve had some trouble recently with the RPi dropping wireless connectivity, but this seems to have resolved itself. Because of the trouble I’ve had, however, I routinely ping the RPi from my laptop or desktop Windows computer and, from time to time, SSH into the Pi to run iperf. For the past month or so, I’ve had no trouble doing so.
Two days ago, when I attempted to use OSMC on the Pi, it was “hung” on one of the screens and would not respond to keyboard input (I use Flirc with a remote control. This has worked well for at least two years.) There was a message on that screen indicating that something had been happening with NextPVR. Regrettably, I did not write that message down.
I pinged the Pi and and was able to reach it and got the normal response. I then tried to SSH into it, planning to restart the media center. I’ve always used PuTTY to SSH, and it’s always worked well. On this occasion, though, I got no response from the RPi. Normally, I get a request for username and password, but this time, I got nothing. I tried running PuTTY on two different computers with the same non-result.
After failing to SSH into the RPi, I unplugged the power and plugged it back in. The Pi rebooted, OSMC restarted, and all was well. I was able to ping the Pi and SSH into it successfully. I didn’t think to take a log at that time, but I did so several hours later. It’s here: http://paste.osmc.tv/mijosozeqe I hope it’s helpful.
I’m very curious why I was able to ping the Pi successfully but was unable to SSH into it.
Because the system logs are not persistent any information that may have been present about the situation was lost with the reboot and the current logs are not going to show anything as the condition is no longer present. Pinging and ssh are unrelated services so one of them working does not really say much about the other. Pinging is a common first step to confirm you can see a particular device, but that does not extend off to knowing a particular services is running and having an open port to communicate through.
If it happens again you could connect a keyboard and access the terminal locally to generate the logs. As of this moment though the logs are gone and you say ssh is working. It is exceedingly difficult to fix something that is not currently broken.
As noted earlier, the Pi had become unresponsive to keyboard inputs (using Flirc and a remote control). Are you suggesting that plugging in a physical keyboard is going to work where Flirc did not? I thought Flirc was actually generating keyboard inputs.
I understand that we can’t fix what’s not broken. I’m just trying to understand how to collect diagnostic information in case this happens again.
Good point. If the flirc, which is presented to the OS as similar to a keyboard, stops having effect then there is probably nothing that can be done. If this is just a one off thing then I would just chalk it up to a fluke and the system probably just crashed. If it becomes a regular occurrence then it would be prudent to start investigating. Is this frequent regular behavior? RBPi’s even in perfect condition are not immune from locking up on occasion.
This is the first time I’ve noticed this behavior in the 4-5 years I’ve been running OSMC on the Pi. However, in the past, when OSMC locked up, I always unplugged the power cable and plugged it back in again to both reboot the Pi and restart OSMC; i.e., I’ve never tested the ability to SSH into the Pi when OSMC became unresponsive.
I read somewhere that the unplugging/plugging in was not the recommended approach to restarting OSMC if the Pi was still running. In this case, given that a ping was successful, I concluded that the Pi was running and hence the right approach to restarting OSMC was to SSH in and issue the appropriate command.
Pulling the plug is best to avoid, and doing it excessively is just asking for problems. That being said, when something to do with the OS locks up your not often left with any other option. If just Kodi locks up then you can normally ssh in and restart the mediacenter service or just issue a restart command from the terminal. But just pulling the plug every once and a while is unlikely to cause any harm.
Well, this “hanging” problem is happening about once per week. I’m left with no option other than pulling the plug.
Is there any way to cause OSMC to log and save what’s happening when I’m not using it so that if it hangs, I can get diagnostic information and maybe post it here? I have WinSCP installed on one of my Windows machines and should have visibility to the pi’s file system when this happens if it would be helpful.
You can enable debug logging with the overlay suppressed by adding it via an advancedsettings.xml file. If you have another PSU laying around you might want to give that a try first though.
The power supply is a educated guess based on a random lockup when not being used.
When you turn on debug logging via the Kodi GUI settings it puts an overlay on the screen that shows a bunch of text. This is rather annoying when watching video. As such their is a way to turn on debug logging without it displaying on the screen. More information can be found on Kodi’s wiki…
My “hanging” problem happens when OSMC is idle, and it’s intermittent. I’ve never had it hang up while I’m using it; e.g., watching a recorded program. As such, the overlay isn’t a big deal because I can always turn logging off while I’m watching.
If I have logging turned on while I’m not watching, are the logs saved somewhere so I can access them after “pulling the plug”, or do they get deleted on a reboot? I only discover there has been a problem when I try to use OSMC by switching my TV to the appropriate HDMI input. When I do that and it’s hung, my only option is to pull the plug.
When you pull the plug the system logs are lost but the Kodi log is written to disk and upon the next reboot it gets renamed to ~/.kodi/temp/kodi.old.log before the new Kodi log gets created allowing you to see what was happening in Kodi during the previous session.
If you can ping the Pi, it means that the kernel is still running. Not being able to use OSMC or SSH could be down to some resource issue occurring, such as something chewing up all the available CPU and/or memory.
By default, OSMC keeps the system journal in memory, mainly in order to conserve the life of SD card. As mentioned, this means that it is lost on reboot. However, it is possible enable a permanent system journal so that it will survive a reboot, with this command:
sudo mkdir /var/log/journal
then reboot. Remember to remove the directory once it’s no longer needed.
With regards to pulling the plug on your Pi, there is a slim possibility that you can reboot it by sending a message via SSH. At it’s most basic, you can send a “reboot immediately” message, which will look like:
ssh osmc@<IP-address> 'echo b | sudo tee /proc/sysrq-trigger'
but there is a range of commands you can (try to) send. For more information, take a look at Magic SysRq key - Wikipedia , and check out the “Uses” section.
I suspect this is right. It remains to figure out what’s chewing up CPU/memory.
For what it’s worth, these hanging problems started after I started using comskip with NextPVR. I suspect that’s a red herring, but I thought I’d mention it for completeness.
I’m going to more closely track the frequency of occurrence of this problem. The next time it happens, I’ll go searching for the “old” Kodi log and see if there is anything useful there. Failing that, I’ll enable the permanent system journal and see if that helps. Where do I find that journal? Is it a file in the /var/log/journal directory? I suspect this file gets pretty big pretty quickly.
This sounds like a good option to try instead of pulling the plug. Can I issue this command from a Windows “run” window? Note that I can’t log in via SSH when this problem occurs.
I will check out the SysRq link. Maybe I can find a keystroke(s) to set in my FLIRC to reboot?
Already noted. That’s why I said there was a “slim possibility” that it would work. If your device is being overloaded, a relatively small command might have a better chance of getting some CPU time.
Thanks again. I’m basically a Linux illiterate, so apologies for my elementary questions.
I found an old log for the day my last “hanging” incident occurred. I’ll post a reply here containing an excerpt from that log in case it would help diagnose what’s going on. It suggests a python problem
