Can ping RPi, but can't SSH into it

Here is an excerpt from an “old” Kodi log file from the date my last hanging problem occurred. The log has more information in it in case that’s needed, but I didn’t want to post the whole thing in case it isn’t necessary

2021-01-28 03:28:03.968 T:1689243872   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                         - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                        Error Type: <type 'exceptions.OSError'>
                                        Error Contents: [Errno 12] Cannot allocate memory
                                        Traceback (most recent call last):
                                          File "/usr/share/kodi/addons/script.module.osmcsetting.updates/service.py", line 38, in <module>
                                            m = update_service.Main()
                                          File "/usr/share/kodi/addons/script.module.osmcsetting.updates/resources/lib/update_service.py", line 244, in __init__
                                            self._daemon()
                                          File "/usr/share/kodi/addons/script.module.osmcsetting.updates/resources/lib/update_service.py", line 270, in _daemon
                                            self.update_now()
                                          File "/usr/share/kodi/addons/script.module.osmcsetting.updates/resources/lib/update_service.py", line 898, in update_now
                                            if self.check_for_unsupported_version() == 'alpha': return
                                          File "/usr/share/kodi/addons/script.module.osmcsetting.updates/resources/lib/update_service.py", line 1500, in check_for_unsupported_version
                                            process = subprocess.call(['/usr/bin/dpkg-query', '-l', 'rbp-mediacenter-osmc'], stderr=fnull, stdout=fnull)
                                          File "/usr/lib/python2.7/subprocess.py", line 172, in call
                                            return Popen(*popenargs, **kwargs).wait()
                                          File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
                                            errread, errwrite)
                                          File "/usr/lib/python2.7/subprocess.py", line 938, in _execute_child
                                            self.pid = os.fork()
                                        OSError: [Errno 12] Cannot allocate memory
                                        -->End of Python script error report<--

As always, thanks for any help.

Well, that seems to be a problem. :slight_smile: Next will be to find why that’s happening.

Thanks! Can I post anything else that might be helpful to find out why?

Have you installed any “external”, ie non-standard, programs on the system? (I couldn’t see anything in the log.) If not, then there’s a better-than-evens chance that it’s a Kodi-related problem. Often, that means an add-on.

To my knowledge, I haven’t installed anything non-standard. I have the NextPVR addon installed and recently activated comskip on my server. I’m not sure now how to figure out which other addons I’ve installed.

I see references to osmc and updates in the log excerpt. Do you suggest I post this over on the Kodi forum?

Thanks again.

Would it be possible to disable NextPVR when you’re not using OSMC?

WRT Comskip, is any part of it also installed on the Pi?

Yes, I could disable the NextPVR addon in OSMC when I’m not using it, then re-enable it when I do. I guess the idea there is that if the problem doesn’t recur, then it must have something to do with NextPVR?

Nothing was installed on the Pi (at least by me) when I installed it on the server.

It will at least suggest that NextPVR could be responsible.

Understood.

I’ve gone ahead and posted over in the Kodi forum here. I’ll let you know if I get any help there.

Well, as expected, they told me to get lost over at the Kodi forum. I got yelled at for posting a very small excerpt from the log file over there instead of using pastebin. They did tell me to enable debug logging so I’ve done that. I’ll switch that off when I try to use OSMC. Other than that, I got no assistance over there at all.

The next time this problem occurs, I’ll go look at the old kodi logs and try to post the whole log file on pastebin. I’ll bring that here first, and then I guess we’ll decide if it’s an OSMC problem or a Kodi problem or a NextPVR problem.

Right now, I’m going to look through the NextPVR logs for the day in question. I won’t dare go over there right now though. :slightly_smiling_face:

Does my plan look reasonable?

I appreciate the great support I get here. Thank you!

@darwindesign
It seems from recent posts that I’m running out of memory, probably caused by one of my addons (NextPVR?), so I’ll skip the power supply swap for now and enable debug logging without the overlay as you suggest.

I want to keep the option to turn on debug logging available in OSMC’s settings page, but I recognize that this setting will override anything in the advancedsettings.xml file so I’ll turn it off in settings and put following line in advancesettings.xml:

<advancedsettings>
    <loglevel>1</loglevel>  
</advancedsettings>

Setting loglevel to 1 is supposed to enable debug logging with no onscreen display.

Does this look right?

Thank you.

If that is the only thing in your advancedsettings.xml file then that will turn on debug logging without the overlay, but that will hide the debug option in the GUI.

If you enabled the GUI button then it would toggle between debug with overlay and debug without overlay (ie debug level 2 and 1). If that is what you want then you would…

<advancedsettings>
    <loglevel hide="false">1</loglevel>
</advancedsettings>

There is no option to toggle between 0 and 1 with the GUI if that is what you were thinking. As such this should only be used during diagnostics and you may want to reboot at the end of your TV watching sessions to keep the size of the logs down.

That’s actually what I was going for. Oh, well.

There is only this one setting in my advancedsettings.xml file. I set it as I originally proposed and, to your point, will take care to reboot after every use to manage the logfile size.

Slightly off-topic: I’ve enabled component-specfic and event logging in the GUI. Do you recommend that I keep those enabled?

I’m now looking forward to my system locking up quickly so my problem can be diagnosed and fixed.

Thank you again for your help.

You would normally only enable component specific logging for items that you knew were at issue and needed even more verbose logging to track down specifics. Adding more logging than necessary can be counter productive.

Well, my system locked up on me tonight. I had Kodi debug logging enabled and saved the last 10 minutes or so here: http://paste.osmc.tv/usaqufipoc.coffee

I have the full Kodi (old) log file if more information is needed.

I’m not sure that this is an OSMC problem, but I thought I’d start here.

Thank you for any help or suggestions.

Just a quick update. After I pulled the plug last night, I watched a couple recorded programs and all was well. I stopped using OSMC/Pi at around 22:00 local time.

This morning at 0800, the Pi was locked up again. I pulled the plug, then retrieved and saved kodi.old.log. I can’t save that full log to paste.osmc.tv, but I have it as a text file in case it would be helpful. Among other things, this log file reports numerous failed attempts by NextPVR to connect to my server before it locks up. I don’t understand that. My network and server appeared to be running normally at the time. The Pi shouldn’t lock up for a failed network connection, should it?

Thanks again for any assistance.

Does NextPVR download EPG info? If so, how many days worth of EPG info?

I don’t know if it actually downloads EPG info. I suspect it does, but that’s probably a question for the NextPVR guys. I’ve saved NextPVR logs to correspond with the Kodi logs.

Right now, I have OSMC set to display one past day and 7 future days of Guide information.

During the past week, I deliberately restarted OSMC every night before going to bed, and the Pi never locked up. Then I decided to let it go without restarting OSMC, and after 2 days, it locked up. I didn’t change any EPG setting in doing so.

Thank you!

You were going to switch off the NextPVR add-on each evening to see if the problem stopped occurring. And did it?

I did not do that yet, but it sounds like something I need to try. No disrespect to you intended.

Instead, I switched on kodi debug logging and restarted OSMC each night to manage the log file size. I thought this might help us get to the source of the problem quicker. What I found was that, after about two weeks of doing this, the Pi never locked up. So I decided to not restart OSMC each night to see how long the Pi would run without locking up. The answer was two days, and the debug log of that is posted here. Interestingly, perhaps, after pulling the plug, it locked up again overnight. I have a log file of that as well, but I can’t figure out how to save it on the paste site. It’s too big. It’s now 14:00 and the Pi is still accessible via SSH, 6 hours after I last rebooted it.

Looking at the log files, it appears that the only activity that occurs on the Pi is interactions with NextPVR, and these interactions occur very frequently. That makes me think the problem is almost certainly involving NextPVR. Something did come up regarding “skin helper” too. That’s documented in the logfile that I pasted, but it may mean nothing.

Anyway, if we agree that the next step needs to be disabling NextPVR each night, I’ll do that. If that’s the case, then I won’t restart the Pi each night, since that mitigates against a lockup – or so it seems.

Are the logs not useful? Should I turn off debug level logging to avoid the large log file size while we turn off the NextPVR addon each night?

The other thing I’ve thought about doing is running a cron job each night to automatically restart OSMC. That will take a bit of research on my part, and doing that will prevent us from figuring out what’s going on.

Thanks for your interest and patience.