Can ping RPi, but can't SSH into it

Help and Support Raspberry Pi
85

@FrogFan has described the situation clearly.

While NextPVR/Kodi was a prime suspect, evidence so far doesn’t seem to confirm this hypothesis.

Disabling NextPVR didn’t seem to stop the available memory from decreasing.

Just in case there was some background NextPVR process still running, though the add-on was disabled, Kodi was restarted. That still showed a progressive reduction in available memory, though Kodi memory usage remained flat.

Rather than let it crash and try to figure out what caused it, I preferred to stop Kodi and see if we can identify a process that’s progressively eating the memory. Alternatively, if the available memory stops falling when Kodi is shut off, we need to revisit Kodi as a potential culprit.

86

Tonight, just after starting Kodi, the Pi hung. It had been running since Thursday. Here is an excerpt of the session where I gathered kernel logs:

osmc@osmc:~$ sudo journalctl --list-boots --no-pager
-1 25e4c50875e74d58a8b291a5936f6519 Thu 2019-02-14 04:11:58 CST—Mon 2021-03-01 21:06:23 CST
 0 044a059709d242a5aa243102894939e4 Mon 2021-03-01 21:08:44 CST—Mon 2021-03-01 21:16:30 CST
osmc@osmc:~$ sudo journalctl -k -b -1 --no-pager|paste-log
https://paste.osmc.tv/hesovurawu
osmc@osmc:~$ sudo journalctl -b -1 --no-pager|paste-log
https://paste.osmc.tv/zodomomupi

Note that logs are here: https://paste.osmc.tv/hesovurawu and full logs are here: https://paste.osmc.tv/zodomomupi

As noted earlier, I have been tracking output of ps aux for two days, saving output before and after issuing start and stop Kodi commands. Here is the output for that period:

Before Stop
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
osmc       462 25.9 26.9 677356 206116 ?       Sl   Feb26 869:41 /usr/lib/kodi/kodi.bin --standalone -fs
root       160  0.0  1.1  59952  9012 ?        Ss   Feb26   0:01 /lib/systemd/systemd-journald
After Stop
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       160  0.0  1.2  59952  9336 ?        Ss   Feb26   0:01 /lib/systemd/systemd-journald
osmc      3594  0.3  0.7  12036  5712 ?        Ss   16:58   0:00 /lib/systemd/systemd --user
Sun Feb 28 17:00:10 CST 2021
Before Start
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       160  0.0  1.2  59952  9336 ?        Ss   Feb26   0:01 /lib/systemd/systemd-journald
osmc      3594  0.1  0.7  12036  5712 ?        Ss   16:58   0:00 /lib/systemd/systemd --user
After Start
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       160  0.0  1.2  59952  9404 ?        Ss   Feb26   0:01 /lib/systemd/systemd-journald
osmc      3594  0.1  0.7  12036  5712 ?        Ss   16:58   0:00 /lib/systemd/systemd --user
Sun Feb 28 17:46:21 CST 2021
Before Stop
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
osmc      3687 26.2 26.1 654240 199656 ?       Sl   17:00  12:06 /usr/lib/kodi/kodi.bin --standalone -fs
root       160  0.0  1.2  59944  9520 ?        Ss   Feb26   0:02 /lib/systemd/systemd-journald
After Stop
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       160  0.0  1.2  59944  9780 ?        Ss   Feb26   0:02 /lib/systemd/systemd-journald
osmc      4009  0.6  0.7  12036  5640 ?        Ss   17:46   0:00 /lib/systemd/systemd --user
Sun Feb 28 21:28:11 CST 2021
Before Start
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       160  0.0  1.4  59936 10892 ?        Ss   Feb26   0:02 /lib/systemd/systemd-journald
osmc      4178  1.1  0.7  12036  5688 ?        Ss   21:27   0:00 /lib/systemd/systemd --user
After Start
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       160  0.0  1.4  59936 10904 ?        Ss   Feb26   0:02 /lib/systemd/systemd-journald
osmc      4178  1.0  0.7  12036  5688 ?        Ss   21:27   0:00 /lib/systemd/systemd --user
Sun Feb 28 22:25:26 CST 2021
Before Stop
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
osmc      4248 31.5 18.6 601876 142832 ?       Sl   21:28  18:03 /usr/lib/kodi/kodi.bin --standalone -fs
root       160  0.0  1.1  59944  8420 ?        Ss   Feb26   0:02 /lib/systemd/systemd-journald
After Stop
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       160  0.0  0.9  59944  7556 ?        Ss   Feb26   0:02 /lib/systemd/systemd-journald
root      4783  0.3  0.6  10492  5292 ?        Ss   22:24   0:00 sshd: osmc [priv]
Mon Mar  1 21:05:29 CST 2021
Before Start
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       160  0.0  1.5  59936 11620 ?        Ss   Feb26   0:02 /lib/systemd/systemd-journald
osmc      5431  0.7  0.7  12036  5720 ?        Ss   21:05   0:00 /lib/systemd/systemd --user
After Start
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       160  0.0  1.5  59936 11620 ?        Ss   Feb26   0:02 /lib/systemd/systemd-journald
osmc      5431  0.7  0.7  12036  5720 ?        Ss   21:05   0:00 /lib/systemd/systemd --user

I’m confused by the above data; it does not seem to indicate that any process was consuming a lot of memory when I started Kodi, but yet the Pi hung right after I did.

As always, I look forward to any insight we can get from this data. Thanks very much for the support.

87

Looked at the kernel logs and I cannot find any helpful information which explains what happened after 2021-03-01 21:06:23.

Just some brainstorming:

  • there are no new connmand entries before the end of the kernel logs, so there does not seem to be an issue with the ip address like having got a new but different ipv4
  • this system uses wifi … does this occur as well if using wired eth0 ?
    (Already saw your efforts in OSMC 5G wifi adapter installation on a fresh install of OSMC)
  • although there are no mmc0 related timeouts/errors/warnings: Is the sd card still ok?
  • is the CPU temperatur within healthy range all the time: vcgencmd measure_temp?
    (You can also monitor the hardware temperature and free memory at GUI->Settings->System Info->Hardware in kodi)
  • is the power supply sufficient for the pi or are there any voltage drops making the system operation unstable? Is there a possible correlation switching on/off other electricity consumers in the house?
  • I suggest to monitor all processes for a while using the ps afvx command I mentioned before … till now there were no symptoms of an out of memory condition in the logs
  • effect of solar winds? (just kidding :wink: )
88

I have one location in my house that is like this. Over the years it has seen three different RPi’s, a handful of PSU’s, the outlet itself has been changed, but it has always had a limited amount of time it could stay up without locking up. I always knew it was a power delivery issue at just that plug so never looked into how it manifested itself in the logs. I have since switched to powering it using POE (power over ethernet) using a cheap splitter and the issue went away. I think these RPi’s can be quite sensitive about having clean power come in.

1 Like
89

This is another cosmetic issue (unrelated to the freeze) you could solve by

sudo umount /boot
sudo fsck.fat -va /dev/mmcblk0p1
sudo mount /boot
90

Thank you @JimKnopf and @darwindesign, for your responses. I’ve reflected a bit on them and here are my thoughts:

Regarding wifi vs. ethernet: It is awkward to run an ethernet cable to my pi. I have to stretch a cable across the living room floor to an electrical outlet with a powerline ethernet adapter connected to it. I’ve tried connecting the powerline adapter to an outlet closer to the pi, but I can’t get enough bandwidth from it, presumably due to interference from other power consumers – TV, satellite receiver, audio receiver, etc. So, I guess I could test using ethernet, but it’s not a long term solution for me.

Regarding the sd card: As I documented elsewhere on this site, the card I’m currently using was new in December. I certainly hope it’s not worn out already, but a fresh install using another new card is something I could try.

Regarding CPU temperature: Several months ago, I started getting the “temperature icon” in the display while running OSMC. I moved the pi further away from my TV and at a spot with better air circulation (it has a heat sink installed), and the icon disappeared and never returned. I just issued the vcgencmd measure_temp command and the result was 50.5 degrees C. That seems well within the acceptable range (<83 degrees C). The pi is currently idle. I’ll have a look at temp when we’re using it.

Regarding power supply: The pi I’m using is part of a CanaKit that included a CanaKit power supply. I’m currently using that power supply. It’s now 3 years old (as is the pi), so I guess it could be wearing out. A new power supply is something I could try.

We’ve had challenges with power recently in Texas, but my issues pre-dated all that drama. There are several nearby power consumers on the same circuit as the pi (see above) and all has been well with that for quite awhile. I suppose another approach would be to plug the pi’s power supply into another circuit, but that would involve stretching a cord across my living room floor. That won’t work long term.

Regarding the cosmetic issue: I’ll issue the umount and mount commands as you specify, @JimKnopf.

@darwindesign, It looks like POE involves the purchase of some more hardware and an ethernet drop. I don’t mind buying more hardware, but I don’t have an ethernet drop near the pi, so this option appears infeasible for the same reason I can’t use ethernet for data.

It appears that my options at this point, unless there are other recommendations to come, are to replace the power supply or the sd card.

It occurs to me that there are a couple other options: I could swap out the current pi, sd card, and power supply with a 3B+ or 4 kit. The 3B+ takes the troublesome (for me) OSMC 5G wifi adapter out of the loop. The 4 positions me for the upcoming major OSMC release that supports the pi 4. Any thoughts from the experts on these options?

In the meantime, I guess I’ll explore the bash shell some more and figure out how to implement a cron job to reboot the pi every day or so at, say, 0300, when I’m almost certainly not going to be running it.

As always, I appreciate the great support I get here. Thank you.

91

Correct. My comment was not intended as a suggestion as the cost to get a POE enable jack at that location would be prohibitive to most. My point was just to spell out that it is possible to have problematic power at the wall, and in a single location, that isn’t easily mitigated.

92

It was not my intention to suggest to use a permanent ethernet connection. But if you can reproduce the issue within x days with active wifi but you cannot using wired ethernet, this would bring some new information about a possible root cause. If the problem is the same using a wired connection you know it’s independent from using wifi.

93

I understand and I’ll try it if my wife will let me. :slightly_smiling_face:

My thinking was that if the problem occurs using wifi, we’ve learned nothing. If it does not, then the implication is that there is a wifi problem and I need to use ethernet. I can’t use ethernet long term. But, I understand now from your response that if it is a wifi problem, then maybe we can fix it.

I’ll put this on my list of things to try.

94

LOL. I thought we were looking at boring old memory usage, though it looks like you’ve moved on a bit since then.

TBH, this thread seems to be losing focus and I’m not sure where we’re at right now. Did you ever establish if the device ran out of memory?

95

TBH, I don’t know where we are either. Maybe you can help me regain focus? :slightly_smiling_face:

Over the period Feb 28 through Mar 1 (2 days), I stopped and started Kodi three times, as documented above. I took ps aux readings before and after each stop and start event. Here are the relevant ps aux readings over time for the relevant events, extracted from the previous post:

Before start OSMC
osmc RSS:  5712,  5688,  5720
root RSS:  9336, 10892, 11620

After start OSMC
osmc RSS:  5712,  5688,  5720
root RSS:  9404, 10904, 11620

Before stop OSMC
osmc RSS: 206116, 199656, 142832
root RSS:   9012,   9520,   8420

After stop OSMC
osmc RSS:   5712,   5640,      -
root RSS:   9336,   9780,    7557
root RSS:                    5792

If we look at RSS for successive readings before OSMC is stopped, then we see a pattern of slight reduction in memory assigned to OSMC. That does not suggest a problem with OSMC’s memory consumption to me.

Looking at memory usage after I started OSMC for the last time (before the pi hung), you’ll see very little memory used, yet the pi hung almost immediately after starting OSMC.

Indeed, these data do NOT show a memory issue to me.

The other piece I don’t understand is that, when I issue free -m commands over time, the pattern is a consistent decrease in “available” memory, around 5-7 MB per hour, independent of what’s supposed to be running. That is, the free and ps commands appear to be giving us conflicting information.

Everyone responding to this thread knows a lot more about OSMC and Linux than I do, so I’m quite happy to take a steer from anyone. That’s part of the reason why the thread has lost focus.

I am absolutely willing to continue to pursuing memory as the issue if that’s indicated. While I suppose it’s quite possible for the problem to be with the power supply, the wifi adapter, or the sd card, it seems unlikely that any of those would contribute to what I’m seeing as a near-constant rate of memory leakage, at least when measured by free -m.

@dillthedog, I am putty in your hands! (No pun intended! :smiley:)

Thanks!

96

We dogs don’t have hands, so that’s a particularly paw pun.

Anyway, write 100 lines: “Plan to work. Work to plan”.

Back on focus, I’ll repeat what I wrote above:

For that apprach to have any real value, you need to monitor the available memory, using the free command, ideally run via cron, as well as the output from ps. (Did you ever get to grips with cron?)

97

:joy:

I was about to take a stab at cron. I installed crontab via the My OSMC interface. I was actually going to give up on this exercise and use a cron job to reboot every night, but I’m now heady with optimism and will prepare a cron job to continue monitoring memory. A couple questions and I’ll take a stab, with, perhaps some more help from you.

  1. Should we continue stopping and starting OSMC? That doesn’t seem indicated to me since it now appears that Kodi is not the problem

  2. I’m thinking of writing a script, to be executed via cron, that creates and updates two files: one with ps data and the other with free data. I’ll also put the date (and time) in these files

  3. Is ps aux the right command, or should we get additional data to monitor additional processes? This seems right, since neither root nor osmc seem to be the culprits

  4. How often should we run the cron job?

I get the big idea with cron, but I’m sure I will have some problems with it, so based on your recommendations, I’ll prepare a script together with a crontab entry for your review.

Back in business! Onward through the fog!

Thanks!

98

Sorry for taking so long to reply. Busy day.

I think it’s fair to say that nothing is 100% set in stone. We saw that Kodi’s memory usage (VSZ/RSS) remained flat while the available memory continued to fall. That’s a strong pointer but never say never. So, I’d recommend that you continue to stop Kodi when you’re not using it, while also continuing to collect memory-usage statistics.

Sounds good.

It’s the one I often use. :slight_smile: From the man page:

   To see every process on the system using BSD syntax:
      ps ax
      ps axu

Every 30 minutes should be enough. That’ll be */30 * * * * on the crontab file.

And don’t forget to look for a second SD card.

99

No problem at all. I used to have busy days, but not so much anymore! :slightly_smiling_face:

Here is what I’ve done in response to your recommendations. Feel free to steer me differently as appropriate.

  1. I created a new directory: /home/osmc/bin, where I put my scripts and where I’ll put my data. I read somewhere if I created a bin directory, it would simplify script execution, but that didn’t happen. I need to use full paths to execute them. I tested them, but haven’t seen anything work in crontab yet.

  2. I created three scripts: mediastart, mediastop, and monitor, all in /home/osmc/bin. Monitor is intended to be executed via cron. mediastart and mediastop are used to start and stop the mediacenter and record when it was done. I’ll run these manually when I stop and start using OSMC. Monitor looks like this:

     #!/bin/bash
 date >> free
 free -m >> free
 date >> ps
 ps aux --sort -pid >> ps

My crontab currently looks like this:

0 3 * * * sudo shutdown -r now >> /home/osmc/bin/rebootlog 2>&1
*/30 * * * * /home/osmc/bin/monitor >> /home/osmc/bin/cronlog 2>&1

I updated the crontab more than 30 minutes ago, but I can’t find anything in cronlog, so I’m thinking I’ve got something wrong. If you spot something, let me know. Meantime I’ll go looking myself.

You’ll note I went ahead and put a reboot in for every “night” at 3:00 am to reclaim some space. If you want me to take that out for now, just say the word. I was hoping to use that command to test my newfound cron expertise. :slightly_smiling_face:

If you spot anything I could do differently or better, please advise.

Thanks for your help.

100

  1. Full paths in cron are always a good idea. The scripts can be anywhere you like.

  2. I’d suggest you rename the output files to something like free.out or free.log. Similarly for the ps output. You should also specify a directory for these files.

  3. If the scripts aren’t running is cron installed and enabled:

    systemctl status cron

    and are the scripts executable?

  4. I see you remembered to send stderr to the log file (2>&1). Nice one.

  5. Don’t reboot the machine each night. That might be a workaround for the future but for now you want to see what’s eating the memory, so a nightly restart would be self-defeating. (Just to be clear, don’t reboot the machine. By then I’d expect Kodi to have been stopped, anyway.)

1 Like
101

Definitely – PATH has a habit of omitting some paths like /sbin when running from cron.

102

Uh oh. I’ve got Sam’s interest. I better behave myself! :smile:

Thank you Sam!

103

I’ve specified full paths for all files. Cron just ran successfully. I’m a bit confused because there’s nothing in cron.log (I renamed the file), but it was created. Also, ps.log and free.log were created and populated with data.

:blush: I’m blushing! Thanks for noticing!

I took the reboot entry out of crontab. I understand that’s self defeating at this point. One point I noted though, was that stopping and starting mediacenter did not “reclaim” memory like a reboot did. The system eventually did hang up. (I realize as I type this that’s probably obvious to everyone here!)

Thanks again

104

It would show the terminal output from running the script. But since you’re redirected terminal output to free.log and ps.log, that only leaves error messages, so, as long as everything works, nothing should be written to cron.log.