The current process I’m following is to disable NextPVR when I’m not using it, then enable it when I am, then disable when I’m finished, etc. After disabling NextPVR, I restart OSMC, which “refreshes” the Pi’s memory. Since I use the Pi/OSMC at least once a day, hangs don’t happen. My understanding of your request was that I reboot, generate kernel logs until it hangs, then post them. Is that correct?
I will continue to follow the current process for another day or so, then I’ll post my results and follow your approach to collect results when it hangs. It takes 2-3 days for that to happen.
If we can’t figure out what’s happening, I’ll try another fresh installation and see if that helps. Failing that, I’ll step into the Linux command line even further and write a cron job to restart (or reboot) once a day, or something like that.
I seem to be the only one with this problem, so it must be caused by something I’ve done or how I use OSMC?
Note that I’ve added a column, OSMC, that indicates when OSMC was restarted. One thing that struck me about the latest results is that each restart of OSMC brought back less “available” memory than before. Last night, after restart, free reported 296 MB of memory, while the previous restart reported 363 MB.
One thing that jumped out at me was that now, ps is reporting two users named “OSMC” and is not reporting “Root” anymore.
If this is helping get to a resolution of my problem, I’ll keep doing it. If not, perhaps it’s time to let my system crash and report kernel logs, as requested by @JimKnopf?
I’ve posted additional results of my data collection efforts and await @dillthedog recommendations relative to them. I hate to disrupt that effort now if it’s becoming beneficial.
In reviewing your instructions above, I had some questions:
If there are multiple index id’s, which one is the correct one?
What is the “full log”? I’m not sure which index to report here versus above. Sorry to be thick.
Also, do I need to turn on debug logging for this exercise? I think not, but I want to confirm.
It seems that after disabling NextPVR, the Kodi VSZ/RSS figures remain pretty stable. In parallel with these, the buff/cache figures also flatline, so there’s probably little file I/O occurring.
The progressive reduction of available memory – even when NextPVR is disabled and the Kodi VSZ/RSS numbers are flatlining – is interesting. Perhaps something else is eating the available memory. Switching off Kodi altogether each night would be a useful test to see if available memory follows a similar pattern.
WRT to your question about kernel versus “full” log, I always thought (though am happy to be corrected) that using -k just produced a subset of the full log, and would therefore be of little help in this case.
Debug logs are specific to Kodi. You can leave it switched on but it’s an overhead and will create large logs if left to run for too long.
@JimKnopf described this above. Generally, you’ll be interested in the log for the previous boot, which would be -1. The boot before that one is -2, etc.
You’re sorting on RSS size, so the second “OSMC” line is a process that just happens to make it into second position by dint of it having the second largest RSS. It’s a process that’s run as part of your logon.
To switch off Kodi, run: systemctl stop mediacenter
OK. I understand now. I’ll go ahead and upload both the partial and full logs as described above, now that I understand what I’m supposed to do.
I’ll go ahead and proceed with @JimKnopf steps 1-11. Depending on what happens with that, I’ll proceed to shut down Kodi and see what effect that has on memory leakage.
Well, my power went out at 0300 and disrupted my kernel logging. I decided to go ahead an post the kernel logs as directed above in case they may be interesting and to confirm I’m doing the right thing. For what it’s worth, I continued monitoring memory myself and my system was chewing it up throughout the kernel logging process.
I’ll start over again today, with step 11, and see if our power company can keep electricity on long enough for my Pi to run out of memory.
Here is an excerpt from my terminal session this morning:
Have had a quick look but except you often do ssh logins as user osmc there is not anything unusual so far.
So, keep on till the problem repeats with active persistent kernel logs.
As I mentioned earlier, I suspect the Pi will run out of memory space today or tomorrow, and it will “hang”. I’ve noticed that available memory only declines. It never gets larger, as if one or more processes are not relinquishing memory after they run – or they never quit running. (I am a Linux illiterate, so I really have no idea what’s happening.)
Apologies. I misunderstood my instructions. I have instituted kernel logging as described by @JimKnopf above. All I’ve been doing since then leaving NextPVR enabled and waiting for the Pi to hang. When it does, my plan was to post the kernel logs.
I’ll begin stopping the media center when I’m finished using it and issuing a ps aux command before and after I do that. I’ll continue doing that until it hangs again.
Does the above make sense, or should I start over?
Here’s what I’m doing now: When I’m finished with OSMC, I run a script that shuts down Kodi, but takes ps aux readings before and after the stop mediacenter command is run. When I want to use OSMC, I run a similar script that starts Kodi and takes ps aux readings before and after the start mediacenter command is run. The scripts write out results in a file together with the time they’re run.
Using this approach, we can see how memory usage by processes changes while Kodi is “idle” and while it’s running. I’m also running persistent kernel logs in case they might become useful. I’ve just gone through a cycle where I shut Kodi down, started it, and shut it down again. ps aux results are presented below:
Sun Feb 28 16:59:16 CST 2021
Before Stop
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
osmc 462 25.9 26.9 677356 206116 ? Sl Feb26 869:41 /usr/lib/kodi/kodi.bin --standalone -fs
root 160 0.0 1.1 59952 9012 ? Ss Feb26 0:01 /lib/systemd/systemd-journald
After Stop
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 160 0.0 1.2 59952 9336 ? Ss Feb26 0:01 /lib/systemd/systemd-journald
osmc 3594 0.3 0.7 12036 5712 ? Ss 16:58 0:00 /lib/systemd/systemd --user
Sun Feb 28 17:00:10 CST 2021
Before Start
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 160 0.0 1.2 59952 9336 ? Ss Feb26 0:01 /lib/systemd/systemd-journald
osmc 3594 0.1 0.7 12036 5712 ? Ss 16:58 0:00 /lib/systemd/systemd --user
After Start
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 160 0.0 1.2 59952 9404 ? Ss Feb26 0:01 /lib/systemd/systemd-journald
osmc 3594 0.1 0.7 12036 5712 ? Ss 16:58 0:00 /lib/systemd/systemd --user
Sun Feb 28 17:46:21 CST 2021
Before Stop
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
osmc 3687 26.2 26.1 654240 199656 ? Sl 17:00 12:06 /usr/lib/kodi/kodi.bin --standalone -fs
root 160 0.0 1.2 59944 9520 ? Ss Feb26 0:02 /lib/systemd/systemd-journald
After Stop
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 160 0.0 1.2 59944 9780 ? Ss Feb26 0:02 /lib/systemd/systemd-journald
osmc 4009 0.6 0.7 12036 5640 ? Ss 17:46 0:00 /lib/systemd/systemd --user
I hope the above results are readable.
Before I get too far along with the approach, I want to make sure I’m doing the right thing. I’ll note that just before I shut Kodi down the first time, it had only 63 MB of “available” memory left. It was cruising toward a hang event, but once I shut Kodi down, the memory was relinquished and the process consuming the most memory was root.
I think, per @dillthedog, we’re looking to see what happens to “RSS” on repeated restarts of Kodi, and perhaps if there is a growth in memory usage while Kodi is stopped, but correct me if I’m wrong.
We’ll probably go through another cycle or two of starts and stops tonight, and I’ll post results in the above format if that’s helpful.
@FrogFan@dillthedog: I don’t understand the idea behind starting and stopping the mediacenter?
If you know it’s most likely that kodi is the devil with a memory leak, why not let it run continously which is the normal usage?
If you are not sure that kodi.bin is causing a memory leak, why only monitor user specific processes instead of all with all available memory information like ps aefvx or at least ps afvx ?
Sorry, if I missed something in the beginning of the thread.
I’ll let @dillthedog provide the answers to your questions. For my part, I’m just a user with a problem looking for a solution and thankful for all the help I can get.
My Pi is chewing up a lot of memory just sitting there – so much, I think, that it has to be one of the “top” processes that is responsible.
We first thought the problem was with NextPVR, so we went through a series of exercises to disable and enable that addon. We discovered, I think, that it was not causing the problem, so attention shifted to Kodi. That’s when we started stopping and starting Kodi. I went through an additional start/stop cycle last night and can post results of that, or I can go another day and then post.
I’m still running persistent kernel logs and can post them at any time. In particular, if we decide to quit starting and stopping Kodi, then I predict the Pi will run out of memory and hang in 2-3 days.
If there are other things I need to be doing, please advise.
I appreciate your support and the support from @dillthedog. I really hope to get this problem resolved soon. I’ve got my wife using OSMC now, and that was a struggle. Now I have to babysit her when she’s running it in order to collect this data and help her if it hangs.
