OSMC, RPi2 - continuous out-of-memory problem

Help and Support Raspberry Pi
1

Hi All,

I would like to use OSMC on a Rapsberry Pi 2 hardware, but the “kodi.bin” process after some hours always is killed by the kernel with “out-of-memory” problem.

The “journalctl” always records the following:

aug 27 21:16:55 osmc mediacenter[250]: Kodi exited with return code 137 after 24 hours,20 minutes and 4 s
aug 27 21:16:55 osmc sudo[277]: pam_unix(sudo:session): session closed for user osmc
aug 27 21:16:54 osmc kernel: Killed process 278 (kodi.bin) total-vm:1205908kB, anon-rss:657304kB, file-rss
aug 27 21:16:54 osmc kernel: Out of memory: Kill process 278 (kodi.bin) score 877 or sacrifice child
aug 27 21:16:54 osmc kernel: [ 941] 1000 941 1300 148 6 0 0 0
aug 27 21:16:54 osmc kernel: [ 940] 1000 940 2713 168 8 0 0 0
aug 27 21:16:54 osmc kernel: [ 938] 1000 938 6095 277 9 0 0 0
aug 27 21:16:54 osmc kernel: [ 937] 1000 937 1083 98 5 0 0 0
aug 27 21:16:54 osmc kernel: [ 935] 0 935 2713 191 10 0 0 0
aug 27 21:16:54 osmc kernel: [ 547] 106 547 1323 167 6 0 0 0
aug 27 21:16:54 osmc kernel: [ 322] 0 322 1406 30 5 0 0 0
aug 27 21:16:54 osmc kernel: [ 321] 0 321 18486 1525 17 0 0 0
aug 27 21:16:54 osmc kernel: [ 305] 1000 305 78638 6490 63 0 0 0
aug 27 21:16:54 osmc kernel: [ 301] 0 301 8726 201 13 0 0 0
aug 27 21:16:54 osmc kernel: [ 296] 0 296 1020 78 7 0 0 0
aug 27 21:16:54 osmc kernel: [ 295] 0 295 1757 81 7 0 0 0
aug 27 21:16:54 osmc kernel: [ 293] 0 293 4097 154 9 0 0 0
aug 27 21:16:54 osmc kernel: [ 286] 0 286 1445 113 8 0 0 0
aug 27 21:16:54 osmc kernel: [ 281] 1000 281 1395 72 6 0 0 0 aug 27 21:16:54 osmc kernel: [ 278] 1000 278 301477 164326 452 0 0 0
aug 27 21:16:54 osmc kernel: [ 277] 0 277 1437 117 7 0 0 0
aug 27 21:16:54 osmc kernel: [ 262] 0 262 533 62 6 0 0 0
aug 27 21:16:54 osmc kernel: [ 250] 0 250 837 84 6 0 0 0
aug 27 21:16:54 osmc kernel: [ 242] 0 242 1737 139 9 0 0 -1000
aug 27 21:16:54 osmc kernel: [ 237] 0 237 1628 138 6 0 0 0
aug 27 21:16:54 osmc kernel: [ 236] 105 236 852 76 5 0 0 0
aug 27 21:16:54 osmc kernel: [ 235] 0 235 927 56 6 0 0 0
aug 27 21:16:54 osmc kernel: [ 234] 104 234 1216 143 6 0 0 -900
aug 27 21:16:54 osmc kernel: [ 232] 105 232 879 73 5 0 0 0
aug 27 21:16:54 osmc kernel: [ 231] 0 231 855 69 6 0 0 0
aug 27 21:16:54 osmc kernel: [ 137] 0 137 2768 149 7 0 0 -1000
aug 27 21:16:54 osmc kernel: [ 109] 0 109 2098 260 9 0 0 0
aug 27 21:16:54 osmc kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj
aug 27 21:16:54 osmc kernel: 2048 pages cma reserved
aug 27 21:16:54 osmc kernel: 2603 pages reserved
aug 27 21:16:54 osmc kernel: 0 pages HighMem/MovableOnly
aug 27 21:16:54 osmc kernel: 192512 pages RAM
aug 27 21:16:54 osmc kernel: Total swap = 0kB
aug 27 21:16:54 osmc kernel: Free swap = 0kB
aug 27 21:16:54 osmc kernel: Swap cache stats: add 0, delete 0, find 0/0
aug 27 21:16:54 osmc kernel: 0 pages in swap cache
aug 27 21:16:54 osmc kernel: 2925 total pagecache pages
aug 27 21:16:54 osmc kernel: Normal: 2644kB (UEMC) 608kB (UEMC) 4616kB (UEMC) 1332kB (UEMC) 9*64kB (EM
aug 27 21:16:54 osmc kernel: lowmem_reserve: 0 0
aug 27 21:16:54 osmc kernel: Normal free:13104kB min:12288kB low:15360kB high:18432kB active_anon:696564kB
aug 27 21:16:54 osmc kernel: active_anon:174141 inactive_anon:1877 isolated_anon:0
active_file:303 inactive_file:319 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:1626 slab_unreclaimable:2247
mapped:1357 shmem:2204 pagetables:717 bounce:0
free:3276 free_pcp:47 free_cma:39
aug 27 21:16:54 osmc kernel: Mem-Info:
aug 27 21:16:54 osmc kernel: 5fe0: 015114b4 7ef0c338 00761a8d 00761a8c 60010030 ffffffff
aug 27 21:16:54 osmc kernel: 5fc0: 7ef0c344 01ddb728 01ddb870 7ef0c400 01ddb870 7ef0c3d4 7ef0c58c 7ef0c3f8
aug 27 21:16:54 osmc kernel: 5fa0: 7ef0c344 00000001 03b95860 03b95864
aug 27 21:16:54 osmc kernel: Exception stack(0xac325fb0 to 0xac325ff8)
aug 27 21:16:54 osmc kernel: [<800092e0>] (do_PrefetchAbort) from [<806fe7e4>] (ret_from_exception+0x0/0x1
aug 27 21:16:54 osmc kernel: [<806fec88>] (do_page_fault) from [<800092e0>] (do_PrefetchAbort+0x44/0xa8)
aug 27 21:16:54 osmc kernel: [<8018ac88>] (handle_mm_fault) from [<806fec88>] (do_page_fault+0x330/0x3e4)
aug 27 21:16:54 osmc kernel: [<80152938>] (filemap_fault) from [<8018ac88>] (handle_mm_fault+0x94c/0x1408)
aug 27 21:16:54 osmc kernel: [<8015b8dc>] (__alloc_pages_nodemask) from [<80152938>] (filemap_fault+0x248/
aug 27 21:16:54 osmc kernel: [<80154a98>] (out_of_memory) from [<8015b8dc>] (__alloc_pages_nodemask+0xa54/
aug 27 21:16:54 osmc kernel: [<8015428c>] (oom_kill_process) from [<80154a98>] (out_of_memory+0x460/0x494)
aug 27 21:16:54 osmc kernel: [<80153968>] (dump_header.isra.15) from [<8015428c>] (oom_kill_process+0x25c/
aug 27 21:16:54 osmc kernel: [<806f6c48>] (dump_stack) from [<80153968>] (dump_header.isra.15+0xa0/0x26c)
aug 27 21:16:54 osmc kernel: [<8001afd8>] (show_stack) from [<806f6c48>] (dump_stack+0x9c/0xdc)
aug 27 21:16:54 osmc kernel: [<8001f5c0>] (unwind_backtrace) from [<8001afd8>] (show_stack+0x20/0x24)
aug 27 21:16:54 osmc kernel: Hardware name: BCM2709
aug 27 21:16:54 osmc kernel: CPU: 0 PID: 278 Comm: kodi.bin Not tainted 4.1.3-2-osmc #1
aug 27 21:16:54 osmc kernel: kodi.bin invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

The most interesting thing in this symptom: the out-of-memory crash always occurs independently that I use the Kodi - or not (not used = just power up the RPi2 but nothing watch or listen on Kodi).

If I check the amount of free memory (date || free) my experience is same as the kernel predicates: the amount of free memory is continuously decreasing, for example:

osmc@osmc:~$ date && free
2015. aug. 28., péntek, 09.47.02 CEST
total used free shared buffers cached
Mem: 751444 385252 366192 8816 39656 186344
-/+ buffers/cache: 159252 592192
Swap: 0 0 0

osmc@osmc:~$ date && free
2015. aug. 28., péntek, 16.18.41 CEST
total used free shared buffers cached
Mem: 751444 470452 280992 8816 46148 262828
-/+ buffers/cache: 161476 589968
Swap: 0 0 0

osmc@osmc:~$ date && free
2015. aug. 28., péntek, 18.45.24 CEST
total used free shared buffers cached
Mem: 751444 473788 277656 8816 49164 262832
-/+ buffers/cache: 161792 589652
Swap: 0 0 0

Does anybody have any idea what do I have (or Can I do ) to solve this problem?

Or Does somebody have same experience?

Thanks in advance,
SziroG

3

Have you edited advancedsettings.xml? In particular setting too high a cachemembuffersize would cause this issue.

4

Hi,

Thank you for your answer - but my Kodi configuration does not use any network share (nor SAMBA neither NFS).

(I would like to use Kodi+TVHeadend+Sundtek USB DVB-T stick on a Raspberry Pi 2 for watching live TV)

What I cannot understand: why decreasing the operation system free memory if the Kodi is in idle state???

I think the gist is somewhere about this…

BTW, Does somebody know what does the “return code 137” mean (“02:25:22 osmc mediacenter[249]: Kodi exited with return code 137 after 4 hours, 51 minutes and 35 s”)?

Thanks in advance,

5

Return code 137 means that Kodi was terminated with signal 9, (137-128=9) which is SIGKILL. So all that is telling you is that the out of memory killer killed Kodi with SIGKILL.

What would be useful is if you monitor free memory via SSH using ‘top’ to see what is actually using the memory - run top, then press ‘c’ to show full process names and then M (capital M) to sort by memory use.

BTW please use the paste-log command to upload logs, don’t paste large logs directly into your post.

6

Thank you for your answer!

So the return code also achieves to others what I have experienced.

And sorry for my first “large” post…

If you are an OSMC Developer I have played a lot of times with the Kodi settings and I have observed the next in connection with my symptom:

  1. The free memory slimming does not depend on the acceleration settings (dvbplayer with MMAL or OMXplayer),

  2. But the speed of the decreasing (e.g. how many free memory decreases in one hour or one day) strongly depends on the following settings:

2/a “Adjust display refresh rate” (Settings / Video / Playback / Adjust display refresh rate): if this setting is not in its “Off” state, the free memory is more quickly decreasing,
2/b “Sync playback to display” and its “A/V sync method” (Settings /Video / Playback / Sync playback to display ): if this setting is not in its “Off” state, the free memory is more quickly decreasing,
2/c The video output resolution and refresh rate (Settings / System / Video output / Resolution, Refresh rate): if the selected output resolution is not divisible by 25 the free memory is more quickly decreasing

And some background information (I think these are also important):

  • I have got an old “desktop” LCD monitor (HP L1950) which populate the 1280x1024@60Hz settings as its “recommended settings”,

  • I have also got in that configuration a HDMI-VGA converter (width audio),

  • in my country the DVB-T broadcast are encoded in standard HD/SD resolution with “i25” (1440x1080i25 or 704x576i25) !!!

I am really very concerned how can the OSMC harmonize the totally different refresh rates and resolutions (monitor - video output - broadcast source)?

7

A more likely cause of increasing memory consumption is some addon you have installed, especially a service addon.

Have you tried disabling all addons that you have installed to see if memory usage still keeps increasing ?

8

I think, I have installed only one addon (TVHeadend HTSP Client) but this is necessary for the live TV watching.

Or does the Kodi automatically install any addons?

I try to check the addons and going to repeat my test case - I will come back with the result soon!

(My non “official world record” is 43 hours, 6 minutes and 43 seconds operation of the “Kodi.bin” before the kernel is killed it. I could not find any combination of settings when the “Kodi.bin” can operate longer period of time than the previously specified.)

9

So, the latest test was not too successful…

I disabled all of the plugins what I could (where was enabled the “Disable” button in the addon settings window).

In my first test I tried to watch a HD channel but within 2 hours the “Kodi.bin” was killed.

Secondly I tried a SD channel but the result was as I get generally…(Kodi exited with return code 137 after 22 hours, 46 minutes and 13 seconds)

After this I do not think there is any relationship between the “memory eating” and any addons!

Next experience (sorry if I repeat myself):

After my second test (“Kodi.bin” restarted automatically) the system is kept in idle state.

If I now check the free memory I get the following (shortly just the gist):

KiB Mem: 751444 total, 554060 used, 197384 free, 103024 buffers

After a normal boot the the used and the free memory generally are vica-versa (used is generally around 200000, free is generally around 500000).

Why did the Kodi reverse more than 300000 KiB free memory in idle state during 24-26 hours???

I think this is the biggest question!

Does somebody know the answer for this question?

10

Try renaming /home/osmc/.kodi directory and rebooting. This will reset Kodi to defaults, if this fixes the memory problem then add back your addons etc one at a time until you see the problem again

11

Thanks your reply, but your idea is not the best…because of:

  1. I have tried 3 distributions (OSMC, Openelec, XBian) from Kodi version 14 and all versions on all distributions have this behaviour, the Kodi is always “eats” all free memory (Just my favourite distribution - the OSMC - I have rebuilt at least 10 times from zero: format the microSD card,…). After this I am (almost) sure the problem is in somewhere the Kodi source!

  2. If you were right, what should it worth for me? I would like to use the Kodi for live TV watching. In that case I need some plugins (minmum the HTPS TVHeadend client) and the TVHeadend server. I have to use them.

But I will try your idea, I do not want to predestinate the result but (I think) it will be same…

13

The current state is now (after the directory deleting & boot)

KiB Mem: 751444 total, 242544 used, 508900 free, 23480 buffers

Let’s see the values tomorrow evening!

14

The thing is you are the only one experiencing this problem out of some 400,000 or so. Now we need to figure out what is unique to you, maybe a combination of things working together?

15

As dilligaf mentions you seem to be the only one reporting this issue, so there is something specific to your configuration that is triggering this issue. Perhaps an addon installed, perhaps some configuration setting, hard to know for sure without a process of elimination.

I leave my Pi 2 running for days at a time and see no such out of memory problem in fact I think I’ve only seen maybe one or two out of memory reports on the forum at all, and both of those were 256MB models. Running out of memory on a 1GB Pi 2 is unheard of on our forum.

Nobody is suggesting that you go without your addons forever, but if you are serious about trying to solve this problem only a process of elimination will help you.

Start with a completely fresh .kodi directory with no custom configuration at all and leave it running overnight then check free memory in the morning - if there is no problem with free memory add your configuration settings or addons back only one or two at a time and then give it time to see if the problem returns.

Why would you think that ? What you are saying is that a fresh install of any Kodi distribution (OE/OSMC/Xbian) for everyone suffers from an issue where it runs out of memory when left for a day or two. This is clearly not the case both based of personal experience and looking at our forum postings.

Something that you are adding or configuring is triggering it, it is just a matter of identifying what.

16

Ok, I understand and what I can do, I will do!

Unfortunately we do not have to wait tomorrow.
Here are the values after 12 mins “operation” in idle state:

osmc@osmc:~$ uptime
23:27:16 up 12 min, 1 user, load average: 0,52, 0,50, 0,28
osmc@osmc:~$ free
total used free shared buffers cached
Mem: 751444 302208 449236 8816 26476 133720
-/+ buffers/cache: 142012 609432
Swap: 0 0 0

But I leave the system untouched till tomorrow evening and I will check the values again!

17

By the way, running free is not very informative because you want to see the memory used by each process, not free memory, which will vary with buffer and page caches use and other processes. Use the following command:

ps aux --sort -rss | head -n 20

This will show the top 20 memory hogging programs sorted on RSS (resident set size) - RSS is how much physical ram a process is currently using.

Don’t take too much notice of the larger VSZ figure - this is virtual memory size, and it is normal for a lot of virtual memory to be allocated which is never actually used or is not used all at the same time.

As an example here is what I currently see:

osmc@rpi2:~$ ps aux --sort -rss | head -n 20
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
osmc       370 22.3 15.5 548736 116868 ?       Sl   21:48  11:14 /usr/lib/kodi/kodi.bin --standalone -fs --lircdev /var/run/lirc/lircd
root       451  0.0  1.4  28144 10976 ?        Ss   21:49   0:00 /usr/sbin/smbd -D
osmc       472  0.0  1.0 190828  8016 ?        Ssl  21:49   0:02 /usr/bin/tvheadend -f -p /run/tvheadend.pid -C -u osmc -g video
root       386  0.0  0.6  34940  4520 ?        Ssl  21:48   0:00 /usr/lib/policykit-1/polkitd --no-debug
root       382  0.0  0.5  16304  4416 ?        Ssl  21:48   0:00 /usr/lib/udisks/udisks-daemon --no-debug
root       608  0.0  0.5  10596  4248 ?        Ss   21:53   0:00 sshd: osmc [priv]
root       266  0.0  0.5   6420  3960 ?        Ss   21:48   0:00 /usr/sbin/connmand -n --nodnsproxy --config=/etc/connman.conf
root       423  0.0  0.5  17768  3884 ?        Ss   21:48   0:00 /usr/sbin/nmbd -D
root       276  0.0  0.5   6604  3780 ?        Ss   21:48   0:00 /usr/sbin/sshd -D
root         1  0.1  0.4  22012  3412 ?        Ss   21:48   0:03 /sbin/init
root       381  0.0  0.4   5412  3036 ?        Ss   21:48   0:00 /sbin/wpa_supplicant -u -s -O /run/wpa_supplicant
osmc       613  0.0  0.4  10736  3032 ?        S    21:53   0:01 sshd: osmc@pts/0
ntp        527  0.0  0.3   4808  2996 ?        Ss   21:49   0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 106:110
osmc       374  0.0  0.3   5400  2688 ?        Ss   21:48   0:00 /usr/bin/udisks-glue --foreground
osmc       610  0.0  0.3   4080  2616 ?        Ss   21:53   0:00 /lib/systemd/systemd --user
osmc       614  0.0  0.3   3144  2440 pts/0    Ss   21:53   0:00 -bash
message+   260  0.0  0.3   4776  2400 ?        Ss   21:48   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       138  0.0  0.3  10608  2392 ?        Ss   21:48   0:00 /lib/systemd/systemd-udevd
root       368  0.0  0.3   5492  2348 ?        S    21:48   0:00 sudo -u osmc /usr/lib/kodi/kodi.bin --standalone -fs --lircdev /var/run/lirc/lircd

Virtual memory use for kodi.bin is 548MB but physical memory use is only 116MB - which on a 1GB Pi 2 is about normal.

18

DBMandrake,

Thank you for your idea and I totally agree with you!

If I understand you, let the first step approximately one day “idle” with an empty “.kodi” directory and after this the RSS checking!
So - your command output very shortly:

osmc@osmc:~$ ps aux --sort -rss | head -n 20
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
osmc 277 9.1 14.2 464740 107416 ? Sl 23:14 3:06 /usr/lib/kodi/kodi.bin --standalone -fs -
-lircdev /var/run/lirc/lircd

Let’s continue from here tomorrow!

19

To make the text that you post from the ps command more readable, highlight the block of text and press the </> button in the editor - this will put it in a pre-formatted text block, like in the example I gave. It can be difficult to read the output of a command like ps with a variable width font.

20

Ok, I understand and sorry, I am not a “forum fun”…

So, very shortly the status is:

osmc@osmc:~$ uptime

 20:41:20 up 21:26,  1 user,  load average: 0,10, 0,10, 0,07

osmc@osmc:~$ ps aux --sort -rss | head -n 20

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

osmc       277  5.3 16.7 485888 126132 ?       Sl   szept02  68:38 /usr/lib/kodi/kodi.bin --standalone -fs
 --lircdev /var/run/lirc/lircd

osmc       309  0.1  3.3 248984 25384 ?        Ssl  szept02   1:47 /usr/local/bin/tvheadend -f -p /run/tvh
eadend.pid -C -u osmc -g video

root       320  0.7  1.0  73944  8128 ?        Sl   szept02   9:38 /opt/bin/mediasrv -d --pluginpath=/opt/
bin

root       299  0.0  0.6  34900  4532 ?        Ssl  szept02   0:00 /usr/lib/policykit-1/polkitd --no-debug

root       283  0.0  0.5  16392  4480 ?        Ssl  szept02   0:00 /usr/lib/udisks/udisks-daemon --no-debu
g

root      1545  0.0  0.5  10852  4456 ?        Ss   20:36   0:00 sshd: osmc [priv]    
root       295  0.0  0.5   5764  4332 ?        Ss   szept02   0:03 /sbin/wpa_supplicant -u -s -O /run/wpa_
supplicant

root       235  0.0  0.5   6512  4044 ?        Ss   szept02   0:01 /usr/sbin/connmand -n --nodnsproxy --co
nfig=/etc/connman.conf

root       242  0.0  0.5   6948  3856 ?        Ss   szept02   0:00 /usr/sbin/sshd -D

root         1  0.0  0.4  23196  3344 ?        Ss   szept02   0:05 /sbin/init

ntp        545  0.0  0.4   5292  3260 ?        Ss   szept02   0:12 /usr/sbin/ntpd -p /var/run/ntpd.pid -g 
-u 106:110

osmc      1550  0.0  0.4  10988  3128 ?        S    20:36   0:00 sshd: osmc@pts/0     
root       107  0.0  0.4   8392  3096 ?        Ss   szept02   0:02 /lib/systemd/systemd-journald

osmc      1551  0.0  0.4   5204  3088 pts/0    Ss   20:36   0:00 -bash

osmc      1547  0.0  0.3   4332  2736 ?        Ss   20:36   0:00 /lib/systemd/systemd --user

osmc       280  0.0  0.3   5580  2684 ?        Ss   szept02   0:00 /usr/bin/udisks-glue --foreground

root       135  0.0  0.3  11056  2540 ?        Ss   szept02   0:00 /lib/systemd/systemd-udevd

message+   232  0.0  0.3   4860  2488 ?        Ss   szept02   0:09 /usr/bin/dbus-daemon --system --address
=systemd: --nofork --nopidfile --systemd-activation

root       276  0.0  0.3   5748  2460 ?        S    szept02   0:00 sudo -u osmc /usr/lib/kodi/kodi.bin --s
tandalone -fs --lircdev /var/run/lirc/lircd

osmc@osmc:~$

After the “clean” boot the initial RSS value was 107416 KiB - now 126132 KiB.

DBMandrake what is your opinion: this memory consumption (~18…19 KiB / 21,5 hours) in idle state without any installed addons and modified settings Is acceptable (live up to expectations) or not?

If it is normal, what will be the next step?

Thanks in advance,

21

Seems OK to me so far. The figure of 116MB RSS that I showed on my system was taken only 1 hour after booting…

The next step will depend on what you use Kodi for - I assume you have both a library and addons - I would start by adding back your library then see how that goes.

22

Thanks and ok!

I need only the TVHeadend client plugin - nothing more (the TVHeadend server was running from the boot).

I will install it only and I am not modifying anything else in the system settings.

Do I leave the system in idle state or do I try to watch live SD TV channel on the next test period (~24 hours)?

Thanks advance,