Lockup issues since update from June

Help and Support Raspberry Pi
1

Hi,

like statet in the Topic my pi2 become very unstable after a few days of use.
i can still ping and get repsonses but all services are unrepsonsive like ssh kodi my dns etc.
only a reboot fixes it.

bevor that happens i see issues like this in the logfile

Aug  2 12:18:15 rpi2 vmunix: [219525.076075] kswapd0: page allocation failure: order:4, mode:0x40a20(GFP_ATOMIC|__GFP_COMP), nodemask=(null),cpuset=/,mems_allowed=0
Aug  2 12:18:15 rpi2 vmunix: [219525.076173] CPU: 0 PID: 70 Comm: kswapd0 Tainted: G         C O      5.15.92-1-osmc #1
Aug  2 12:18:15 rpi2 vmunix: [219525.076186] Hardware name: BCM2835
Aug  2 12:18:15 rpi2 vmunix: [219525.076194] Backtrace:
Aug  2 12:18:15 rpi2 vmunix: [219525.076206] [<80be11e8>] (dump_backtrace) from [<80be1430>] (show_stack+0x20/0x24)
Aug  2 12:18:15 rpi2 vmunix: [219525.076247]  r7:ffffe000 r6:00000000 r5:80e07480 r4:60070113
Aug  2 12:18:15 rpi2 vmunix: [219525.076253] [<80be1410>] (show_stack) from [<80be6abc>] (dump_stack_lvl+0x70/0x94)
Aug  2 12:18:15 rpi2 vmunix: [219525.076275] [<80be6a4c>] (dump_stack_lvl) from [<80be6af8>] (dump_stack+0x18/0x1c)
Aug  2 12:18:15 rpi2 vmunix: [219525.076297]  r7:ffffe000 r6:00000000 r5:80e1bb38 r4:00000000
Aug  2 12:18:15 rpi2 vmunix: [219525.076303] [<80be6ae0>] (dump_stack) from [<803871c0>] (warn_alloc+0x110/0x1b4)
Aug  2 12:18:15 rpi2 vmunix: [219525.076323] [<803870b0>] (warn_alloc) from [<80388afc>] (__alloc_pages+0x1898/0x194c)
Aug  2 12:18:15 rpi2 vmunix: [219525.076345]  r3:00000004 r2:80e1bb38
Aug  2 12:18:15 rpi2 vmunix: [219525.076355]  r7:000000c1 r6:00000004 r5:00000800 r4:00000000
Aug  2 12:18:15 rpi2 vmunix: [219525.076360] [<80387264>] (__alloc_pages) from [<7f4b6c58>] (alloc_rx_agg+0x60/0x14c [r8152])
Aug  2 12:18:15 rpi2 vmunix: [219525.076462]  r10:82920380 r9:856d3d00 r8:39932000 r7:00000a20 r6:00000004 r5:81f305c0
Aug  2 12:18:15 rpi2 vmunix: [219525.076476]  r4:b71a8fc0
Aug  2 12:18:15 rpi2 vmunix: [219525.076484] [<7f4b6bf8>] (alloc_rx_agg [r8152]) from [<7f4b8134>] (r8152_poll+0x740/0x7bc [r8152])
Aug  2 12:18:15 rpi2 vmunix: [219525.076612]  r7:81c15a84 r6:20070113 r5:81f305c8 r4:00000000
Aug  2 12:18:15 rpi2 vmunix: [219525.076619] [<7f4b79f4>] (r8152_poll [r8152]) from [<80a40888>] (__napi_poll+0x40/0x444)
Aug  2 12:18:15 rpi2 vmunix: [219525.076702]  r10:014e825d r9:00000013 r8:39932000 r7:81c15a84 r6:81f305c8 r5:0000012c
Aug  2 12:18:15 rpi2 vmunix: [219525.076710]  r4:00000100
Aug  2 12:18:15 rpi2 vmunix: [219525.076716] [<80a40848>] (__napi_poll) from [<80a41140>] (net_rx_action+0x2f8/0x364)
Aug  2 12:18:15 rpi2 vmunix: [219525.076743]  r10:014e825d r9:81203d00 r8:39932000 r7:81c15a84 r6:baa56ac0 r5:0000012c
Aug  2 12:18:15 rpi2 vmunix: [219525.076750]  r4:81f305c8
Aug  2 12:18:15 rpi2 vmunix: [219525.076755] [<80a40e48>] (net_rx_action) from [<80101640>] (__do_softirq+0x1a0/0x524)
Aug  2 12:18:15 rpi2 vmunix: [219525.076783]  r10:00000004 r9:80e85a64 r8:81c14000 r7:00000100 r6:00000003 r5:00000003
Aug  2 12:18:15 rpi2 vmunix: [219525.076791]  r4:8120308c
Aug  2 12:18:15 rpi2 vmunix: [219525.076798] [<801014a0>] (__do_softirq) from [<8012edfc>] (irq_exit+0x128/0x188)
Aug  2 12:18:15 rpi2 vmunix: [219525.076826]  r10:81c15b58 r9:81c15ba0 r8:00000008 r7:00000000 r6:81c15b38 r5:8111c2cc
Aug  2 12:18:15 rpi2 vmunix: [219525.076833]  r4:00000001
Aug  2 12:18:15 rpi2 vmunix: [219525.076839] [<8012ecd4>] (irq_exit) from [<801a18c4>] (handle_domain_irq+0x90/0xc0)
Aug  2 12:18:15 rpi2 vmunix: [219525.076864]  r7:00000000 r6:81405180 r5:00000000 r4:8112324c
Aug  2 12:18:15 rpi2 vmunix: [219525.076872] [<801a1834>] (handle_domain_irq) from [<801013e0>] (bcm2836_arm_irqchip_handle_irq+0x54/0x58)
Aug  2 12:18:15 rpi2 vmunix: [219525.076899]  r10:bab85240 r9:81c14000 r8:00000020 r7:81c15bd4 r6:ffffffff r5:60070013
Aug  2 12:18:15 rpi2 vmunix: [219525.076907]  r4:81c15ba0 r3:bb811060
Aug  2 12:18:15 rpi2 vmunix: [219525.076913] [<8010138c>] (bcm2836_arm_irqchip_handle_irq) from [<80100aec>] (__irq_svc+0x4c/0x7c)
Aug  2 12:18:15 rpi2 vmunix: [219525.076932] Exception stack(0x81c15ba0 to 0x81c15be8)
Aug  2 12:18:15 rpi2 vmunix: [219525.076947] 5ba0: 00000000 39932000 60070013 60070013 00000000 81331700 bab85244 00000018
Aug  2 12:18:15 rpi2 vmunix: [219525.076961] 5bc0: 00000020 81122e1c bab85240 81c15c5c 00000000 81c15bf0 802690dc 803808c0
Aug  2 12:18:15 rpi2 vmunix: [219525.076970] 5be0: 60070013 ffffffff
Aug  2 12:18:15 rpi2 vmunix: [219525.076979]  r5:60070013 r4:803808c0
Aug  2 12:18:15 rpi2 vmunix: [219525.076984] [<803805b8>] (free_unref_page_list) from [<80332ffc>] (shrink_page_list+0xbac/0x10f0)
Aug  2 12:18:15 rpi2 vmunix: [219525.077012]  r10:81c15ef8 r9:81c15db8 r8:81c15d74 r7:00000001 r6:00020845 r5:baf64320
Aug  2 12:18:15 rpi2 vmunix: [219525.077019]  r4:00000000
Aug  2 12:18:15 rpi2 vmunix: [219525.077025] [<80332450>] (shrink_page_list) from [<80333c00>] (shrink_lruvec+0x6c0/0xc40)
Aug  2 12:18:15 rpi2 vmunix: [219525.077047]  r10:81332200 r9:81c15ef8 r8:00000000 r7:81c15d9c r6:81332228 r5:00000078
Aug  2 12:18:15 rpi2 vmunix: [219525.077055]  r4:81122de8
Aug  2 12:18:15 rpi2 vmunix: [219525.077062] [<80333540>] (shrink_lruvec) from [<803344e8>] (shrink_node+0x368/0x8e0)
Aug  2 12:18:15 rpi2 vmunix: [219525.077085]  r10:8120593c r9:81c15ef8 r8:00000000 r7:81331700 r6:00000000 r5:81332200
Aug  2 12:18:15 rpi2 vmunix: [219525.077093]  r4:0000fccf
Aug  2 12:18:15 rpi2 vmunix: [219525.077100] [<80334180>] (shrink_node) from [<80335c94>] (kswapd+0x5f8/0xdd0)
Aug  2 12:18:15 rpi2 vmunix: [219525.077123]  r10:00000000 r9:81331700 r8:00000001 r7:81332200 r6:00000000 r5:00000001
Aug  2 12:18:15 rpi2 vmunix: [219525.077130]  r4:00000000
Aug  2 12:18:15 rpi2 vmunix: [219525.077135] [<8033569c>] (kswapd) from [<801550e8>] (kthread+0x16c/0x1a0)
Aug  2 12:18:15 rpi2 vmunix: [219525.077162]  r10:815e4420 r9:8151be34 r8:81c14000 r7:81331700 r6:8033569c r5:815e4400
Aug  2 12:18:15 rpi2 vmunix: [219525.077169]  r4:81d41ec0
Aug  2 12:18:15 rpi2 vmunix: [219525.077175] [<80154f7c>] (kthread) from [<801000d4>] (ret_from_fork+0x14/0x20)
Aug  2 12:18:15 rpi2 vmunix: [219525.077194] Exception stack(0x81c15fb0 to 0x81c15ff8)
Aug  2 12:18:15 rpi2 vmunix: [219525.077204] 5fa0:                                     00000000 00000000 00000000 00000000
Aug  2 12:18:15 rpi2 vmunix: [219525.077217] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug  2 12:18:15 rpi2 vmunix: [219525.077228] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Aug  2 12:18:15 rpi2 vmunix: [219525.077241]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80154f7c
Aug  2 12:18:15 rpi2 vmunix: [219525.077249]  r4:81d41ec0 r3:00000000
Aug  2 12:18:15 rpi2 vmunix: [219525.077323] Mem-Info:
Aug  2 12:18:15 rpi2 vmunix: [219525.077355] active_anon:37190 inactive_anon:34487 isolated_anon:0
Aug  2 12:18:15 rpi2 vmunix: [219525.077355]  active_file:44297 inactive_file:49187 isolated_file:32
Aug  2 12:18:15 rpi2 vmunix: [219525.077355]  unevictable:4 dirty:10634 writeback:525
Aug  2 12:18:15 rpi2 vmunix: [219525.077355]  slab_reclaimable:10498 slab_unreclaimable:7539
Aug  2 12:18:15 rpi2 vmunix: [219525.077355]  mapped:34646 shmem:856 pagetables:2120 bounce:0
Aug  2 12:18:15 rpi2 vmunix: [219525.077355]  kernel_misc_reclaimable:0
Aug  2 12:18:15 rpi2 vmunix: [219525.077355]  free:19937 free_pcp:739 free_cma:13467
Aug  2 12:18:15 rpi2 vmunix: [219525.077388] Node 0 active_anon:148760kB inactive_anon:137948kB active_file:177188kB inactive_file:196748kB unevictable:16kB isolated(anon):0kB isolated(file):128kB mapped:138584kB dirty:42536kB writeback:2100kB shmem:3424kB writeback_tmp:0kB kernel_stack:3616kB pagetables:8480kB all_unreclaimable? no
Aug  2 12:18:15 rpi2 vmunix: [219525.077417] DMA free:79748kB min:16384kB low:19456kB high:22528kB reserved_highatomic:16384KB active_anon:148760kB inactive_anon:137948kB active_file:177364kB inactive_file:196804kB unevictable:16kB writepending:44084kB present:970752kB managed:943768kB mlocked:16kB bounce:0kB free_pcp:2956kB local_pcp:716kB free_cma:53868kB
Aug  2 12:18:15 rpi2 vmunix: [219525.077449] lowmem_reserve[]: 0 0 0
Aug  2 12:18:15 rpi2 vmunix: [219525.077488] DMA: 7746*4kB (UMEHC) 1801*8kB (UMEHC) 439*16kB (UMEHC) 118*32kB (UMEHC) 3*64kB (C) 1*128kB (C) 0*256kB 1*512kB (C) 0*1024kB 9*2048kB (C) 1*4096kB (C) = 79552kB
Aug  2 12:18:15 rpi2 vmunix: [219525.077710] 102424 total pagecache pages
Aug  2 12:18:15 rpi2 vmunix: [219525.077720] 8076 pages in swap cache
Aug  2 12:18:15 rpi2 vmunix: [219525.077732] Swap cache stats: add 120995, delete 112919, find 216013/234928
Aug  2 12:18:15 rpi2 vmunix: [219525.077765] Free swap  = 831160kB
Aug  2 12:18:15 rpi2 vmunix: [219525.077775] Total swap = 1048572kB
Aug  2 12:18:15 rpi2 vmunix: [219525.077784] 242688 pages RAM
Aug  2 12:18:15 rpi2 vmunix: [219525.077793] 0 pages HighMem/MovableOnly
Aug  2 12:18:15 rpi2 vmunix: [219525.077804] 6746 pages reserved
Aug  2 12:18:15 rpi2 vmunix: [219525.077812] 131072 pages cma reserved

It’s not always the kswapd process i’ve also seen docker oder kodi
free -m

              gesamt       benutzt     frei      gemns.  Puffer/Cache verfügbar
Speicher:        921         401         134           3         386         470
Swap:           1023         226         797

even the new update from July few Days ago doesnt fix it.

2

Hard to say without full logs, but my guess would be that you are running out of RAM.

3

yeah but i wonder why never did until June update.
wonder if increasing values in

/etc/sysctl.d/101-osmc-device.conf
# OSMC device specific sysctl configuration
# Do not modify: define overrides in /etc/sysctl.d with a higher weighting

vm.min_free_kbytes=16384
vm.swappiness=10

Might have an effect ? What would be sane increasements ?

So far i tryd when i see this alloc error in the logs todo a
Sync and echo 3 >/proc/sys/vm/drop_caches

That seems to help for like 3 days no new alloc error and no lockups so far

4

Seems like increasing the value to
vm.min_free_kbytes=24576
did the trick no more lockups and errors

5

Are you running additional applications / services on the device?

6

yes i do that’s why i choose osmc many years ago because of ruuning aditional services.
was working fine for many years just had once some memory issues which could be fixed by adding a litle bit of swap.
but this time its was different and swap didnt helped at all.

1 Like
7

That setting will get overwritten on updates

Best to make your own file like 999-custom-settings.conf

1 Like