Kodi crash after external IP change using tailscale

RichToy · 15 September 2025 12:27

I have an RPI 3B OSMC instance in a second home that is within a network connected to the internet via a 4G modem/router.

The inderlying ISP issue a CGNAT external IP address that is annoyingly changed every 6 hours, they literally drop the connection forcing a modem reconnection every 6 hours after the modem/router is booted.

Tailscale is installed on the OSMC instance and accesses NFS shares on a NAS on my home network.

The OSMC instance is configured to rescan the NAS shares on each restart.

Tailscale gracefully deals with the 6 hourly change in external IP address, there is an interruption but the NFS mounts can be accesses within few seconds.

OSMC also initially appears ok but if something is being watched then buffering starts every so often. If the player is stopped and started the buffering remains.

Eventually, OSMC crashes and restarts whether something is being watched or not.

Sep 13 23:01:53 osmc-Bexhill kernel: ------------[ cut here ]------------
Sep 13 23:01:53 osmc-Bexhill kernel: WARNING: CPU: 1 PID: 501 at lib/refcount.c:87 refcount_dec_not_one+0xb4/0xc4
Sep 13 23:01:53 osmc-Bexhill kernel: refcount_t: underflow; use-after-free.
Sep 13 23:01:53 osmc-Bexhill kernel: Modules linked in: rpcsec_gss_krb5 xt_MASQUERADE xt_tcpudp xt_mark binfmt_misc tun nf_tables nfnetlink ip6table_nat ip6table_filter ip6_tables cmac 8021q garp stp algif_hash llc aes_arm_bs crypto_simd cryptd algif_skcipher af_alg bnep iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_filter vc4 snd_soc_hdmi_codec hci_uart cec drm_kms_helper btbcm brcmfmac bluetooth snd_soc_core snd_compress snd_pcm_dmaengine snd_pcm bcm2835_codec(C) snd_timer ecdh_generic ecc brcmutil snd bcm2835_v4l2(C) v4l2_mem2mem bcm2835_isp(C) bcm2835_mmal_vchiq(C) syscopyarea videobuf2_dma_contig sysfillrect videobuf2_vmalloc sysimgblt fb_sys_fops sha256_generic videobuf2_memops videobuf2_v4l2 videobuf2_common cfg80211 videodev raspberrypi_hwmon rfkill i2c_bcm2835 mc vc_sm_cma(C) fixed uio_pdrv_genirq uio drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
Sep 13 23:01:53 osmc-Bexhill kernel: CPU: 1 PID: 501 Comm: kodi.bin Tainted: G         C        5.15.92-1-osmc #1
Sep 13 23:01:53 osmc-Bexhill kernel: Hardware name: BCM2835
Sep 13 23:01:53 osmc-Bexhill kernel: Backtrace: 
Sep 13 23:01:53 osmc-Bexhill kernel: [<80be11e8>] (dump_backtrace) from [<80be1430>] (show_stack+0x20/0x24)
Sep 13 23:01:53 osmc-Bexhill kernel:  r7:00000057 r6:00000000 r5:80e07480 r4:60010013
Sep 13 23:01:53 osmc-Bexhill kernel: [<80be1410>] (show_stack) from [<80be6abc>] (dump_stack_lvl+0x70/0x94)
Sep 13 23:01:53 osmc-Bexhill kernel: [<80be6a4c>] (dump_stack_lvl) from [<80be6af8>] (dump_stack+0x18/0x1c)
Sep 13 23:01:53 osmc-Bexhill kernel:  r7:00000057 r6:00000009 r5:80745fbc r4:80e44a24
Sep 13 23:01:53 osmc-Bexhill kernel: [<80be6ae0>] (dump_stack) from [<801276a8>] (__warn+0x98/0x12c)
Sep 13 23:01:53 osmc-Bexhill kernel: [<80127610>] (__warn) from [<80be1a2c>] (warn_slowpath_fmt+0xa4/0xe4)
Sep 13 23:01:53 osmc-Bexhill kernel:  r7:80745fbc r6:00000057 r5:80e44a24 r4:80e44a60
Sep 13 23:01:53 osmc-Bexhill kernel: [<80be198c>] (warn_slowpath_fmt) from [<80745fbc>] (refcount_dec_not_one+0xb4/0xc4)
Sep 13 23:01:53 osmc-Bexhill kernel:  r8:83424040 r7:83424040 r6:7f564114 r5:835b2710 r4:00000001
Sep 13 23:01:53 osmc-Bexhill kernel: [<80745f08>] (refcount_dec_not_one) from [<7f52f664>] (vc4_bo_dec_usecnt+0x34/0xe0 [vc4])
Sep 13 23:01:53 osmc-Bexhill kernel:  r5:835b2710 r4:835b2600
Sep 13 23:01:53 osmc-Bexhill kernel: [<7f52f630>] (vc4_bo_dec_usecnt [vc4]) from [<7f55661c>] (vc4_cleanup_fb+0x3c/0x40 [vc4])
Sep 13 23:01:53 osmc-Bexhill kernel:  r7:83424040 r6:7f564114 r5:86d70980 r4:00000004
Sep 13 23:01:53 osmc-Bexhill kernel: [<7f5565e0>] (vc4_cleanup_fb [vc4]) from [<7f669144>] (drm_atomic_helper_cleanup_planes+0x6c/0x80 [drm_kms_helper])
Sep 13 23:01:53 osmc-Bexhill kernel: [<7f6690d8>] (drm_atomic_helper_cleanup_planes [drm_kms_helper]) from [<7f53b358>] (vc4_atomic_commit_tail+0x420/0x790 [vc4])
Sep 13 23:01:53 osmc-Bexhill kernel:  r7:83424040 r6:00000000 r5:8438c000 r4:86d70980
Sep 13 23:01:53 osmc-Bexhill kernel: [<7f53af38>] (vc4_atomic_commit_tail [vc4]) from [<7f66d104>] (commit_tail+0xac/0x190 [drm_kms_helper])
Sep 13 23:01:53 osmc-Bexhill kernel:  r10:86d70980 r9:00000000 r8:7f56d224 r7:00001cb4 r6:642e2d2a r5:00000000
Sep 13 23:01:53 osmc-Bexhill kernel:  r4:86d70980
Sep 13 23:01:53 osmc-Bexhill kernel: [<7f66d058>] (commit_tail [drm_kms_helper]) from [<7f66dc1c>] (drm_atomic_helper_commit+0x1f4/0x218 [drm_kms_helper])
Sep 13 23:01:53 osmc-Bexhill kernel:  r9:00000000 r8:8438c000 r7:00000000 r6:00000000 r5:00000000 r4:86d70980
Sep 13 23:01:53 osmc-Bexhill kernel: [<7f66da28>] (drm_atomic_helper_commit [drm_kms_helper]) from [<7f108680>] (drm_atomic_commit+0x54/0x60 [drm])
Sep 13 23:01:53 osmc-Bexhill kernel:  r9:00000000 r8:00000000 r7:00000004 r6:8438c000 r5:86d70980 r4:00000000
Sep 13 23:01:53 osmc-Bexhill kernel: [<7f10862c>] (drm_atomic_commit [drm]) from [<7f12b294>] (drm_mode_atomic_ioctl+0x8f4/0xab0 [drm])
Sep 13 23:01:53 osmc-Bexhill kernel:  r7:00000004 r6:81dc7040 r5:8818be3c r4:00000000
Sep 13 23:01:53 osmc-Bexhill kernel: [<7f12a9a0>] (drm_mode_atomic_ioctl [drm]) from [<7f0ee4ac>] (drm_ioctl_kernel+0xcc/0x174 [drm])
Sep 13 23:01:53 osmc-Bexhill kernel:  r10:00000038 r9:828f6a00 r8:828f6a00 r7:8818be3c r6:7f12a9a0 r5:8438c000
Sep 13 23:01:53 osmc-Bexhill kernel:  r4:00000002
Sep 13 23:01:53 osmc-Bexhill kernel: [<7f0ee3e0>] (drm_ioctl_kernel [drm]) from [<7f0ee740>] (drm_ioctl+0x1ec/0x3a8 [drm])
Sep 13 23:01:53 osmc-Bexhill kernel:  r8:000000bc r7:8818be3c r6:7f135d04 r5:00000038 r4:c03864bc
Sep 13 23:01:53 osmc-Bexhill kernel: [<7f0ee554>] (drm_ioctl [drm]) from [<803e168c>] (sys_ioctl+0x118/0xc0c)
Sep 13 23:01:53 osmc-Bexhill kernel:  r10:82bdc408 r9:00000000 r8:826f9840 r7:7ea8baa8 r6:826f9841 r5:00000000
Sep 13 23:01:53 osmc-Bexhill kernel:  r4:c03864bc
Sep 13 23:01:53 osmc-Bexhill kernel: [<803e1574>] (sys_ioctl) from [<80100040>] (ret_fast_syscall+0x0/0x1c)
Sep 13 23:01:53 osmc-Bexhill kernel: Exception stack(0x8818bfa8 to 0x8818bff0)
Sep 13 23:01:53 osmc-Bexhill kernel: bfa0:                   07c02e48 7ea8baa8 00000000 c03864bc 7ea8baa8 00000000
Sep 13 23:01:53 osmc-Bexhill kernel: bfc0: 07c02e48 7ea8baa8 c03864bc 00000036 7ea8baa8 0755dfd0 080cfd54 00000000
Sep 13 23:01:53 osmc-Bexhill kernel: bfe0: 75ee7090 7ea8ba7c 75ecef17 759d9418
Sep 13 23:01:53 osmc-Bexhill kernel:  r10:00000036 r9:8818a000 r8:80100244 r7:00000036 r6:c03864bc r5:7ea8baa8
Sep 13 23:01:53 osmc-Bexhill kernel:  r4:07c02e48
Sep 13 23:01:53 osmc-Bexhill kernel: ---[ end trace 94498879b00c5cab ]---

Any hints on how to proceed?

fzinken · 15 September 2025 15:06

Providing full debug enabled logs via OSMC uploader.

RichToy · 16 September 2025 21:39

Capturing logs from My OSMC fails because the log file is too big.

Same from the command line

osmc@osmc-Bexhill:/boot$ grab-logs -A
Grabbing log UNAME ...
Grabbing log cmdline ...
Grabbing log Debian version ...
Grabbing log OSMC Build Information ...
Grabbing log Pi config ...
Grabbing log Pi config-user ...
Grabbing log GUI Settings (abridged) ...
Grabbing log guisettings.xml ...
Masking private information ...
Grabbing log advancedsettings.xml ...
An error occurred while grabbing advancedsettings.xml:
 FileNotFoundError: [Errno 2] No such file or directory: '/home/osmc/.kodi/userdata/advancedsettings.xml'
Grabbing log sources.xml ...
Masking private information ...
Grabbing log fstab ...
Masking private information ...
Grabbing log mounts ...
Grabbing log OSMC Packages ...
Grabbing log All Other Packages ...
Grabbing log APT term.log ...
Grabbing log APT history.log ...
Grabbing log APT sources.list ...
Grabbing log APT apt.conf.d ...
Grabbing log APT preferences.d ...
Grabbing log APT sources.list.d ...
Grabbing log System Journal ...
Grabbing log lircd.conf ...
Grabbing log init.d ...
Grabbing log systemd ...
Grabbing log Kernel Message Log ...
Grabbing log Memory ...
Grabbing log Diskspace ...
Grabbing log /boot Contents ...
Grabbing log edid-decode ...
/sys/class/drm/card1-HDMI-A-1/edid: No such file or directory
An error occurred while grabbing edid-decode:
 subprocess.CalledProcessError: Command '['/usr/bin/edid-decode', '/sys/class/drm/card1-HDMI-A-1/edid']' returned non-zero exit status 255.
Grabbing log ifconfig ...
Grabbing log Kodi Log ...
Masking private information ...
Grabbing log Kodi Old Log ...
Masking private information ...
Writing logs to temp file ...
Dispatching logs ...
Exception Details:

Traceback (most recent call last):
  File "/usr/bin/grab-logs", line 1059, in dispatch_logs
    raise Exception('Log file too large for upload')
Exception: Log file too large for upload

Failed to upload log files, copying to /boot instead. (Unable to verify)
osmc@osmc-Bexhill:/boot$ ls -ltr ./uploadlog.txt
-rwxr-xr-x 1 root root 13350504 Sep 16 22:34 ./uploadlog.txt
osmc@osmc-Bexhill:/boot$ paste-log /boot/uploadlog.txt
Unable to upload log. Log file is too large. (13MB)

I compressed it and uploaded
https://paste.osmc.tv/unusokojoq

Is this OK?

fzinken · 17 September 2025 00:43

No, not working that way.
Anyway if the logs are too big means either the machine has run too long or something massively is spamming your logs.

Suggest to check if something is spamming your logs
Reboot twice and then try to reproduce the crash with shorter time frame

RichToy · 17 September 2025 22:04

This morning I examined the logs and found thousands of entries like these that mysteriously appear to start after the extermal ip address has changed

2025-09-16 09:19:00.916 T:726     debug <CWebserver[80]>: request received for /jsonrpc
2025-09-16 09:19:10.918 T:726     debug <CWebserver[80]>: request received for /jsonrpc
2025-09-16 09:19:20.919 T:726     debug <CWebserver[80]>: request received for /jsonrpc


2025-09-16 20:34:18.590 T:19619   debug <UPNP::BuildObject>: Building didl for plain object 'musicdb://songs/10117.m4a' (encoded value: 'bXVzaWNkYjovL3NvbmdzLzEwMTE3Lm00YQ==')
2025-09-16 20:34:18.665 T:19619   debug <CUPnPServer[Kodi (osmc-Bexhill)]>: Preparing upnp object for item 'musicdb://songs/10118.m4a'

However, I know of nothing on my network using UPNP or the HTTP server so I turned them both and the Airplay server off and rebooted and the problem has not reoccurred. Furthermore, if something is being played over tailscale when the external address changes then there is a pause and single bit of buffering but the player resumes gracefully

I therefore assume the issue is caused by one of these unused services UPNP or HTTP.

I will try and find time over the weekend so try to pin the issue down futher.

sam_nazarko · 19 September 2025 19:30

Keep us posted.

RichToy · 22 September 2025 06:59

It looks like the spamming of the osmc browser was coming from the “smart” TV!

The TV is an 18 month old LG OLED48C36LA that contains an astonishing amount of functionality that is nothing to do with watching TV. It contains all sorts of home automation stuff and gets frequent updates.

When I first set it up it discovered the open http interface for osmc and I must have added the credentials.

The TV is supposedly configured to deeply turn off but it would appear stuff is still going on when it is off, in the background, to build some kind of database by interrogating the osmc instance. It looks like this slowly kills the osmc process by causing it to just run out of CPU.

It takes hours before something happens which makes detailed debugging difficult.

We leave our second home this week until next year so I doubt I will get chance to investigate this issue further. It does appear solved for me by turning off the http and upnp services.