[SOLVED] April 2019 update error - failed to allocate memory

ransur0t · 23 April 2019 00:34

When attempting to update OSMC to the April 2019 build, the OSMC updater fails with an unknown error.

When updating via cli, dpkg returns the following:

dpkg: unrecoverable fatal error, aborting:
 failed to allocate memory: Cannot allocate memory
E: Sub-process /usr/bin/dpkg returned an error code (2)

Logs available at https://paste.osmc.tv/reqojefitu
Your analysis is greatly appreciated.

Thanks

bmillham · 23 April 2019 00:40

First thing will be to get rid of the banned addons. Clean up your sources.xml, reboot and try again.

ransur0t · 23 April 2019 05:04

Done, first post edited with new paste url. The problem persists after the cleanup and removal. It is readily visible in the apt section of the logs … APT term.log and APT history.log.

Thanks

fzinken · 23 April 2019 05:48

Your first error was

Log started: 2019-04-22 07:00:47 unable to open files list file for package 'libxcb-xinerama0:armhf': Structure needs cleaning Log ended: 2019-04-22 07:00:47

So my gut feel is a failing SD Card.

ransur0t · 23 April 2019 13:44

I have the same gut feeling … I suppose I’ll pull the SD card and run a fsck using my laptop to see if it’s recoverable.

fzinken · 23 April 2019 13:45

Well more important write a file with a known md5sum to it. Reboot and check the files md5sum afterwards.

ransur0t · 23 April 2019 19:34

Wrote a simple text file to /home and compared md5sum before and after reboot - they match …

ransur0t · 25 April 2019 04:03

Pulled the board from its case, pulled the sdcard, inserted into an adapter and slid it into my XPS13’s sdcard slot. After a quick umount, running fsck revealed the issue:

sudo fsck /dev/mmcblk0p1
fsck from util-linux 2.29.2
fsck.fat 4.1 (2017-01-24)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
1) Remove dirty bit
2) No action
? 1
Perform changes ? (y/n) y
/dev/mmcblk0p1: 519 files, 71687/489976 clusters

Unfortunately, with the dirty bit removed and a clean fsck, a system upgrade was unsuccessful.

Not sure where the problem lies, gotta dig deeper …

Cheers

dillthedog · 25 April 2019 06:56

The SD card will have two partitions: the DOS-format boot partition on mmcblk0p1, and the ext4-format root partition on mmcblk0p2.

You should be performing the fsck on the root partition, mmcblk0p2.

The dirty-bit error is normal on OSMC for partition mmcblk0p1 and can safely be ignored.

ransur0t · 25 April 2019 13:24

Indeed, that is the first scan I ran and the mmcblk0p2 partition is clean …

Thanks

ransur0t · 27 April 2019 23:01

Update – I’ve also tried, without success, the following:

echo 3 > /proc/sys/vm/drop_caches
killing processes that consume large amounts of RAM (kodi and tvheadend)
adjusting gpu_ram in config.txt to a much lower value

So far, no luck!

ransur0t · 3 May 2019 15:18

More troubleshooting:

strace -f -o /tmp/log apt install -f rsync
cat /tmp/log | grep ENOMEM
30818 mmap2(NULL, 1073750016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
30818 mmap2(NULL, 1073881088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
30818 mmap2(NULL, 1073750016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)

dillthedog · 3 May 2019 17:04

I congratulate you on your persistence, though I have a feeling that you might be focusing on the wrong area – by which I mean that the apparent memory problem could be the result of an issue elsewhere.

If you recall, before you saw the failed to allocate memory: Cannot allocate memory error messages, your log showed the following errors:

unable to open files list file for package 'libxcb-xinerama0:armhf': Structure needs cleaning

but I understand that running fsck on /dev/mmcblk0p2 didn’t fix anything.

It might be worth investigating the file that originally couldn’t be opened, so please run the following commands:

ls -l /var/lib/dpkg/info/libxcb-xinerama0:armhf*
cat /var/lib/dpkg/info/libxcb-xinerama0:armhf.list

ransur0t · 3 May 2019 17:32

Thanks! Looking at the log previously referenced shows a repeating pattern of the unable to open file issues. I’m not in the position at the moment to post additional log details but will do so later for analysis.

Kind regards.

ransur0t · 3 May 2019 20:50

:~$ ls -l /var/lib/dpkg/info/libxcb-xinerama0:armhf*
-rw-r--r-- 1 57371 root 44023418880 Aug 24  2006 /var/lib/dpkg/info/libxcb-xinerama0:armhf.list
-rw-r--r-- 1 root  root         325 Sep 24  2016 /var/lib/dpkg/info/libxcb-xinerama0:armhf.md5sums
-rw-r--r-- 1 root  root          45 Sep 24  2016 /var/lib/dpkg/info/libxcb-xinerama0:armhf.shlibs
-rw-r--r-- 1 root  root        1064 Sep 24  2016 /var/lib/dpkg/info/libxcb-xinerama0:armhf.symbols
-rw-r--r-- 1 root  root          60 Sep 24  2016 /var/lib/dpkg/info/libxcb-xinerama0:armhf.triggers

The cat command just hangs, no output returned …

dillthedog · 3 May 2019 21:27

Yep, the file is broken. It’s allegedly 44GB in size, which is clearly going to bust the Pi’s RAM, and owned by a non-existent user id. The correct contents of the file you can see here.

You need to try to delete the file: sudo rm -f /var/lib/dpkg/info/libxcb-xinerama0:armhf.list

and then reinstall the package: sudo apt-get install --reinstall libxcb-xinerama0

ransur0t · 3 May 2019 21:39

Deletion of the file was fine, but I encounter an error when attempting to reinstall:

~$ sudo apt-get install --reinstall libxcb-xinerama0
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 15 not upgraded.
Need to get 0 B/95.5 kB of archives.
After this operation, 0 B of additional disk space will be used.
dpkg: warning: files list file for package 'libxcb-xinerama0:armhf' missing; assuming package has no files currently installed
dpkg: unrecoverable fatal error, aborting:
 reading files list for package 'python-ptyprocess': Bad address
E: Sub-process /usr/bin/dpkg returned an error code (2)

dillthedog · 3 May 2019 21:48

I suspect something similar for this package. You can repeat the same process for the python-ptyprocess package and hope it’s the only other one. If not, you’ll get a rough idea of the scale of the corruption by running

file /var/lib/dpkg/info/* | grep data$

bmillham · 3 May 2019 21:49

I think you’ve wasted enough time on this, get a new SD card, your current one is clearly having issues!

ransur0t · 3 May 2019 21:55

Well, once I chased down the rabbit hole, I pursue until, well, you know …

~$ file /var/lib/dpkg/info/* | grep data$
/var/lib/dpkg/info/armv7-irqbalance-osmc.list:                         data
/var/lib/dpkg/info/libsgutils2-2.md5sums:                              data
/var/lib/dpkg/info/libwebp5:armhf.postrm:                              data
/var/lib/dpkg/info/libwebpdemux1:armhf.list:                           data
/var/lib/dpkg/info/libxmuu1:armhf.md5sums:                             data
/var/lib/dpkg/info/libyajl2:armhf.md5sums:                             data
/var/lib/dpkg/info/login.list:                                         data
/var/lib/dpkg/info/python-ptyprocess.list:                             data