Vsftpd causes insane load and crash

Help and Support
1

Reopening the former issue from https://discourse.osmc.tv/t/high-load-low-cpu-usage-and-kernel-errors/71571.

After trying several ways to use the vsftpd built-in server I can’t resolve the following: When connecting to my pi via the ftp server and uploading/downloading files the load goes through the roof and causing a system crash after a while.

I tried the following:

  • Disable via the OSMC menu and remove the vsftpd server via apt-get purge and reinstalling the server
  • As above but then installing via apt-get install vsftpd

My pi has an excellent power supply, runs samba+pihole+transmission and KODI without any issues. Disabling those services has no influence.

System logs are here: https://paste.osmc.tv/xinuxaduse

piece of the problem lies here I think, I notice a kernel oops

6057.959559] Process vsftpd (pid: 15678, stack limit = 0xae3fa210)
[16057.965664] Stack: (0xae3fbc38 to 0xae3fc000)
[16057.970035] bc20: 000005a0 a8e53dbc
[16057.978239] bc40: 00000080 a8e53dbc 00000000 a4cc3800 ae3fbd04 ae3fbc60 8088c08c 80887a08
[16057.986439] bc60: 000005a0 00000e9a ae3fbcd4 81003024 00004002 00000040 38e38e39 ae3fa000
[16057.994636] bc80: 00010000 00000000 00000000 3ad3e379 00000000 00000000 adec4a88 ade1ee10
[16058.002834] bca0: ae3fbd04 ae3fbcb0 00000000 00000000 00028000 00000520 00000000 000005a0
[16058.011031] bcc0: 000005a0 00000080 7fffffff 481ae3cb 5ad36522 00000000 3ad3e379 ae3fbd40
[16058.019233] bce0: a8e53cc0 9faa2700 00028000 a8e53cc0 00000000 ae054780 ae3fbd2c ae3fbd08
[16058.027434] bd00: 807f2f6c 8088b5dc 00001000 000753a0 014000c0 af6ea2e0 00001000 00000000
[16058.035631] bd20: ae3fbd94 ae3fbd30 807f8518 807f2f18 00001000 ae3fbd40 a88f8000 00001000
[16058.043829] bd40: 00000000 00000000 00000003 00000080 00000f80 ae3fbd38 00000001 00000000
[16058.052026] bd60: 00000000 00000000 00028000 00000000 a8e53cc0 af6ea2e0 00001000 00000000
[16058.060230] bd80: 00000000 00001000 ae3fbdbc ae3fbd98 8088b568 807f848c 00028000 3ad3e379
[16058.068434] bda0: a8e53cc0 8088b4e0 9faa2700 af6ea2e0 ae3fbdec ae3fbdc0 808c1804 8088b4ec
[16058.076632] bdc0: 00028000 00002000 808c1790 ade66af0 00001000 00006000 807f38d4 000000f0
[16058.084829] bde0: ae3fbe0c ae3fbdf0 807f3924 808c179c 00028000 8035db14 abc0bb00 abc0bb00
[16058.093026] be00: ae3fbe74 ae3fbe10 8035db7c 807f38e0 ae3fbe40 00028000 ae3fa000 00000000
[16058.101222] be20: 00028000 00000000 00000000 00008000 ade66a00 0000a000 0000a000 00000000
[16058.109426] be40: 0000a000 00000000 00000000 abc0bb00 753afdac 8035d904 adec4a80 ae054780
[16058.117624] be60: 7ffff000 00000004 ae3fbedc ae3fbe78 8035ea30 8035d910 00000004 805a3364
[16058.125821] be80: ae3fbf14 00000000 ae3fbf00 8035d190 8035dd34 ae3fbf08 00010000 0ac4f254
[16058.134017] bea0: 0dd19000 00000000 0dd29000 00000000 00000000 adec4a80 adec4a80 9faa2720
[16058.142214] bec0: ae3fbf08 fffff000 00000fff ae054780 ae3fbf3c ae3fbee0 803123dc 8035e898
[16058.150417] bee0: 7ffff000 00000000 5ac8a5a9 2c6f9edc 4ddd738b 7ffff000 ae054780 00000000
[16058.158619] bf00: 030c9dac 00000000 00000000 00000000 8031cc80 00000000 00000000 00000000
[16058.166820] bf20: 000000bb 7fffffff 00000006 00000000 ae3fbfa4 ae3fbf40 80314d18 80312218
[16058.175017] bf40: fffff000 00000fff 030c9dac 7e9678a8 00000008 00000000 7e9678a8 805a380c
[16058.183214] bf60: ae3fbfa4 ae3fbf70 80311d18 805a37f0 00000000 00000000 030c9dac 42a0b8c5
[16058.191409] bf80: 7fffffff 00000006 000000bb 8010c7a8 ae3fa000 00000000 00000000 ae3fbfa8
[16058.199613] bfa0: 8010c5c0 80314bc4 42a0b8c5 7fffffff 00000006 00000004 00000000 7fffffff
[16058.207812] bfc0: 42a0b8c5 7fffffff 00000006 000000bb 00000004 0047ced0 0047ced0 00000004
[16058.216010] bfe0: 0047bc48 7e9678cc 004663d7 76c8e966 20030030 00000006 2f7fa861 2f7fac61
[16058.224227] [<80887a5c>] (tcp_push) from [<8088c08c>] (tcp_sendmsg_locked+0xabc/0x1038)
[16058.232261] [<8088c08c>] (tcp_sendmsg_locked) from [<807f2f6c>] (kernel_sendmsg_locked+0x60/0x78)
[16058.241175] [<807f2f6c>] (kernel_sendmsg_locked) from [<807f8518>] (sock_no_sendpage_locked+0x98/0xa0)
[16058.250520] [<807f8518>] (sock_no_sendpage_locked) from [<8088b568>] (tcp_sendpage+0x88/0xf0)
[16058.259073] [<8088b568>] (tcp_sendpage) from [<808c1804>] (inet_sendpage+0x74/0x158)
[16058.266840] [<808c1804>] (inet_sendpage) from [<807f3924>] (sock_sendpage+0x50/0x64)
[16058.274607] [<807f3924>] (sock_sendpage) from [<8035db7c>] (generic_splice_sendpage+0x278/0x430)
[16058.283418] [<8035db7c>] (generic_splice_sendpage) from [<8035ea30>] (do_splice_direct+0x1a4/0x370)
[16058.292500] [<8035ea30>] (do_splice_direct) from [<803123dc>] (do_sendfile+0x1d0/0x480)
[16058.300534] [<803123dc>] (do_sendfile) from [<80314d18>] (SyS_sendfile+0x160/0x4b8)
[16058.308223] [<80314d18>] (SyS_sendfile) from [<8010c5c0>] (ret_fast_syscall+0x0/0x28)
[16058.316079] Code: e5840514 e3110001 158404a4 e59b0004 (e595105c)
[16058.322548] —[ end trace 2f8723f1c26b4df9 ]—
[98138.949205] Unable to handle kernel NULL pointer dereference at virtual address 0000005c
[98138.957508] pgd = abd5c000
[98138.961985] [0000005c] *pgd=00000000
[98138.965645] Internal error: Oops: 5 [#5] PREEMPT SMP ARM
[98138.971005] Modules linked in: xfs 8021q garp stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_filter snd_soc_wm8804_i2c snd_soc_wm8804 sg snd_soc_hifiberry_digi snd_soc_bcm2835_i2s snd_soc_core bcm2835_thermal snd_compress snd_pcm_dmaengine snd_pcm snd_timer bcm2835_gpiomem snd i2c_bcm2835 uio_pdrv_genirq fixed uio ip_tables x_tables ipv6
[98139.006547] CPU: 3 PID: 27855 Comm: vsftpd Tainted: G D 4.14.26-2-osmc #1
[98139.014507] Hardware name: BCM2835

2

Raising minimum free kbytes may help

3

Thanks for your fast reply, but what command is it?

4

Pihole can cause problems as it messes up the connman installation. If raising the memory limit as Sam suggested doesn’t work, I’d recommend removing Pihole.

5

I think Sam, in his usual enigmatic way, might be suggesting that you increase vm.min_free_kbytes. If so, using sudo, edit the file /etc/sysctl.conf and add this line to the end:

vm.min_free_kbytes = 32768

(It’s 16384 on my Pi, so you’d be doubling it.) Then reboot.

Just to check it’s taken, run:

sysctl vm.min_free_kbytes
6

Okay, it took me sometime to look into it:

The suggestion of vm.min_free_kbytes = 32768 didn’t work. I double checked it, but still got crashes.

I did a fresh reinstall of the 2018.03-2 and installed the ftp via the menu on top of it. Again kernel panics.

Reverted to 2017.10-1 build and everything is fine. So there must be a bug somewhere in OSMC

7

As already mentioned, you should now consider removing pihole because of incompatibility issues with OSMC. It seems not to have installed correctly, anyway:

Errors were encountered while processing:
 php7.0-common
 php7.0-readline
 php7.0-opcache
 php7.0-cgi
 php-cgi
 php7.0-cli
 php7.0-sqlite3
 php7.0-json
 php-sqlite3
8

@dillthedog as mentioned in my post I did a fresh reinstall of OSMC and didn’t install pihole on it.

For clarity: the vsftpd server causes kernel panic in the 2018.3-2 build. I did another fresh reinstall with 2017.10-1 where the vsftpd server functions without any issues.

9

I see. I’ll see if I can reproduce it. Does it simply panic without any help or do you do something that causes it to panic?

10

@dillthedog I see a kernel panic when downloading from the ftp server. Not sure if uploading caused it too.

11

Perhaps, there are some helpful information also logged by vsftpd before the crash, so give the vsftpd logging a try. I used it long time ago, look at this post.

12

I’m currently unable to cause a crash. On build 2018.03-2, which has not yet been updated to 2018.04, I installed vsftpd:

osmc@osmc:~$ sudo apt-get install vsftpd
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Recommended packages:
  logrotate ssl-cert
The following NEW packages will be installed:
  vsftpd
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 143 kB of archives.
After this operation, 298 kB of additional disk space will be used.
Get:1 http://ftp.debian.org/debian stretch/main armhf vsftpd armhf 3.0.3-8+b1 [143 kB]
Fetched 143 kB in 1s (81.8 kB/s)                         
Preconfiguring packages ...
Selecting previously unselected package vsftpd.
(Reading database ... 62195 files and directories currently installed.)
Preparing to unpack .../vsftpd_3.0.3-8+b1_armhf.deb ...
Unpacking vsftpd (3.0.3-8+b1) ...
Setting up vsftpd (3.0.3-8+b1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/vsftpd.service → /lib/systemd/system/vsftpd.service.
Processing triggers for systemd (232-25+deb9u3) ...
Processing triggers for man-db (2.7.6.1-2) ...

I have made no changes to any vsftpd configuration file.

On a separate system:

[user@fedora-26-dvm ~]$ ftp 192.168.8.33
Connected to 192.168.8.33 (192.168.8.33).
220 (vsFTPd 3.0.3)
Name (192.168.8.33:user): osmc
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
227 Entering Passive Mode (192,168,8,33,191,67).
150 Here comes the directory listing.
drwxr-xr-x    2 1000     1000         4096 Mar 16 03:36 Movies
drwxr-xr-x    2 1000     1000         4096 Apr 06 19:17 Music
drwxr-xr-x    2 1000     1000         4096 Mar 16 03:36 Pictures
drwxr-xr-x    2 1000     1000         4096 Mar 16 03:36 TV Shows
drwxr-xr-x    5 1000     1000         4096 Apr 08 06:35 a2dp
-rw-r--r--    1 1000     1000       110782 Apr 08 06:35 a2dp-app-osmc_1.1.3_all.deb
-rwxr-xr-x    1 1000     1000         9732 Apr 30 10:40 apt-real
-rw-r--r--    1 1000     1000          143 Apr 09 14:00 asoundrc-bluealsa
-rw-r--r--    1 1000     1000        47432 Apr 10 19:39 bluealsa_0.7_armhf.deb
226 Directory send OK.
ftp> get a2dp-app-osmc_1.1.3_all.deb
local: a2dp-app-osmc_1.1.3_all.deb remote: a2dp-app-osmc_1.1.3_all.deb
227 Entering Passive Mode (192,168,8,33,43,83).
150 Opening BINARY mode data connection for a2dp-app-osmc_1.1.3_all.deb (110782 bytes).
226 Transfer complete.
110782 bytes received in 0.0601 secs (1844.52 Kbytes/sec)
ftp> quit
221 Goodbye.

No high CPU or kernel panic and the daemon is running just fine:

osmc@osmc:~$ systemctl status vsftpd
● vsftpd.service - vsftpd FTP server
   Loaded: loaded (/lib/systemd/system/vsftpd.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-04-30 21:14:08 BST; 6min ago
 Main PID: 1373 (vsftpd)
   CGroup: /system.slice/vsftpd.service
           ├─1373 /usr/sbin/vsftpd /etc/vsftpd.conf
           ├─1601 /usr/sbin/vsftpd /etc/vsftpd.conf
           └─1603 /usr/sbin/vsftpd /etc/vsftpd.conf

Apr 30 21:14:08 osmc systemd[1]: Starting vsftpd FTP server...
Apr 30 21:14:08 osmc systemd[1]: Started vsftpd FTP server.

After a reboot, it’s still working without any problems.

Just for completeness, I then installed the package ftp-app-osmc and rebooted but it’s still working correctly.

osmc@osmc:~$ sudo apt-get install ftp-app-osmc
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  ftp-app-osmc
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 1,668 B of archives.
After this operation, 31.7 kB of additional disk space will be used.
Get:1 http://ftp.fau.de/osmc/osmc/apt stretch/main armhf ftp-app-osmc all 1.1.4 [1,668 B]
Fetched 1,668 B in 1s (906 B/s)               
Selecting previously unselected package ftp-app-osmc.
(Reading database ... 62254 files and directories currently installed.)
Preparing to unpack .../ftp-app-osmc_1.1.4_all.deb ...
Adding 'diversion of /etc/vsftpd.conf to /etc/vsftpd.conf.distributed by ftp-app-osmc'
Unpacking ftp-app-osmc (1.1.4) ...
Setting up ftp-app-osmc (1.1.4) ...

You mention that you’ve you’ve done a fresh install of OSMC. Did you try vsftpd before adding the other packages?