Some services failing to start at boot

Help and Support Raspberry Pi
1

I’ve just migrated to OSMC from Raspbmc and am having a problem with systemd not starting a number of services (MySQL, OpenVPN, Samba and NFS) at boot time.

Most problematic of these are MySQL server and OpenVPN. The former as it prevents me seeing my library and the latter as it stops me fixing the former from outside of the house.

Looking at MySQL I see “[FAILED]” during the boot process and that tells me to run “systemctl status mysql.service” for details. When I run that I see the following:

osmc@osmc:~$ sudo systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: failed (Result: exit-code) since Mon 2015-08-24 10:24:13 BST; 1min 39s ago
  Process: 390 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)

Aug 24 10:24:13 osmc /etc/init.d/mysql[1236]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Aug 24 10:24:13 osmc mysql[390]: Starting MySQL database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
Aug 24 10:24:13 osmc systemd[1]: mysql.service: control process exited, code=exited status=1
Aug 24 10:24:13 osmc systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
Aug 24 10:24:13 osmc systemd[1]: Unit mysql.service entered failed state.
osmc@osmc:~$

which seems to tell me absolutely nothing except that it failed.

However when I start it manually, it works fine…

osmc@osmc:~$ sudo /etc/init.d/mysql start
[ ok ] Starting mysql (via systemctl): mysql.service.
osmc@osmc:~$ sudo systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: active (running) since Mon 2015-08-24 10:53:33 BST; 37s ago
  Process: 1355 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/mysql.service
	       ├─1383 /bin/sh /usr/bin/mysqld_safe
	       └─1752 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socke...

Aug 24 10:53:33 osmc mysql[1355]: Starting MySQL database server: mysqld . . ..
Aug 24 10:53:33 osmc mysql[1355]: Checking for tables which need an upgrade, are corrupt or were
Aug 24 10:53:33 osmc mysql[1355]: not closed cleanly..
Aug 24 10:53:34 osmc /etc/mysql/debian-start[1812]: Upgrading MySQL tables if necessary.
Aug 24 10:53:36 osmc /etc/mysql/debian-start[1815]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Aug 24 10:53:36 osmc /etc/mysql/debian-start[1850]: Checking for insecure root accounts.
Aug 24 10:53:36 osmc /etc/mysql/debian-start[1855]: Triggering myisam-recover for all MyISAM tables
osmc@osmc:~$

As it clearly works manually, can anyone tell me how I can find out why it doesn’t work automatically at boot?

2

Osmc uses systemd not initd so the startup scripts are different.

Something similar on this page might be what you need. Just adjust to suit your locations.
http://mysqlserverteam.com/mysql-5-7-native-systemd-support/
Should also start it using
Sudo systemctl start mysql.service

3

Hi, thanks for your reply.

Something must be configured in systemd already because the boot process tries to start MySQL (I see it in the boot up text) and I can start it using “sudo systemctl start mysql.service” (and “sudo service mysql start” too).

Do I have to set up all the systemd files manually for this to work? This seems a rather retrograde step from the package configured sysv-init world of raspbmc.

The link suggests that the there should be a “mysql.conf” file, but the package provided but OSMC appears to use the more traditional /etc/mysql/my.cnf. The link also states that there should be a bash file called “mysqld_pre_systemd” but this hasn’t been installed by the OSMC package. Further it suggests that there should be a “mysql.service” file in “/usr/bin/systemd/system/” which is a location which doesn’t exist on OSMC, these files appear to be in “/etc/systemd/system/” but there aren’t “.service” files for every service there.

Has anyone else had this problem running a mysql server on OSMC? Is there a step by step for getting this working? (The instructions on the website provided are for Fedora and include paths which don’t exist on OSMC).

4

As I said. It’s something similar. Thought it might help to show what you need for a systemd service file. Of course the locations are going to be different.
Not sure but maybe MySQL is already preinstalled.

Just done a quick search and it looks like it is already installed with kodi.
Which is configured with the /etc/mysql/my.cnf
So no need to install it like you have.

5

Hi @jelf10 again thanks for your reply. I don’t doubt that your first reply contains everything I need to solve my problem. Unfortunately I am not experienced enough to know what to change.

The link was for MySQL5.7 running on Fedora. OSMC only provides MySQL5.5 (which doesn’t natively support systemd).

I really do appreciate your help, I just don’t quite understand how to get to that point.

MySQL server certainly wasn’t installed by default for me, I had to install it using “apt-get install mysql-server”, on every other Debian based distribution I have used this is sufficient for it to configure itself and restart automatically at every boot.

Even if I could find out why it was failing, that might point be in the right direction, but it just says “code=exited, status=1/FAILURE” and “Unit mysql.service entered failed state.”, which doesn’t really help me much.

6

I only use MySQL with an external database so haven’t had to set it up before.
Looks like the site I looked at omitted to show that you need to install it first lol.
What I don’t understand is that you must have a mysql.service file looking at what you posted to begin with.
All you do to make that start at boot is
Sudo systemctl enable mysql.service

Give that a try and see if the service starts on boot. Should be the same as manually doing it.

7

Hi @jelf10 I have tried “sudo systemctl enable mysql.service” but it still fails during the boot cycle.

It is definitely enabled somehow within systemd as it comes up on the splash screen during the boot process that it is trying to start it. It also says that it failed to start.

I have no doubt that it is configured to start, what I need to try and determine is why it doesn’t start successfully.

8

if it fails at boot but starts fine manually then that would suggest it is trying to start before somethings ready.
Have a look at this link

[Unit]
After = mysql-server.service`

do you have mysql-server.service or just mysql.service maybe alter that line to mysql.service and try?

9

Running the init script directly is not the proper way to start a service on systemd, or on sysv init systems either for that matter. On sysvinit systems you should be using ‘sudo service mysql start’, and on systemd it should be ‘sudo systemctl start mysql.service’.

It might be an idea to have a quick read over the following to get you up to speed with service management on systemd:

Have you checked the mysql log itself ? Or check the full system journal using ‘sudo journalctl’.

No it is not. MySQL is not installed by default on OSMC.

The link you reference is 4 months out of date and isn’t relevant to this issue - that After line has been added to the mediacenter service months ago - it causes Kodi startup to be delayed until MySQL has been started, but is nothing to do with MySQL itself not starting up.

The apt package is called mysql-server, however the service is just called mysql or mysql.service in systemd parlance.

I just installed mysql-server on my Pi 2 and rebooted and see the following:

osmc@rpi2:~$ sudo systemctl status mysql.service -l
* mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: active (running) since Mon 2015-08-24 22:42:58 BST; 1min 29s ago
  Process: 276 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/mysql.service
           |-392 /bin/sh /usr/bin/mysqld_safe
           `-746 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306

Aug 24 22:42:58 rpi2 mysql[276]: Starting MySQL database server: mysqld . ..
Aug 24 22:42:58 rpi2 mysql[276]: Checking for tables which need an upgrade, are corrupt or were
Aug 24 22:42:58 rpi2 mysql[276]: not closed cleanly..

So it seems to be running fine here. Check /var/log/mysql/error.log to see if anything is logged there…

10

Hi DBMandrake,

Thank you for your detailed reply, much appreciated! Thank you for the link too, very informative.

If I start MySQL with “systemctl start mysql.service” it works too.

Looking at “/var/log/mysql/error.log” the error seems to be that it cannot allocate enough memory during startup, however when I start it manually this allocation works seemlessly. I wonder if it is trying to start before the swap file is available? I’m not sure how to tell though as I can’t see an entry in journalctl for initiating swap.

I’m using an RPi-B so only have 512MB split between RAM and GRAM. This wasn’t a problem on Raspbmc, maybe OSMC requires more RAM?

11

What is your gpu memory split ?

Swap file is not enabled at all on OSMC on a 512MB model.

You can enable swap even on a 512MB model with

touch /enable_swap

However funnily enough we were just discussing yesterday that the swap file doesn’t get enabled until about 30 seconds after boot starts due to it being done after a delay in this performance tuner script:

So at the moment it wouldn’t help you.

12

Hi @DBMandrake, sorry for the delay, I’m on holiday.

I’m using the default OSMC config.txt file.

I use kodi for HD movies, what gpu split would you recommend?

Would be useful if all memory could be used during boot and split assigned only when correct run level achieved…

13

I would recommend the default GPU memory allocations, which are:

128MB (256MB Pi 1)
144MB (512MB Pi 1)
256MB (1GB Pi 2)

14

Hi @DBMandrake, I’ve just checked and my 512MB Pi1B has 144MB allocated to the GPU.

Given the memory issues I’m having at boot could I reduce that to 128MB and still watch HD movies?

15

Changing the GPU memory to 128MB resolved the issue with MySQL and I can still watch HD movies so that’s the primary issue solved! Many thanks @DBMandrake. :grinning: :grinning: :grinning:

Now to the other services which aren’t starting; “openvpn” and “nfs-kernel-server”.

I should rephrase that really as they are starting, just in the wrong order to work.

nfs-kernel-server seems to be starting before rpcbind, which means it isn’t registering and so doesn’t work (see below)

[18:50 gavin@osmc ~] > sudo showmount -e --no-headers
clnt_create: RPC: Program not registered
[18:53 gavin@osmc ~] > sudo systemctl restart nfs-kernel-server
[18:53 gavin@osmc ~] > sudo showmount -e --no-headers
/pidata 10.*.*.0/24,192.168.*.0/24
[18:53 gavin@osmc ~] >

How can I make nfs-kernel-server start by default after rpcbind?

And OpenVPN has a similar issue, it seems to be starting before the network is up (see below)

[18:49 gavin@osmc ~] > sudo cat /var/log/openvpn.log
Fri Sep  4 16:15:20 2015 OpenVPN 2.3.4 arm-unknown-linux-gnueabihf [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Dec  5 2014
Fri Sep  4 16:15:20 2015 library versions: OpenSSL 1.0.1k 8 Jan 2015, LZO 2.08
Fri Sep  4 16:15:22 2015 Control Channel Authentication: using '/etc/openvpn/easy-rsa/keys/ta.key' as a OpenVPN static key file
Fri Sep  4 16:15:22 2015 TCP/UDP: Socket bind failed on local address [AF_INET]192.168.*.*:1194: Cannot assign requested address
Fri Sep  4 16:15:22 2015 Exiting due to fatal error
[18:49 gavin@osmc ~] > sudo systemctl restart openvpn
[18:49 gavin@osmc ~] > sudo cat /var/log/openvpn.log
Sun Sep  6 18:49:58 2015 OpenVPN 2.3.4 arm-unknown-linux-gnueabihf [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Dec  5 2014
Sun Sep  6 18:49:58 2015 library versions: OpenSSL 1.0.1k 8 Jan 2015, LZO 2.08
Sun Sep  6 18:49:58 2015 NOTE: your local LAN uses the extremely common subnet address 192.168.0.x or 192.168.1.x.  Be aware that this might create routing conflicts if you connect to the VPN server from public locations such as internet cafes that use the same subnet.
Sun Sep  6 18:49:59 2015 Control Channel Authentication: using '/etc/openvpn/easy-rsa/keys/ta.key' as a OpenVPN static key file
Sun Sep  6 18:49:59 2015 TUN/TAP device tun0 opened
Sun Sep  6 18:49:59 2015 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Sun Sep  6 18:49:59 2015 /sbin/ip link set dev tun0 up mtu 1500
Sun Sep  6 18:49:59 2015 /sbin/ip addr add dev tun0 local 10.*.*.1 peer 10.*.*.2
Sun Sep  6 18:49:59 2015 GID set to nogroup
Sun Sep  6 18:49:59 2015 UID set to nobody
Sun Sep  6 18:49:59 2015 UDPv4 link local (bound): [AF_INET]192.168.*.*:1194
Sun Sep  6 18:49:59 2015 UDPv4 link remote: [undef]
Sun Sep  6 18:49:59 2015 Initialization Sequence Completed

In a systemV-init world the order was controlled by changing the number in the symlink in the /etc/rc*.d directories, but I have tried that and I suspect that this is not followed in a systemd world.

The file “/lib/systemd/system/openvpn.service” contains the following:

# This service is actually a systemd target,
# but we are using a service since targets cannot be reloaded.

[Unit]
Description=OpenVPN service
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecReload=/bin/true
WorkingDirectory=/etc/openvpn

[Install]
WantedBy=multi-user.target

To me, this says (just as the legacy /etc/init.d/openvpn script does) that OpenVPN shouldn’t be started until after the network is up. Is it failing to do this because OSMC uses connman instead of NM? In which case what should I change this line to to wait for connman to have configured the network?

16

If in openvpn.service I change “After=network.target” to “After=network-online.target” would that resolve my OpenVPN startup issue?

17

I don’t know about openvpn but unfortunately the issue with rpcbind is a well known issue with Debian Jessie where rpcbind does not get started early enough due to a dependency loop. You’ll see the following in your log:

Aug 25 14:05:57 vero systemd[1]: Job rpcbind.service/start deleted to break ordering cycle starting with basic.target/start

(Google search the above error and you will see a lot of discussion about it outside of OSMC)

This is due to rpcbind and nfs-server still using legacy init scripts that are not fully compatible with systemd. Unfortunately although there are ways to work around it it is basically a waiting game waiting for the upstream debian maintainers to come up with a real solution to this problem, which probably entails writing proper systemd service units for these services instead of continuing to rely on old broken legacy sysv init scripts.

18

there are some solution to autostart this service on boot? my problem is openvpn