Emergency mode after reboot - no SSH access

After a reboot, my device is in Emergency mode. I cannot access it via SSH. According to my router, there is no device, even though the Vero is connected via network cable.

All worked fine before the reboot. OSMC is installed normally on the internal memory. I have a USB drive attached and mounted it via FSTAB in /mnt/USBDRIVE. The Vero 4K also has NFS server running 2 RPis access the USBDRIVE via NFS. This has worked flawlessly, until this reboot.

Screenshot of my TV screen attached.

Have you tried removing the USB drive and seeing if boot succeeds?
Did you edit fstab at all?

It looks like the drive may be corrupted, which is causing this.

I hope the drive is not corrupt. Disconnected the USB drive, Now it waits for 1m30s, continues boot and waits again, eventually ending up in emergency mode. photo attached.

You need to attach a USB keyboard and reboot.

I’ve never been in emergency mode on a V4K, so I’m not sure what capabilities you’ll find. Sometimes it might be in a restricted shell with read-only access, so you’ll have to experiment a bit to see what you’re allowed to do.

To get out of your current hole you need to edit /etc/fstab and comment out the line for /mnt/USBDRIVE. (You’re already root, so no need for sudo.) Then reboot without the drive attached.

Now this I find interesting:

  1. Disconnected the drive, connected it to an Intel NUC with the same USB cable. I ran the standard Windows disk check tool, it found and fixed a few errors (corrupted files). The disk works fine.
  2. Connected it back to my Vero 4K. Still the same issue. Even though the drive has enough power (I can feel the vibration).
  3. Edited fstab, commented out the line for usb drive, shutdown Vero, disconnected the USB drive and booted Vero. Successful boot.
  4. While in Kodi, connected the usb drive. It is automounted and I can view all my files via Kodi File Manager.

It seems my fstab entry simply doesn’t work anymore since the reboot. I don’t know what changed, because since I use fstab, I have rebooted several times without any issues.

This is my fstab:

# rootfs is not mounted in fstab as we do it via initramfs. Uncomment for remount (slower boot)
#/dev/vero-nand/root  /    ext4      defaults,noatime    0   0
#UUID=01D0491ECAD9C3E0 /mnt/USBDRIVE fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096,uhelper=udisks

Is there a reason you’re using fstab? We automatically mount disks under /media by default.

Yes, it was the conclusion of a different topic here. I use NFS server on the Vero. An RPi3 with Transmission needs access to it, downloads to the USB drive.

With automount, NFS server fails to start at boot because the drive is not mounted yet. With FStab, it is mounted because the folder /mnt/USBDRIVE always exists.

But regardless of the reason behind using Fstab, shouldn’t fstab simply work?

The journal will tell you what the issue with mounting is.

I tried that on the TV, but journalctl -xb just gives me a huge logfile.
How can I check this via SSH?
When I uncomment the line in fstab and run sudo mount -a, the drive is successfully mounted (perhaps because it was already automounted in /media ?)

So it only fails at boot.

Might be an issue with using FUSE in fstab so early.
I don’t have an NTFS based drive to test with here unfortunately.

What’s the reason for the uhelper=udisks bit in the fstab?

Put another way, why aren’t you using systemd for mounting?

I am not a Linux pro. All I want is NFS server to start automatically and successfully at boot.
I learned on this forum how to setup fstab. I simply copied the options after looking at how it was mounted via automount. I don’t actually know what they mean. It worked well for a few months so I was happy.

I didn’t know I could also mount using systemd. Sorry I am really not an expert with file systems and Linux… is there an example for the systemd service file to mount a drive?

Generally, you would add noauto,x-systemd.automount to the fstab line – and remove the uhelper=udisks bit.

Then cross your fingers and see if it works. :wink:

I searched through the journal:

3rd line from the bottom that’s the USB drive:

And here it fails with error code 32:

Systemd is mentioned so fstab is being executed vis systemd? I didn’t modify any system related configuration except for fstab and installing NFS server.

Ah! Now OSMC boots but during boot I can see:
USBDRIVE failed to mount
NFS server fails to start (perhaps expected since /mnt/usbdrive is not accessible)

Now I have OSMC, but no harddrive :frowning:

-- Unit mnt-USBDRIVE.mount has begun starting up.
Apr 07 22:43:02 Vero connmand[313]: iptables support missing error 2 (No such file or directory)
Apr 07 22:43:02 Vero connmand[313]: Failed to flush table 'mangle': Invalid argument
Apr 07 22:43:02 Vero systemd[1]: mnt-USBDRIVE.mount: Mount process exited, code=exited status=32
Apr 07 22:43:02 Vero systemd[1]: Failed to mount /mnt/USBDRIVE.
-- Subject: Unit mnt-USBDRIVE.mount has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mnt-USBDRIVE.mount has failed.

I don’t understand why IP tables is also being mentioned here. Not sure if thats related.

EDIT: it even fails if I remove all options and only keep noauto,x-systemd.automount.

osmc@Vero:~$ sudo systemctl status mnt-USBDRIVE.mount
● mnt-USBDRIVE.mount - /mnt/USBDRIVE
   Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2018-04-07 22:55:05 CEST; 2min 27s ago
    Where: /mnt/USBDRIVE
     What: /dev/disk/by-uuid/01D0491ECAD9C3E0
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
  Process: 393 ExecMount=/bin/mount /dev/disk/by-uuid/01D0491ECAD9C3E0 /mnt/USBDRIVE -t fuseblk -o x-systemd.automount (code=exited, status=32)

Apr 07 22:55:05 Vero systemd[1]: Mounting /mnt/USBDRIVE...
Apr 07 22:55:05 Vero systemd[1]: mnt-USBDRIVE.mount: Mount process exited, code=exited status=32
Apr 07 22:55:05 Vero systemd[1]: Failed to mount /mnt/USBDRIVE.
Apr 07 22:55:05 Vero systemd[1]: mnt-USBDRIVE.mount: Unit entered failed state.

And with more options:

osmc@Vero:~$ sudo systemctl status mnt-USBDRIVE.mount
● mnt-USBDRIVE.mount - /mnt/USBDRIVE
   Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2018-04-07 22:51:23 CEST; 37s ago
    Where: /mnt/USBDRIVE
     What: /dev/disk/by-uuid/01D0491ECAD9C3E0
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
  Process: 388 ExecMount=/bin/mount /dev/disk/by-uuid/01D0491ECAD9C3E0 /mnt/USBDRIVE -t fuseblk -o rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,x-systemd.au

Apr 07 22:51:23 Vero systemd[1]: Mounting /mnt/USBDRIVE...
Apr 07 22:51:23 Vero systemd[1]: mnt-USBDRIVE.mount: Mount process exited, code=exited status=32
Apr 07 22:51:23 Vero systemd[1]: Failed to mount /mnt/USBDRIVE.
Apr 07 22:51:23 Vero systemd[1]: mnt-USBDRIVE.mount: Unit entered failed state.

EDIT:
fstab works just fine if I comment out the whole line, reboot (drive will get automounted), uncomment the line (with all the options) and run mount -a, the drive gets mounted to /mnt/USBDRIVE but this only works with uhelper=udisks, if I use noauto,x-systemd.automountthis trick does not work and the drive is not mounted, even though sudo mount -a does not throw an error.

So:

  1. it used to work just fine.
  2. it still works but not at boot.
  3. no matter what options I remove, it still fails at boot.

That might be incorrect. Systemd should automount /mnt/USBDRIVE on demand, so if you try ls /mnt/USBDRIVE it should then automatically mount it.

There was something similar a while back and I seem to recall that it was resolved by removing the mount point and re-creating it. Worth a try.

Edit: It’s probably worth trying just the noauto,x-systemd.automount with a filesystem type of ntfs-3g, instead of fuseblk.

I had to first stop nfs-server, then I could use sudo umount /mnt/USBDRIVE.
I deleted the folder /mnt/USBDRIVE, commented out fstab. Rebooted.

Then I uncommented the line in fstab, changed filesystem to ntfs-3g and used noauto,x-systemd.automount with the other options except uhelper=udisks.

Rebooted and I had a successful boot, mount and NFS server!

Now I changed ntfs-3g back to fuseblk, boot was OK but no mount.
So I swapped it again…

Something must have changed during the last update. I will stick to ntfs-3g.

Thanks for helping out!

fuseblk has never been supported. Surprised it worked…

Well only partly related to your current problem but as you use the drive explicitly with OSMC I would recommend to format the drive as ext4 instead of ntfs