Automatic fsck of root filesystem on start/stop


#1

Hi,

I activated an automatic root filesystem check on every start/stop by changing the entry

/dev/mmcblk0p2 / ext4 defaults,noatime 0 1

(last digit changed from 0 (skip check) to 1 (check this with highest prio))
in file

/etc/fstab

and also changing in

 /etc/default/rcS

the last entry to

# automatically repair filesystems with inconsistencies during boot
 FSCKFIX=yes

Is there any argument to not implement it in this way and I do not see a potential danger/problem right now?


Switch on Pi without unplugging power source
OS stability issues
#2

As far as I know FSCKFIX=yes in /etc/default/rcS will not be honoured by systemd as fsck is not run by an /etc/init script but by two systemd services called systemd-fsck-root.service and systemd-fsck.service.

The systemd equivalent is fsck.repair=yes on the kernel command line. (In /boot/cmdline.txt on a Pi or /boot/uEnv.txt on a Vero)

http://www.freedesktop.org/software/systemd/man/systemd-fsck@.service.html

So your fstab change will work and trigger fsck to run on boot, but if there are any errors they will not be automatically fixed and the system will drop to an emergency console. In previous versions of OSMC this would prompt for a non-existent root password and you would be stuck, however with the most recent update you will get a root prompt without a password, where you could manually run fsck with the repair option.

We are working on a proper fsck solution, unfortunately achieving fully automated repairs safely, (on a device that doesn’t typically have a keyboard attached) and without significantly slowing the boot process or contaminating the splash screen with messages when repairs are not needed is not as easy as it appears so we are working to get this right.


#3

So, I tested it the hard way:

  • let the last digit in fstab set to 1 (/), 2 (/boot) and 3 for my connected hdd
  • NO entry in rcS, NO entry in commandline.txt and uEnv.txt
  • started a large file transfer to a smb share located at the hdd
  • simply removed the power supply

Reconnecting to power gave me the following screen:

So, obviously fsck repaired something without a user interaction?

Addendum:
From the link above the following phrase is crucial:

Kernel Command Line

systemd-fsck understands one kernel command line parameter:

fsck.mode=
One of “auto”, “force”, “skip”. Controls the mode of operation. The default is “auto”, and ensures that file system checks are done when the file system checker deems them necessary. “force” unconditionally results in full file system checks. “skip” skips any file system checks.
fsck.repair=
One of “preen”, “yes”, “no”. Controls the mode of operation. The default is " preen", and will automatically repair problems that can be safely fixed. "yes " will answer yes to all questions by fsck and “no” will answer no to all questions.

So, there might be circumstances the problems cannot be safely fixed by the Default “preen” … but these situations will most possible require a new installation of OSMC from scratch anyway. So, the fsck trigger in /etc/fstab is better than nothing as long as there is no better method implemented in OSMC.


#4

I’m watching this with interest. It would be fantastic if OSMC could detect an improper shutdown (like a power cut) and automatically run fsck on the next boot.


#5

Yes the default repair option is preen, which basically means only make minor, “safe” changes that don’t carry a risk of data loss, such as cleaning up orphaned inodes, and replaying the journal.

Because ext4 is journalled many types of “minor” corruption like this can be repaired safely by replaying the journal. These minor types of corruption won’t actually cause you any operational problems though, and would have gone unnoticed.

However without fsck.repair=yes more severe corruption will still trigger systemd’s emergency mode, and it is this type of severe corruption that leads to a system booting in read only mode or causes disk IO errors that cause updates to fail etc that cannot be fixed by preen mode. So any real problems that would have caused you actual issues won’t be fixed by preen.

Before the September update this would have taken you to a prompt asking you to enter the root password to login - which is not possible as we don’t set a root password, so you would be stuck. Since the September update you will at least be taken to a command prompt where the problem can be manually repaired.

However for a device that usually doesn’t have a keyboard connected and is intended to be used somewhat as an “appliance” this is not very user friendly.

Don’t worry, automated fsck support is very high on our priority list, and we were hoping to have it ready for this release but other things have intervened, and there has also been a bit of internal debate about the best way to do it.

For anyone who wants to enable an automated fsck with repair right now all you need to do is edit /etc/fstab and change the second number at the end of each line from 0 to 1 (systemd doesn’t make any distinction between 1,2,3 etc, so using numbers above 1 makes no difference) then edit /boot/cmdline.txt (/boot/uEnv.txt on a vero) and on the end of the same line add a space then:

fsck.repair=yes

On rebooting you should see a two line fsck scan in the top left corner of the splash screen momentarily during boot. When no repairs are needed it only takes a couple of seconds.


#6

Great news. Keep up the good work - OSMC is a fantastic OS.


#7

Hi all,

sorry, to pull out this old thread. I got a brand new Vero2 and would like to know where the kernel command line file is located on this device and and what to insert to activate the automatic fsck repair function.

Thx in advance,


#8

Vero 2 will run fsck on boot automatically


#9

Sry again to revive this, but this automatic fsck is “just a check and no repair” right?
How would I change this to also do a repair no matter what?
Like fsck -y /dev/sdX.
After reinstalling kodi and still getting sad faces/ sad face loops, picture freezes/stuttering when watching videos, strange errors like seg fault in transmission and occasional kernel panics this is basically my last idea before reinstalling osmc completely.


#10

No – it’s a brief check and repair if we spot something, it’s not an unconditional force fsck.

Just attach USB keyboard, hold shift and you can fsck that way. Otherwise you would need to modify the initramfs to do this which changes each kernel.

You reported some problems a long time ago with a hard disk. If this was giving you kernel panics, then some data may not have been written back and you may have ended up with some slight corruption.

Sam


#11

Yeah, gparted on my ubuntu istall is messed up somehow and I used it to format the harddrive. Since formatting the harddrive with gparted run from a live usbstick there were no issues with drive anymore.
I still refuse to reinstall osmc and setup everything again because it seems illogical to me that a malformatted external harddrive could somehow mess up the filesystem on nand.
Could this be possible in your opinion, if so how? I seems like it happened.

Regarding kernel panics: I had a daily, weekly and monthly timer running rsnapshot, basically a rsync backup script. If I think of it now, since I disabled these timers, I didn’t get kernel panics anymore…

Now the problems with kodi prevail. Sad face and often sad face loops but I can’t bind it to certain szenarios, happens randomly. Also these video freezes during playback, audio is fine and after some time the video kind of fast forwards and catches up again to the sound.
This also seems to happen ramdomly and with different audio and video formats/resolutions.
So I tried to reinstall kodi with apt-get install --reinstall vero2-mediacenter-osmc
but I noticed after starting kodi again all custom addons and settings are still there.
Is there maybe a way to really reinstall kodi from scratch?

Sry this got offtopic, thx for the tipp to fsck with keyboard.


#12
  • USB hard drive has a problem
  • Kernel panics
  • Data that was scheduled to be written to the NAND never gets written back
  • Journal is corrupted or left in a partial state / filesystem in bad shape and fsck doesn’t necessarily detect and repair this

The effect can vary depending on what your device was doing at the same time. In an extreme case, it’s the equivalent of pulling the plug during an update. This is similar to a blue screen of death on a Windows box at a crucial time.

You would need to remove your .kodi userdata directory, but that might not be the problem.

Can’t really advise much more, as the usual go-to when you get problems is to reinstall.

Sam


#13

Ok, thx for elaborating on that.
So it’s reinstall.
One stupid question: Do you think it’s safe to copy certain config files or maybe even folders (could copy them an preserve the permissions) before reinstall and use them later?
Or do you think it maybe would be better to kind of copy,paste just the text of these files in the new conf files?
I mean if the files are readable/copyable they should be ok, don’t they?


#14

They may be causing your crashes. But if you copy it over and you get crashes, you know your answer :wink:

It’s probably fine.


#15

Only because you don’t see/feel any problem with the drive it doesn’t mean it doesn’t have issues. Have you run smartctrl against the drive to see any errors building up?