[How to] Make periodic backups of whole OSMC system

HowTo
1

The backup function within OSMC backs up only your libraries and Kodi user settings. A number of people have asked what is the best way to do a complete backup of OSMC. Here is a script which makes a backup of both partitions of a standard RPi SD card install. The backup device is assumed to be a local disc or a network share mounted in the /media/ directory and formatted as ext4. But see discussion below about nfs shares and extended attributes. [TLDR: it doesn’t matter.] Each backup is dated by the name of the directory it is in – backups are assumed to be not more frequent than daily – and is a complete copy of the files on the two partitions, except those generated by the system which are not needed for a full restore. Hard links are used to minimise duplication of files. It can also make a separate copy of just the files that have changed or been added since the last backup.

As written, the script throws away backups after a certain time or so as to keep only the last n backups. You can ‘freeze’ a previous backup by renaming its directory from say 2017-09-30 to 2017-09-30beforeSeptupgrade. There is code to check how much disc space is being used and it could be modified so as to delete old backups when they are are taking up too much space on the backup media.

Some checks are included, for the existence of the backup media for example, but I can’t claim it’s bomb-proof. There are some progress messages which can be re-directed to a log file for unattended operation. The script does a search for the devices mounted at / and /boot so should work with a USB stick install.

As at version 0.1.9 of the backup script and 0.1.5 of the restore script, vero4k is supported. I don’t have any earlier veros but imagine the backup, at least, will work as is. There is also now a facility to shut down kodi, tvheadend and any other services (user specified) that may write to the filesystem while the backup is happening.

Everyone will have their own take on how and when to make backups. This script can keep daily copies of your SD card for ever which would be overkill but you can set up your own schedule. I keep backups for a week. The only dependencies are rsync and pv, which are in the repository, and for unattended operation cron, which is in the App Store. Use sudo crontab -e to edit crontab for root. A suitable crontab line looks like:

15 2 * * * /home/osmc/osmc-backup >> /home/osmc/backlog

The restore script is interactive and is invoked with:

osmc-restore [-f] path/to/backup/dir [destination]

You can specify the destination (eg sda) on the commandline or you will be prompted for it. The switch -f forces a format of the destination sd card. If there aren’t suitable partitions on the card, you will be prompted to format it. You can also run osmc-restore / to clone your working system to another card, which does not have to be the same size.

The restore for vero is currently a bit clunky - the script makes a tarball of the backup directory so it can be installed using a recent copy of a dtb.img and kernel.img from an installation image. These have to be copied to the sd card manually. There is no clone option for vero.

Comments and suggestions for improvement are welcome.

Edit: Having tasted the delights of Github, I have now published it there,

2 Likes
2

Sorry to resurrect this, but it looks very useful (if I could get it to work!)

I get

Wed May 17 23:25:14 BST 2017 Backup version 0.1.4
Going to back up to /media/pidrive/osmcbackup/2017-05-17 (unless I can’t find
it).
looking for mounted media
media for /media/pidrive/osmcbackup not mounted in /media or /mnt or

If I run it, but no backup.
Any ideas? Pidrive is ext4, and mounted in media…

3

Before I plough through the backup code, what does df show?

4

There is a bit of an issue in one line of that checking code but it shouldn’t affect your setup, only where you have /media/username/mountpoint/. Just going out now but I’ll have a look tonight.

5

Could you also post the output of findmnt -l please?

6

Many thanks

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu May 18 15:42:06 2017 from 162.13-255-62.static.virginmediabusiness.co.uk
osmc@osmc:~$ findmnt -l
TARGET                     SOURCE         FSTYPE  OPTIONS
/dev                       devtmpfs       devtmpf rw,relatime,size=370520k,nr/proc                      proc           proc    rw,relatime
/sys                       sysfs          sysfs   rw,relatime
/run                       tmpfs          tmpfs   rw,relatime
/                          /dev/mmcblk0p2 ext4    rw,relatime,stripe=1024,dat/dev/shm                   tmpfs          tmpfs   rw,nosuid,nodev
/dev/pts                   devpts         devpts  rw,nosuid,noexec,relatime,g/run/lock                  tmpfs          tmpfs   rw,nosuid,nodev,noexec,rela/sys/fs/cgroup             tmpfs          tmpfs   ro,nosuid,nodev,noexec,mode/sys/fs/cgroup/systemd     cgroup         cgroup  rw,nosuid,nodev,noexec,rela/sys/fs/cgroup/cpu,cpuacct cgroup         cgroup  rw,nosuid,nodev,noexec,rela/sys/fs/cgroup/blkio       cgroup         cgroup  rw,nosuid,nodev,noexec,rela/sys/fs/cgroup/memory      cgroup         cgroup  rw,nosuid,nodev,noexec,rela/sys/fs/cgroup/devices     cgroup         cgroup  rw,nosuid,nodev,noexec,rela/sys/fs/cgroup/freezer     cgroup         cgroup  rw,nosuid,nodev,noexec,rela/sys/fs/cgroup/net_cls     cgroup         cgroup  rw,nosuid,nodev,noexec,rela/proc/sys/fs/binfmt_misc   systemd-1      autofs  rw,relatime,fd=22,pgrp=1,ti/dev/mqueue                mqueue         mqueue  rw,relatime
/sys/kernel/debug          debugfs        debugfs rw,relatime
/boot                      systemd-1      autofs  rw,relatime,fd=33,pgrp=1,ti/sys/kernel/config         configfs       configf rw,relatime
/sys/fs/fuse/connections   fusectl        fusectl rw,relatime
/media/pidrive             /dev/sda1      ext3    rw,nosuid,nodev,relatime,da/media/DOWNLOADS           /dev/sdc1      vfat    rw,nosuid,nodev,relatime,ui/boot                      /dev/mmcblk0p1 vfat    rw,noatime,fmask=0022,dmask/media/f03a3de0-b985-4320-8039-f555568fc686
                           /dev/sdb1      ext4    rw,nosuid,nodev,relatime,da/run/user/1000             tmpfs          tmpfs   rw,nosuid,nodev,relatime,siosmc@osmc:~$
7

I tried out your script. I can see you put a great deal of work into it, so many thanks for a nice piece of code.

Oh, and it worked just fine, AFAICT, though it did output one odd message right at the end:

mount: special device systemd-1 does not exist

though running mount showed:

systemd-1 on /proc/sys/fs/binfmt_misc type autofs )rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)

and

systemd-1 on /boot type autofs (rw,relatime,fd=33,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)

8

Many thanks, but the output looks a bit strange and may be incomplete. Can you try to get it looking more like you would see in a terminal. Just cutting and pasting and using the code </> button should make it look more like this:

/boot/efi                       /dev/sda1   vfat      rw,relatime,fmask=0022,dmask=0022,codepage=437
/home                           /dev/sdb1   ext4      rw,noatime,data=ordered
/proc/sys/fs/binfmt_misc        binfmt_misc binfmt_mi rw,relatime
/run/cgmanager/fs               cgmfs       tmpfs     rw,relatime,size=100k,mode=755
/run/user/1000                  tmpfs       tmpfs     rw,nosuid,nodev,relatime,size=395952k,mode=700
/run/user/1000/gvfs             gvfsd-fuse  fuse.gvfs rw,nosuid,nodev,relatime,user_id=1000,group_id

This example is from my linux laptop. Also, findmnt -ul would be better - it doesn’t trucate lines.

9

Thanks for the feedback. Both you and Raspingtiao have an entry /boot at systemd-1. I can’t check right now but I’m pretty sure my /boot is listed as /dev/mmcblk0p1. I only have a Pi2 to work with: are you on Pi3 or zero, perhaps?

Did the script successfully get rid of the /media/system and /media/boot mounts while leaving the ‘working’ mounts for / and /boot intact? If not, I will have to put up a health warning until I can figure out what’s happening.

10

I ran it on a Pi3. At first, it seemed that /boot was no longer in a separate partition, though fdisk -l lists a /dev/mmcblk0p1 partition , in addition to /dev/mmcblk0p2.

Device         Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1        2048   499711   497664  243M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      501760 15351807 14850048  7.1G 83 Linux

df showed:

osmc@osmc:~$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs          370520       0    370520   0% /dev
tmpfs             375512    5088    370424   2% /run
/dev/mmcblk0p2   7177372 2971352   3818388  44% /
tmpfs             375512       0    375512   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             375512       0    375512   0% /sys/fs/cgroup
/dev/sda1       29669976 5638620  22501168  21% /media/29GB
tmpfs              75104       0     75104   0% /run/user/1000

but for some strange reason, a later df showed /boot in its own partition:

osmc@osmc:~$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs          370520       0    370520   0% /dev
tmpfs             375512    5088    370424   2% /run
/dev/mmcblk0p2   7177372 2971352   3818388  44% /
tmpfs             375512       0    375512   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             375512       0    375512   0% /sys/fs/cgroup
/dev/mmcblk0p1    244988   24352    220637  10% /boot
/dev/sda1       29669976 5638620  22501168  21% /media/29GB
tmpfs              75104       0     75104   0% /run/user/1000

so something strange has happened.

WRT your second paragraph, I don’t really understand the question, but here’s what’s currently in /media:

osmc@osmc:~$ ls -l /media
total 12
drwxrwxrwx 9 root root 4096 May 18 21:15 29GB
drwxr-xr-x 2 root root 4096 May 18 13:00 boot
-rw-r--r-- 1 root root  232 Feb 29  2016 README

so no /media/system but there is a /media/boot.

11

The script mounts / again at /media/system and /boot (ie /dev/mmcblk0p1/) at /media/boot. Doing this stops cp and rsync recursing into mountpoints (like your /media/29G) and so backing up more than just the SD card - or even backing up itself! At the end, it unmounts /media/system and /media/boot (or should do). This works for me on Pi2 and on my linux PC where I have /home on a different device. I’m afraid I don’t know what systemd-1 is or why it might be associated with /boot, but I’m a bit worried the umount is umounting something it shouldn’t.

12

Here we are- hopefully it’ll display correctly this time

TARGET                                      SOURCE         FSTYPE      OPTIONS
/dev                                        devtmpfs       devtmpfs    rw,relatime,size=370520k,nr_inodes=92630,mode=755
/proc                                       proc           proc        rw,relatime
/sys                                        sysfs          sysfs       rw,relatime
/run                                        tmpfs          tmpfs       rw,relatime
/                                           /dev/mmcblk0p2 ext4        rw,relatime,stripe=1024,data=ordered
/dev/shm                                    tmpfs          tmpfs       rw,nosuid,nodev
/dev/pts                                    devpts         devpts      rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
/run/lock                                   tmpfs          tmpfs       rw,nosuid,nodev,noexec,relatime,size=5120k
/sys/fs/cgroup                              tmpfs          tmpfs       ro,nosuid,nodev,noexec,mode=755
/sys/fs/cgroup/systemd                      cgroup         cgroup      rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
/sys/fs/cgroup/cpu,cpuacct                  cgroup         cgroup      rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
/sys/fs/cgroup/blkio                        cgroup         cgroup      rw,nosuid,nodev,noexec,relatime,blkio
/sys/fs/cgroup/memory                       cgroup         cgroup      rw,nosuid,nodev,noexec,relatime,memory
/sys/fs/cgroup/devices                      cgroup         cgroup      rw,nosuid,nodev,noexec,relatime,devices
/sys/fs/cgroup/freezer                      cgroup         cgroup      rw,nosuid,nodev,noexec,relatime,freezer
/sys/fs/cgroup/net_cls                      cgroup         cgroup      rw,nosuid,nodev,noexec,relatime,net_cls
/proc/sys/fs/binfmt_misc                    systemd-1      autofs      rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/dev/mqueue                                 mqueue         mqueue      rw,relatime
/sys/kernel/debug                           debugfs        debugfs     rw,relatime
/boot                                       systemd-1      autofs      rw,relatime,fd=33,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/sys/kernel/config                          configfs       configfs    rw,relatime
/sys/fs/fuse/connections                    fusectl        fusectl     rw,relatime
/media/pidrive                              /dev/sda1      ext3        rw,nosuid,nodev,relatime,data=ordered
/media/DOWNLOADS                            /dev/sdc1      vfat        rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0111,dmask=0000,allow_utime=0022,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,errors=remount-ro
/boot                                       /dev/mmcblk0p1 vfat        rw,noatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro
/media/f03a3de0-b985-4320-8039-f555568fc686 /dev/sdb1      ext4        rw,nosuid,nodev,relatime,data=ordered
/run/user/1000                              tmpfs          tmpfs       rw,nosuid,nodev,relatime,size=75104k,mode=700,uid=1000,gid=1000
/proc/sys/fs/binfmt_misc                    binfmt_misc    binfmt_misc rw,relatime
13

AFAIK, the build for the Pi2 and Pi3 is the same, though I guess it’s possible that the code might run differently once installed. Perhaps @sam_nazarko could clarify this point.

And if they are the same, where does that leave us?

14

Please stop!
I’ve deleted the original script and started from scratch and it appears to be working now! Rsync wasnt installed oddly (osmc is a fresh install)

Apologies!

15

rsync is not installed by default.

Thanks for testing!

16

Cheers!

It finished in 3 minutes or so. The total backup size was ~672Mb. I don’t appear to have anything in the /boot directory though
This was the command output

osmc@osmc:~$ sudo ./backupscript.sh
Thu May 18 21:34:08 BST 2017 Backup version 0.1.4
Going to back up to /media/f03a3de0-b985-4320-8039-f555568fc686/osmcbackup/2017-05-18 (unless I can't find it).
looking for mounted media
media found at /media/f03a3de0-b985-4320-8039-f555568fc686
112776680 KiB available on drive

Counting and deleting backups older than 2017-04-18 or if more than 3 of them
Now 112776680 KiB available on drive

system is on /dev/mmcblk0p2
Backing up system partition to /media/f03a3de0-b985-4320-8039-f555568fc686/osmcbackup/2017-05-18/system
Backup directory is empty - making first copy
Time to back up system: 00h:03m:58s

boot is on systemd-1
/dev/mmcblk0p1
Backing up boot partition to /media/f03a3de0-b985-4320-8039-f555568fc686/osmcbackup/2017-05-18/boot

Usage:
 mount [-lhV]
 mount -a [options]
 mount [options] [--source] <source> | [--target] <directory>
 mount [options] <source> <directory>
 mount <operation> <mountpoint> [<target>]

Options:
 -a, --all               mount all filesystems mentioned in fstab
 -c, --no-canonicalize   don't canonicalize paths
 -f, --fake              dry run; skip the mount(2) syscall
 -F, --fork              fork off for each device (use with -a)
 -T, --fstab <path>      alternative file to /etc/fstab
 -h, --help              display this help text and exit
 -i, --internal-only     don't call the mount.<type> helpers
 -l, --show-labels       lists all mounts with LABELs
 -n, --no-mtab           don't write to /etc/mtab
 -o, --options <list>    comma-separated list of mount options
 -O, --test-opts <list>  limit the set of filesystems (use with -a)
 -r, --read-only         mount the filesystem read-only (same as -o ro)
 -t, --types <list>      limit the set of filesystem types
     --source <src>      explicitly specifies source (path, label, uuid)
     --target <target>   explicitly specifies mountpoint
 -v, --verbose           say what is being done
 -V, --version           display version information and exit
 -w, --rw, --read-write  mount the filesystem read-write (default)

 -h, --help     display this help and exit
 -V, --version  output version information and exit

Source:
 -L, --label <label>     synonym for LABEL=<label>
 -U, --uuid <uuid>       synonym for UUID=<uuid>
 LABEL=<label>           specifies device by filesystem label
 UUID=<uuid>             specifies device by filesystem UUID
 PARTLABEL=<label>       specifies device by partition label
 PARTUUID=<uuid>         specifies device by partition UUID
 <device>                specifies device by path
 <directory>             mountpoint for bind mounts (see --bind/rbind)
 <file>                  regular file for loopdev setup

Operations:
 -B, --bind              mount a subtree somewhere else (same as -o bind)
 -M, --move              move a subtree to some other place
 -R, --rbind             mount a subtree and all submounts somewhere else
 --make-shared           mark a subtree as shared
 --make-slave            mark a subtree as slave
 --make-private          mark a subtree as private
 --make-unbindable       mark a subtree as unbindable
 --make-rshared          recursively mark a whole subtree as shared
 --make-rslave           recursively mark a whole subtree as slave
 --make-rprivate         recursively mark a whole subtree as private
 --make-runbindable      recursively mark a whole subtree as unbindable

For more details see mount(8).
17

OK. The problem is the two mountpoints for /boot, which I hadn’t appreciated or noticed on my device. I will fix that.

Thanks both of you.

18

Mine took 48 minutes to backup system. It’s a USB3 thumbdrive that never struck me as being that slow. (ext4 formatted, backup is 2.9GB)

osmc@osmc:~$ sudo ./osmc-backup 
Thu May 18 12:12:18 BST 2017 Backup version 0.1.4
Going to back up to /media/29GB/osmcbackup/2017-05-18 (unless I can't find it).
looking for mounted media
media found at /media/29GB
25,513,476 KiB available on drive

Counting and deleting backups older than 2017-05-11 or if more than 7 of them

system is on /dev/mmcblk0p2
Backing up system partition to /media/29GB/osmcbackup/2017-05-18/system
Backup directory is empty - making first copy
Time to back up system: 00h:48m:36s

boot is on systemd-1
Backing up boot partition to /media/29GB/osmcbackup/2017-05-18/boot
mount: special device systemd-1 does not exist
osmc@osmc:~$

My log differs slightly from @Raspingtiao’s

19

Please try version 0.1.5 that I have pasted above. Hopefully it will stop the systemd-1 error message which is aborting the script and let the script complete. Unfortunately, I can’t test it as I’m away from home.

48mins sounds a lot for 2.9GB! From memory, my first full backup of 22GB to a mechanical USB HDD took about 30mins. How long does a plain cp of that size take?

20 
osmc@osmc:~$ sudo ./backupscript.sh
Thu May 18 22:50:54 BST 2017 Backup version 0.1.5
Going to back up to /media/f03a3de0-b985-4320-8039-f555568fc686/osmcbackup/2017-05-18 (unless I can't find it).
looking for mounted media
media found at /media/f03a3de0-b985-4320-8039-f555568fc686
112022100 KiB available on drive

Counting and deleting backups older than 2017-02-17 or if more than 3 of them
Now 112022100 KiB available on drive

system is on /dev/mmcblk0p2
system has already been backed up today and will be overwritten
Press any key within 10 secs to skip ....
OK, deleting earlier backup
/system deleted
Now 112776684 KiB available on drive
Backing up system partition to /media/f03a3de0-b985-4320-8039-f555568fc686/osmcbackup/2017-05-18/system
Backup directory is empty - making first copy
Time to back up system: 00h:04m:52s

boot is on /dev/mmcblk0p1
Backing up boot partition to /media/f03a3de0-b985-4320-8039-f555568fc686/osmcbackup/2017-05-18/boot
Backup directory is empty - making first copy
Time to back up boot: 00h:00m:02s
cp: cannot stat '/home/osmc/rsynclog2017-05-18': No such file or directory

776088 KiB used in this backup
112000596 KiB available on drive
All done!