[How to] Make periodic backups of whole OSMC system


#21

The thumbdrive seemed ok when I just tested it:

osmc@osmc:/media/29GB$ time dd if=/dev/zero of=zero.dat2 bs=1M count=1K conv=fdatasync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 42.4812 s, 25.3 MB/s

real	0m42.496s
user	0m0.030s
sys	0m1.920s

Just over 42 seconds to write 1GB, though I have my doubts about the way the conv= and oflag= directives are working.

I’ll test it tomorrow.


#22

That’s exactly what it should say, thank you!

The cp: cannot stat … message should only occur for the first backup.


#23
osmc@osmc:~$ time sudo ./osmc-backup-0.1.5 
Fri May 19 20:27:32 CEST 2017 Backup version 0.1.5
Going to back up to /media/29GB/osmcbackup/2017-05-19 (unless I can't find it).
looking for mounted media
media found at /media/29GB
20,355,324 KiB available on drive

Counting and deleting backups older than 2017-05-12 or if more than 7 of them
Backup from 2017-05-18 kept
Now 20,355,324 KiB available on drive

system is on /dev/mmcblk0p2
Backing up system partition to /media/29GB/osmcbackup/2017-05-19/system
There are 1 previous backups
Found latest backup at /media/29GB/osmcbackup/2017-05-18/system
Incremental backup: copying new and changed files to /media/29GB/osmcbackup/2017-05-19/system
Making a copy of the new and changed files in /media/29GB/osmcbackup/2017-05-19/system-delta
Cloning unchanged files from previous backup
Deleting files no longer on source
Time to back up system: 00h:17m:04s

boot is on /dev/mmcblk0p1
Backing up boot partition to /media/29GB/osmcbackup/2017-05-19/boot
There are 1 previous backups
Found latest backup at /media/29GB/osmcbackup/2017-05-18/boot
Incremental backup: copying new and changed files to /media/29GB/osmcbackup/2017-05-19/boot
Making a copy of the new and changed files in /media/29GB/osmcbackup/2017-05-19/boot-delta
Cloning unchanged files from previous backup
Deleting files no longer on source
Time to back up boot: 00h:00m:03s

129,128 KiB used in this backup
20,226,196 KiB available on drive
All done!


real	17m48.305s
user	1m2.760s
sys     0m48.010s
osmc@osmc:/media/29GB/osmcbackup$ sudo du -sh *
2.9G	2017-05-18
127M	2017-05-19

So it looks to have worked, taking 17 minutes this time but writing only 127MB to the drive.

Clearly, we have an initial backup and some kind of delta backup. Three questions:

1 Is the delta backup incremental or differential?
2 There’s a message about deleting backups older than a specified date or if more than 7 of them. How does it ensure that a full backup always exists along with the subsequent deltas?
3. Is there a script written for reinstalling to a point in time?

Edit: I think I’ve figured out the answer to Q2. You’ve used hard links to the inodes. Pretty neat!
It therefore follows that the answer to Q1 is differential. (Still figuring out how to restore. ;))
And /media no longer has any iffy mount points left over.


#24

In addition to making a full backup, the script copies just the changed and new files to the delta directory. It is incremental. This is optional - I added it just to check everything was working as expected. Can be interesting if you want to know what was changed in a distro upgrade.

As you have noted, they are not deltas but a full snapshot. The idea is Mike Rubel’s http://www.mikerubel.org/computers/rsync_snapshots/

Yes, but I haven’t tested it as much as the backup script - more of an alpha than a beta. I’ll do some more checking and post it. Basically, you just delete all files on each of the two partitions and copy the files back from /media/29GB/osmcbackup/[date]/boot and /media/29GB/osmcbackup/[date]/system.


#25

Thanks for the link to Mike Rubel’s article. I’d not seen it before but all of a sudden, now I know about it, i’ve noticed it in a few places, including rsnapshot.org.

Were you aware of rsnapshot? It’s in the Debian repo, for example.


#26

Yes, I looked at a number of ‘ready-made’ backup solutions. They are usually targetted at backing up only user files, and in a multi-user environment which we don’t need. What I was after was something more like the ‘whole card’ backup with dd but which runs from within a working OSMC,

Along the way, I found some quirks of rsync which it was not clear were taken care of by rsnapshot and others based on rsync.


#27

Many thanks for this Graham.

Out of interest, if I do a “diff” to compare the root directory with the “system” backup directory, I find many files in /etc /boot /dev that only exist on the sd card, but not in the backup. Is that intentional?


#28

We don’t backup /dev as the system will re-generate it.
Within system there should be just the boot/ directory, empty, because all the files are in the ‘boot’ backup. /boot/ is just a mountpoint.
In /etc there are some links to other parts of the filesystem. Let me check what I have.


Backup whole osmc system (not just kodi settings)/ Acess internal storage
#29

All I get is reports for some broken links (they are broken on both the running filesystem and the backup).

root@osmc:/media/60Gext4/osmcbackup/2017-05-22/system# diff -r etc /etc
diff: etc/alternatives/awk.1.gz: No such file or directory
diff: /etc/alternatives/awk.1.gz: No such file or directory
diff: etc/alternatives/editor.1.gz: No such file or directory
diff: /etc/alternatives/editor.1.gz: No such file or directory
diff: etc/alternatives/nawk.1.gz: No such file or directory
diff: /etc/alternatives/nawk.1.gz: No such file or directory
diff: etc/alternatives/pager.1.gz: No such file or directory
diff: /etc/alternatives/pager.1.gz: No such file or directory
diff: etc/alternatives/pico.1.gz: No such file or directory
diff: /etc/alternatives/pico.1.gz: No such file or directory
diff: etc/alternatives/w.1.gz: No such file or directory
diff: /etc/alternatives/w.1.gz: No such file or directory
diff -r etc/fake-hwclock.data /etc/fake-hwclock.data
1c1
< 2017-05-22 09:17:01
---
> 2017-05-22 10:17:01

Can you post your diff output, please?


#30

is it possible to rsync to a nfs device with this?


#31

I don’t see why not, but I don’t use nfs so not entirely clear how nfs drives are addressed. If the mount point for your nfs share is in /media or /mnt then the script won’t care and should work fine. It won’t handle URIs like smb:// or talk to a remote rsync server.


#32

i just added nfs to this list: FSTYPES=(ext2 ext3 ext4 xfs reiserfs)
and changed the default backup directory to a folder of my nfs mount.

seems to be working … is there a way to check if the backup is complete when its done?

cheers
b


#33

You can use diff and/or meld to compare the backup with the originals.

Let me know if it works!


#34

so far i got this errors:

osmc@B2VERO:~$ sudo ./rsync-backup
Sam Jun 10 19:55:24 CEST 2017 Backup version 0.1.6
Going to back up to /mnt/b2nas/osmc-backup/B2VERO/2017-06-10 (unless I can’t find it).
looking for mounted media
media found at /mnt/b2nas
23’333’056 KiB available on drive

Counting and deleting backups older than 2017-06-03 or if more than 7 of them
Now 23’333’056 KiB available on drive

system is on /dev/vero-nand/root
Backing up system partition to /mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system
Backup directory is empty - making first copy
rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/bin/.systemd-detect-virt.Y61kNI"",“security.capability”) failed: Operation not supported (95)
rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/lib/kodi/.kodi.bin.iLfhi1"",“security.capability”) failed: Operation not supported (95)
rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/bin/.systemd-detect-virt.Y61kNI"",“security.capability”) failed: Operation not supported (95)
rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/lib/kodi/.kodi.bin.iLfhi1"",“security.capability”) failed: Operation not supported (95)rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/bin/.systemd-detect-virt.Y61kNI"",“security.capability”) failed: Operation not supported (95)
rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/lib/kodi/.kodi.bin.iLfhi1"",“security.capability”) failed: Operation not supported (95)rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]


#35

Ah, Vero! I don’t know exactly how Vero’s file system is set up. Can you post findmnt -l for me, please?

Can’t look at it tonight but will have a look tomorrow.


#36
TARGET                 SOURCE     FSTYPE     OPTIONS
/dev                   devtmpfs   devtmpfs   rw,relatime,size=406496k,nr_inodes=101624,mode=755
/proc                  proc       proc       rw,relatime
/sys                   sysfs      sysfs      rw,relatime
/run                   tmpfs      tmpfs      rw,relatime
/                      /dev/vero-nand/root
                                  ext4       rw,relatime,stripe=1024,data=ordered
/sys/kernel/security   securityfs securityfs rw,nosuid,nodev,noexec,relatime
/dev/shm               tmpfs      tmpfs      rw,nosuid,nodev
/dev/pts               devpts     devpts     rw,nosuid,noexec,relatime,gid=5,mode=620
/run/lock              tmpfs      tmpfs      rw,nosuid,nodev,noexec,relatime,size=5120k
/sys/fs/cgroup         tmpfs      tmpfs      ro,nosuid,nodev,noexec,mode=755
/sys/fs/cgroup/systemd cgroup     cgroup     rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
/sys/fs/cgroup/bfqio   cgroup     cgroup     rw,nosuid,nodev,noexec,relatime,bfqio
/mnt/da2nas            systemd-1  autofs     rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/mnt/b2nas             systemd-1  autofs     rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/proc/sys/fs/binfmt_misc
                       systemd-1  autofs     rw,relatime,fd=25,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/dev/mqueue            mqueue     mqueue     rw,relatime
/sys/kernel/debug      debugfs    debugfs    rw,relatime
/sys/fs/fuse/connections
                       fusectl    fusectl    rw,relatime
/mnt/b2nas             192.168.3.38:/share/Qdownload
                                  nfs        rw,nosuid,nodev,noexec,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=14,retrans=2,sec=sys,mountaddr=192.168.3.
/mnt/da2nas            192.168.11.1:/share/Download
                                  nfs        rw,nosuid,nodev,noexec,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=14,retrans=2,sec=sys,mountaddr=192.168.11
/run/user/1000         tmpfs      tmpfs      rw,nosuid,nodev,relatime,size=81516k,mode=700,uid=1000,gid=1000
osmc@B2VERO:~$

Backup whole osmc system (not just kodi settings)/ Acess internal storage
#37

Hmmm. If this is a Vero2 I don’t think I can help. I might be able to figure out the backup but not the restore without having a device to hand. I might just get myself a V4k, but suspect that’s different.


#38

its a vero2, yes.


#39

I have a feeling that the errors in post #34 are related to NFS. I’ll see if I can reproduce the problem on a Pi, so we can see whether it’s Vero- or NFS-related.

I have a Vero 4K myself but believe that the Vero2 uses a similar system for the root partition, which is to say that it uses LVM as a means of consolidating multiple block devices into a single partition. From a backup perspective, the formatted partition is /dev/vero-nand/root.

osmc@osmc:~$ sudo lvdisplay --map
  --- Logical volume ---
  LV Path                /dev/vero-nand/root
  LV Name                root
  VG Name                vero-nand
  LV UUID                dmPd6p-6iBl-qZ0g-taeV-yWmu-ByEA-2ZeAo7
  LV Write Access        read/write
  LV Creation host, time osmc-installer, 2015-01-01 01:00:15 +0100
  LV Status              available
  # open                 1
  LV Size                14.29 GiB
  Current LE             3658
  Segments               4
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0
   
  --- Segments ---
  Logical extents 0 to 3148:
    Type		linear
    Physical volume	/dev/data
    Physical extents	0 to 3148
   
  Logical extents 3149 to 3403:
    Type		linear
    Physical volume	/dev/system
    Physical extents	0 to 254
   
  Logical extents 3404 to 3530:
    Type		linear
    Physical volume	/dev/cache
    Physical extents	0 to 126
   
  Logical extents 3531 to 3657:
    Type		linear
    Physical volume	/dev/instaboot
    Physical extents	0 to 126

#40

Thanks for that. My point above is that even if we had a ‘full’ backup, the process of restoring it is more complicated than just copying a load of files onto an SD card. If you can crack that, then all good. Otherwise, the backup is only good for restoring individual files that might have been corrupted or overwritten by an upgrade, for example. Still worthwhile, for someone who knows what (s)he’s doing.