The thumbdrive seemed ok when I just tested it:
osmc@osmc:/media/29GB$ time dd if=/dev/zero of=zero.dat2 bs=1M count=1K conv=fdatasync oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 42.4812 s, 25.3 MB/s
real 0m42.496s
user 0m0.030s
sys 0m1.920s
Just over 42 seconds to write 1GB, though I have my doubts about the way the conv= and oflag= directives are working.
I’ll test it tomorrow.
That’s exactly what it should say, thank you!
The cp: cannot stat … message should only occur for the first backup.
osmc@osmc:~$ time sudo ./osmc-backup-0.1.5
Fri May 19 20:27:32 CEST 2017 Backup version 0.1.5
Going to back up to /media/29GB/osmcbackup/2017-05-19 (unless I can't find it).
looking for mounted media
media found at /media/29GB
20,355,324 KiB available on drive
Counting and deleting backups older than 2017-05-12 or if more than 7 of them
Backup from 2017-05-18 kept
Now 20,355,324 KiB available on drive
system is on /dev/mmcblk0p2
Backing up system partition to /media/29GB/osmcbackup/2017-05-19/system
There are 1 previous backups
Found latest backup at /media/29GB/osmcbackup/2017-05-18/system
Incremental backup: copying new and changed files to /media/29GB/osmcbackup/2017-05-19/system
Making a copy of the new and changed files in /media/29GB/osmcbackup/2017-05-19/system-delta
Cloning unchanged files from previous backup
Deleting files no longer on source
Time to back up system: 00h:17m:04s
boot is on /dev/mmcblk0p1
Backing up boot partition to /media/29GB/osmcbackup/2017-05-19/boot
There are 1 previous backups
Found latest backup at /media/29GB/osmcbackup/2017-05-18/boot
Incremental backup: copying new and changed files to /media/29GB/osmcbackup/2017-05-19/boot
Making a copy of the new and changed files in /media/29GB/osmcbackup/2017-05-19/boot-delta
Cloning unchanged files from previous backup
Deleting files no longer on source
Time to back up boot: 00h:00m:03s
129,128 KiB used in this backup
20,226,196 KiB available on drive
All done!
real 17m48.305s
user 1m2.760s
sys 0m48.010s
osmc@osmc:/media/29GB/osmcbackup$ sudo du -sh *
2.9G 2017-05-18
127M 2017-05-19
So it looks to have worked, taking 17 minutes this time but writing only 127MB to the drive.
Clearly, we have an initial backup and some kind of delta backup. Three questions:
1 Is the delta backup incremental or differential?
2 There’s a message about deleting backups older than a specified date or if more than 7 of them. How does it ensure that a full backup always exists along with the subsequent deltas?
3. Is there a script written for reinstalling to a point in time?
Edit: I think I’ve figured out the answer to Q2. You’ve used hard links to the inodes. Pretty neat!
It therefore follows that the answer to Q1 is differential. (Still figuring out how to restore. ;))
And /media no longer has any iffy mount points left over.
In addition to making a full backup, the script copies just the changed and new files to the delta directory. It is incremental. This is optional - I added it just to check everything was working as expected. Can be interesting if you want to know what was changed in a distro upgrade.
As you have noted, they are not deltas but a full snapshot. The idea is Mike Rubel’s Easy Automated Snapshot-Style Backups with Rsync
Yes, but I haven’t tested it as much as the backup script - more of an alpha than a beta. I’ll do some more checking and post it. Basically, you just delete all files on each of the two partitions and copy the files back from /media/29GB/osmcbackup/[date]/boot and /media/29GB/osmcbackup/[date]/system.
Thanks for the link to Mike Rubel’s article. I’d not seen it before but all of a sudden, now I know about it, i’ve noticed it in a few places, including rsnapshot.org.
Were you aware of rsnapshot? It’s in the Debian repo, for example.
Yes, I looked at a number of ‘ready-made’ backup solutions. They are usually targetted at backing up only user files, and in a multi-user environment which we don’t need. What I was after was something more like the ‘whole card’ backup with dd but which runs from within a working OSMC,
Along the way, I found some quirks of rsync which it was not clear were taken care of by rsnapshot and others based on rsync.
Many thanks for this Graham.
Out of interest, if I do a “diff” to compare the root directory with the “system” backup directory, I find many files in /etc /boot /dev that only exist on the sd card, but not in the backup. Is that intentional?
We don’t backup /dev as the system will re-generate it.
Within system there should be just the boot/ directory, empty, because all the files are in the ‘boot’ backup. /boot/ is just a mountpoint.
In /etc there are some links to other parts of the filesystem. Let me check what I have.
All I get is reports for some broken links (they are broken on both the running filesystem and the backup).
root@osmc:/media/60Gext4/osmcbackup/2017-05-22/system# diff -r etc /etc
diff: etc/alternatives/awk.1.gz: No such file or directory
diff: /etc/alternatives/awk.1.gz: No such file or directory
diff: etc/alternatives/editor.1.gz: No such file or directory
diff: /etc/alternatives/editor.1.gz: No such file or directory
diff: etc/alternatives/nawk.1.gz: No such file or directory
diff: /etc/alternatives/nawk.1.gz: No such file or directory
diff: etc/alternatives/pager.1.gz: No such file or directory
diff: /etc/alternatives/pager.1.gz: No such file or directory
diff: etc/alternatives/pico.1.gz: No such file or directory
diff: /etc/alternatives/pico.1.gz: No such file or directory
diff: etc/alternatives/w.1.gz: No such file or directory
diff: /etc/alternatives/w.1.gz: No such file or directory
diff -r etc/fake-hwclock.data /etc/fake-hwclock.data
1c1
< 2017-05-22 09:17:01
---
> 2017-05-22 10:17:01
Can you post your diff output, please?
is it possible to rsync to a nfs device with this?
I don’t see why not, but I don’t use nfs so not entirely clear how nfs drives are addressed. If the mount point for your nfs share is in /media or /mnt then the script won’t care and should work fine. It won’t handle URIs like smb:// or talk to a remote rsync server.
i just added nfs to this list: FSTYPES=(ext2 ext3 ext4 xfs reiserfs)
and changed the default backup directory to a folder of my nfs mount.
seems to be working … is there a way to check if the backup is complete when its done?
cheers
b
You can use diff and/or meld to compare the backup with the originals.
Let me know if it works!
so far i got this errors:
osmc@B2VERO:~$ sudo ./rsync-backup
Sam Jun 10 19:55:24 CEST 2017 Backup version 0.1.6
Going to back up to /mnt/b2nas/osmc-backup/B2VERO/2017-06-10 (unless I can’t find it).
looking for mounted media
media found at /mnt/b2nas
23’333’056 KiB available on drive
Counting and deleting backups older than 2017-06-03 or if more than 7 of them
Now 23’333’056 KiB available on drive
system is on /dev/vero-nand/root
Backing up system partition to /mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system
Backup directory is empty - making first copy
rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/bin/.systemd-detect-virt.Y61kNI"",“security.capability”) failed: Operation not supported (95)
rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/lib/kodi/.kodi.bin.iLfhi1"",“security.capability”) failed: Operation not supported (95)
rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/bin/.systemd-detect-virt.Y61kNI"",“security.capability”) failed: Operation not supported (95)
rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/lib/kodi/.kodi.bin.iLfhi1"",“security.capability”) failed: Operation not supported (95)rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/bin/.systemd-detect-virt.Y61kNI"",“security.capability”) failed: Operation not supported (95)
rsync: rsync_xal_set: lsetxattr(""/mnt/b2nas/osmc-backup/B2VERO/2017-06-10/system/usr/lib/kodi/.kodi.bin.iLfhi1"",“security.capability”) failed: Operation not supported (95)rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]
Ah, Vero! I don’t know exactly how Vero’s file system is set up. Can you post findmnt -l for me, please?
Can’t look at it tonight but will have a look tomorrow.
TARGET SOURCE FSTYPE OPTIONS
/dev devtmpfs devtmpfs rw,relatime,size=406496k,nr_inodes=101624,mode=755
/proc proc proc rw,relatime
/sys sysfs sysfs rw,relatime
/run tmpfs tmpfs rw,relatime
/ /dev/vero-nand/root
ext4 rw,relatime,stripe=1024,data=ordered
/sys/kernel/security securityfs securityfs rw,nosuid,nodev,noexec,relatime
/dev/shm tmpfs tmpfs rw,nosuid,nodev
/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=5,mode=620
/run/lock tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k
/sys/fs/cgroup tmpfs tmpfs ro,nosuid,nodev,noexec,mode=755
/sys/fs/cgroup/systemd cgroup cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
/sys/fs/cgroup/bfqio cgroup cgroup rw,nosuid,nodev,noexec,relatime,bfqio
/mnt/da2nas systemd-1 autofs rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/mnt/b2nas systemd-1 autofs rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/proc/sys/fs/binfmt_misc
systemd-1 autofs rw,relatime,fd=25,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/dev/mqueue mqueue mqueue rw,relatime
/sys/kernel/debug debugfs debugfs rw,relatime
/sys/fs/fuse/connections
fusectl fusectl rw,relatime
/mnt/b2nas 192.168.3.38:/share/Qdownload
nfs rw,nosuid,nodev,noexec,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=14,retrans=2,sec=sys,mountaddr=192.168.3.
/mnt/da2nas 192.168.11.1:/share/Download
nfs rw,nosuid,nodev,noexec,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=14,retrans=2,sec=sys,mountaddr=192.168.11
/run/user/1000 tmpfs tmpfs rw,nosuid,nodev,relatime,size=81516k,mode=700,uid=1000,gid=1000
osmc@B2VERO:~$Hmmm. If this is a Vero2 I don’t think I can help. I might be able to figure out the backup but not the restore without having a device to hand. I might just get myself a V4k, but suspect that’s different.
its a vero2, yes.
I have a feeling that the errors in post #34 are related to NFS. I’ll see if I can reproduce the problem on a Pi, so we can see whether it’s Vero- or NFS-related.
I have a Vero 4K myself but believe that the Vero2 uses a similar system for the root partition, which is to say that it uses LVM as a means of consolidating multiple block devices into a single partition. From a backup perspective, the formatted partition is /dev/vero-nand/root.
osmc@osmc:~$ sudo lvdisplay --map
--- Logical volume ---
LV Path /dev/vero-nand/root
LV Name root
VG Name vero-nand
LV UUID dmPd6p-6iBl-qZ0g-taeV-yWmu-ByEA-2ZeAo7
LV Write Access read/write
LV Creation host, time osmc-installer, 2015-01-01 01:00:15 +0100
LV Status available
# open 1
LV Size 14.29 GiB
Current LE 3658
Segments 4
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:0
--- Segments ---
Logical extents 0 to 3148:
Type linear
Physical volume /dev/data
Physical extents 0 to 3148
Logical extents 3149 to 3403:
Type linear
Physical volume /dev/system
Physical extents 0 to 254
Logical extents 3404 to 3530:
Type linear
Physical volume /dev/cache
Physical extents 0 to 126
Logical extents 3531 to 3657:
Type linear
Physical volume /dev/instaboot
Physical extents 0 to 126Thanks for that. My point above is that even if we had a ‘full’ backup, the process of restoring it is more complicated than just copying a load of files onto an SD card. If you can crack that, then all good. Otherwise, the backup is only good for restoring individual files that might have been corrupted or overwritten by an upgrade, for example. Still worthwhile, for someone who knows what (s)he’s doing.