Live backup

Help and Support Raspberry Pi
1

I look after a few OSMC systems remotely for people who aren’t too confident about computers & technology. So far so good.

Soon I’ll be looking after a system that’s considerably further away than what I’m currently handling - the other-side-of-the-world far away.

I have multiply redundant communication options in the devices I administer, so usually I can fix even serious connectivity issues [e.g. router loses port forwarding] remotely.

As dropping in to fix a problem I can’t handle remotely isn’t going to be an option, I’d like to keep a ‘live’ backup, so that if something goes drastically wrong [Pi won’t boot], I can just tell the user to switch off, pull out the SD card, plug in the backup card, & power-on again.

I’m figuring that just rsyncing the root and boot partitions won’t be enough - I’ll have to update the bootloader with the kernel image if that’s touched.

Anything else I need to watch out for?

Thought I’d just check if anyone else done this sort of thing before I take it on.

2

Most of what you want is in my script. [How to] Make periodic backups of whole OSMC system

You have tagged this Raspberry Pi for which it’s straightforward. For vero it would need something more complicated.

3

You’d also need the APT package state as well.
For this kind of task, you’d be better off doing a complete block level backup of the underlying media as @grahamh has highlighted.

4

Maybe I’m missing something here but if all your making is a generic stock image for distribution why would you bother with anything more than just configuring one RPi and then pulling the SD and image it off on a PC?

5

I’m not making a stock image for distribution.

I have a system on the other side of the world I’d like to keep regularly backed up so that a person local to that device can perform disaster recovery by booting from that backup because getting the no-longer-working device to me [or me to the device] for fixing will be very time consuming / expensive.

I do not want to end up trying to talk a complete newbie through imaging a card from the other side of the world.

6

Just to be clear, that is indeed enough (for RPi). I found some unexpected behaviours in rsync when implementing multiple backups with hard links. That’s why I call it 3 times. But you won’t be interested in that if you are just after a nightly mirror of the working SD card.

7

To copy all the data rsync will work, but this won’t be sufficient when the kernel image is updated, as the bootloader needs to point to the storage media block where the current image is located.

I’m looking to create a bootable backup.

8

I’m not that familiar with how RPi boots, but in my experience you just stick a SD card in with the necessary files. To me, that means the bootloader is locating the kernel through the filesystem.

9

If you copy both /boot and root partitions with rsync, you have nothing to worry about.

The bootloader simply looks for a kernel image in the right place at filesystem, not block level.

10

Any backup strategy has to take account of the one big problem with Raspberry Pi systems: the potential fragility of SD cards. They can fail completely or, more often, partially. The data and/or filesystem on the card can also become partially corrupted as a result of minor power fluctuations, and sometimes for no clear reason.

So whatever strategy you adopt should try to minimise the possibility of replicating bad data. It should also recognise the possibility that the backup SD card itself might fail first if continually being written to.

My suggestion is therefore to keep things as simple as possible::

  1. Once you have created a baseline bootable copy, complete with initial configuration, don’t try to keep a fully up-to-date mirror copy of the operating system. This will reduce the wear on the backup SD card. As long as it is capable of booting, you can run an APT update to bring it up to date.
  2. Only backup configuration files. If you have a 24/7 server available, however small, tar them up and write them there, again minimising wear and tear on the SD card. (In fact, if you can backup the config files to a server, there’s no need to keep the backup SD card plugged in to anything.)
  3. When the SHTF, plug in the baseline copy, which is hopefully still in pristine condition, boot it up, apply all necessary APT updates and restore config files.
  4. The above strategy should work well where no new packages are likely to be added by the end users, which seems to be the case here.
11

There are really 2 types of backups.

  • Images
  • Versioned files

Image-based backups can make for easier restores, assuming the exact same hardware will be used on the restore-side. With raspberry-pi systems, that is NOT unreasonable. Images are huge and require a 1-for-1 amount of storage. If you need 2 copies, then you need 2x the storage. 3 copies = 3x the storage. Additionally, creating images requires downtime. It is not safe to make an image backup of a running Unix system. Probably 95% of the time, it will work fine, but that other 5% it will not. Consider yourself warned. That means to make an image, you’ll need some other equipment and a different OS which can be booted. For a beginner, I doubt talking them through this is possible.

There are 2 types of OSMC systems.

  1. Those with all the media locally connected and
  2. those with no media directly loaded, so the Pi/Vero only needs configuration information

By setting up the first OSMC so that media is separate from the OS, on different storage, you can remove many complexities and huge file backups. This effectively makes it the same as the second type of OSMC where all the media is on different devices on the network.
Pretty much all the OSMC configuration data is stored in ~osmc/.kodi/ … so backing that up is really the only necessary thing IME.
Let’s look at an OSMC system:
$ df -hT -x squashfs -x tmpfs -x devtmpfs
Filesystem Type Size Used Avail Use% Mounted on
/dev/mmcblk0p2 ext4 30G 5.1G 23G 19% /
/dev/mmcblk0p1 vfat 240M 40M 200M 17% /boot
This has a 32G microSD for storage and all the media is on the network, not local.
$ du -sh /home/osmc/
3.8G /home/osmc/
So of the 5.1G used by the OS and OSMC, 3.8G are for OSMC. I keep downloaded addons in the HOME for the osmc userid, also called HOME or ~/. However, really all the kodi settings are in ~/.kodi/ … which is 3.8G in my system. I just looked. The non-Kodi stuff is just a few MB, not really important.

By backing up just the HOME (i.e. /home/osmc/ ), I’ll effectively get 99% of what is needed. There are some files in /etc/ and /boot/ that would be very helpful, especially if the Raspberry Pi has mpeg2 or h.264 decoding driver licenses, but it isn’t really “mandatory.” In /etc/ the main things to get are any extra fstab entries, but most people wouldn’t mount remote storage using that as kodi has a serviceable GUI.

In /boot/, these files could be locally modified:
/boot/cmdline.txt
/boot/config.txt
/boot and /etc/ are pretty tiny, so I’d just grab everything.

So, a minimal backup would get /boot/, /etc/, and /home/.

OS/Settings Restore - that’s a little harder. Basically, I’d just provide the remote people with 2 microSD cards, setup identically, so all they need to do is swap in the alternate card and reboot, then connect the backup and call me on the phone/jitsi/whatever for help. Ideally, I’d ssh into the system, see the backup storage media, selectively copy a few files from /etc/ and /boot/, then copy everything from /home/osmc/ onto the fresh drive.

The backup storage would likely be using a USB3-to-microSD adapter. Good, fast, ones are $10 or less. I just picked up one a few weeks ago with a USB3.1 interface which writes and reads data as fast as any microSD can handle.

Notice, I’ve completely ignored media files. In my world, those are on an NFS server running Ubuntu headless and backups are simple 1-copy rsync run a few times weekly. Something like this:
ionice rsync -av --stats --progress $EXCLUDES --delete /d/D1/ /d/b-D1/
That is fine for huge files that I simply can’t afford to have more than 2 copies due to sizes.
It is not sufficient for the OS or /home/osmc/ backups. Those really should be versioned and have perhaps 1 backup each week. rsnapshot is a reasonable tool for making versioned backups. The target storage must use a Linux file system, no FAT-whatever or NTFS. EXT4 would be a good choice. A few how-to articles for rsnapshot:

https://www.cyberciti.biz/faq/linux-unix-apple-osx-bsd-rsync-copy-hard-links/
Just configure /boot/, /etc/ and /home/ for the weekly backups and perhaps 5 versions.

Why not backup the OS and all updates? There’s no need. Just install the current minimal, core OS, then a simple patching round will bring the OS back to current. Once every quarter or every 6 months, the remote admin would want to snail-mail 2 microSD cards with updated OS. If media isn’t placed onto that storage, an 8G would be fine. After all, I’m using less than 6G total. If you go crazy, 16G microSD would be fine too, if you can find them. In the last few months, I’ve picked up high-endurance 32G and 64G microSD cards for Pi projects. These were $8 and $12 each. If you are charging for this support service, just bill for new SD storage too.

2 Likes