Issue backing up OSMC

clude · 2 June 2024 09:59

Hello,

Backup is not finishing correctly " shutil.copyfile" cannot move the backup archive from local temp directory to the remote smb directory.

The remote directory seems reachable since I’m able to configure it from the backup interface configuration page. And from CLI :

~/.kodi/userdata$ ping ds.lan
PING ds.lan (192.168.0.18): 56 data bytes
64 bytes from 192.168.0.18: seq=0 ttl=64 time=1.848 ms
64 bytes from 192.168.0.18: seq=1 ttl=64 time=2.060 ms

The kodi.log :

2024-06-02 11:44:25.446 T:6228    debug <general>: script.module.osmcsetting.updates[service_entry] : update_args - {'percent': 100, 'heading': 'OSMC Backup', 'message': 'Transferring backup file'}
2024-06-02 11:44:25.447 T:6228    debug <general>: script.module.osmcsetting.updates[osmc_backups] : local tarball name: /tmp/osmc_backup/OSMCBACKUP_2024_06_02_11_40_49.tar.gz
2024-06-02 11:44:25.447 T:6228    debug <general>: script.module.osmcsetting.updates[osmc_backups] : local tarball exists: True
2024-06-02 11:44:25.448 T:6228    debug <general>: script.module.osmcsetting.updates[osmc_backups] : remote tarball name: smb://osmc:<password>@ds.lan/video/OSMCBACKUP_2024_06_02_11_40_49.tar.gz
2024-06-02 11:44:25.449 T:6228    debug <general>: script.module.osmcsetting.updates[osmc_backups] : remote tarball exists: False
2024-06-02 11:44:25.449 T:6228    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Copying /tmp/osmc_backup/OSMCBACKUP_2024_06_02_11_40_49.tar.gz to smb://osmc:<password>@ds.lan/video/OSMCBACKUP_2024_06_02_11_40_49.tar.gz using shutil.copyfile
2024-06-02 11:44:25.452 T:6228    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Creating tarball failed
2024-06-02 11:44:25.452 T:6228    debug <general>: script.module.osmcsetting.updates[osmc_backups] : FileNotFoundError
2024-06-02 11:44:25.453 T:6228    debug <general>: script.module.osmcsetting.updates[osmc_backups] : (2, 'No such file or directory')
2024-06-02 11:44:25.457 T:6228    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Traceback (most recent call last):
                                                     File "/usr/share/kodi/addons/script.module.osmcsetting.updates/resources/lib/osmcupdates/osmc_backups.py", line 531, in create_tarball
                                                       shutil.copyfile(local_tarball_name, remote_tarball_name)
                                                     File "/usr/lib/python3.9/shutil.py", line 264, in copyfile
                                                       with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
                                                   FileNotFoundError: [Errno 2] No such file or directory: 'smb://osmc:<password>@ds.lan/video/OSMCBACKUP_2024_06_02_11_40_49.tar.gz'

Thank you for your help

darwindesign · 2 June 2024 13:53

The issue is not on your end. I was able to reproduce the same issue and have reported it to the development team that maintains that add-on.

In the meantime it should work using locally attached storage (if you plug in a drive it should show up in Kodi browsing root filesystem>media), a system mount pointing to your network location [more information], or using the “Backup” add-on found in Kodi’s repository as an alternative to OSMC bespoke utility.

clude · 2 June 2024 14:05

Thank you @darwindesign for your quick answer. Can you share with me the ticket/issue where you reported this issue to the maintainer team of the addon?

darwindesign · 2 June 2024 14:11

There is no ticket. I directly reported it backchannel.

darwindesign · 2 June 2024 14:23

A fix for this issue has been submitted here…

clude · 2 June 2024 14:50

Wow everything is so fast around.

Thank you @darwindesign (autofs and path translation are good option that might be usefull in the future).
Thank you @anxdpanic for your fix I’ve manually updated the script, rebooted and backed up successfully my Vero4k. I hope it will soon end up in the next OSMC updates.

clude · 2 June 2024 14:56

@anxdpanic the backup is successufully uplodaded but a popup show up saying the copy failed here is the log:

2024-06-02 16:47:07.473 T:3020    debug <general>: script.module.osmcsetting.updates[service_entry] : update_args - {'percent': 100, 'heading': 'OSMC Backup', 'message': 'Transferring backup file'}
2024-06-02 16:47:07.473 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : local tarball name: /tmp/osmc_backup/OSMCBACKUP_2024_06_02_16_43_35.tar.gz
2024-06-02 16:47:07.473 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : local tarball exists: True
2024-06-02 16:47:07.474 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : remote tarball name: smb://osmc:<password>@ds.lan/video/OSMCBACKUP_2024_06_02_16_43_35.tar.gz
2024-06-02 16:47:07.475 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : remote tarball exists: False
2024-06-02 16:47:07.476 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Copying /tmp/osmc_backup/OSMCBACKUP_2024_06_02_16_43_35.tar.gz to smb://osmc:<password>@ds.lan/video/OSMCBACKUP_2024_06_02_16_43_35.tar.gz using shutil.copyfile
2024-06-02 16:47:07.477 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Failed to copy /tmp/osmc_backup/OSMCBACKUP_2024_06_02_16_43_35.tar.gz to smb://osmc:<password>@ds.lan/video/OSMCBACKUP_2024_06_02_16_43_35.tar.gz using shutil.copyfile
2024-06-02 16:47:07.478 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Copying /tmp/osmc_backup/OSMCBACKUP_2024_06_02_16_43_35.tar.gz to smb://osmc:<password>@ds.lan/video/OSMCBACKUP_2024_06_02_16_43_35.tar.gz using xbmcvfs.copy
2024-06-02 16:47:07.486 T:3020  warning <general>: SMBFile::OpenForWrite() called with overwriting enabled! - smb://USERNAME:PASSWORD@192.168.0.18/video/OSMCBACKUP_2024_06_02_16_43_35.tar.gz
2024-06-02 16:48:08.575 T:3020    debug <general>: CSMBFile::Close closing fd 10000
2024-06-02 16:48:08.880 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Failed to copy /tmp/osmc_backup/OSMCBACKUP_2024_06_02_16_43_35.tar.gz to smb://osmc:<password>@ds.lan/video/OSMCBACKUP_2024_06_02_16_43_35.tar.gz using xbmcvfs.copy
2024-06-02 16:48:08.881 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Copying /tmp/osmc_backup/OSMCBACKUP_2024_06_02_16_43_35.tar.gz to smb://osmc:<password>@ds.lan/video/OSMCBACKUP_2024_06_02_16_43_35.tar.gz using subprocess.Popen
2024-06-02 16:48:09.242 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Failed to copy /tmp/osmc_backup/OSMCBACKUP_2024_06_02_16_43_35.tar.gz to smb://osmc:<password>@ds.lan/video/OSMCBACKUP_2024_06_02_16_43_35.tar.gz using subprocess.Popen
2024-06-02 16:48:09.242 T:3020    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Transfer of backup file not successful: False

darwindesign · 2 June 2024 20:43

Normally not quite this fast. I guess you just caught us in a particularly choice moment.

It should be in the next test build update. Changes get merged rather fast there. Assuming all is good it will be part of the Omega stable release.

I’ll make a note in backchannel where he will see it. I don’t think he spends too much time in this forum. Thanks for the feedback.

sam_nazarko · 2 June 2024 21:19

Don’t say that

Lewpy · 21 July 2024 11:54

I have exactly the same issue once I altered osmc_backups.py to match the fix.
The backup is actually copied to the SMB location, but the script pops up a dialogue to say it failed to complete.
This is confirmed by immediately going to Manual Restore and the new tarball file is shown in the list. of restores.
My suspicion is that the calls to os.path.isfile aren’t working for the SMB file locations, but don’t know for sure (I have next to zero direct Python knowledge, only other languages).

success = os.path.isfile(remote_tarball_name)

Happy to have backups working again, and the pop-up is just a cosmetic annoyance at the moment

Update
Okay, I had a stab at fixing the issue: I’ve resolved it, but can’t say it is the “correct” fix, so will let other’s decide on that

A quick Internet search confirmed my suspicion that os.path.isfile only works for local filesystems, and not remote paths.
So I then looked for an alternative way of doing the same check.
Since the code was already using xbmcvfs, I checked that and found it has a .exists function that does basically the same thing, so substituted that in to the code section already using xbmcvfs.
Since the line numbering has been altered for the fixed code, and I don’t have any fancy CVS diff tool, I’ll just show the modified section here (starting at line 538 in the modified file osmc_backups.py file)

Old Code

            if not success:
                log('Failed to copy {local} to {remote} using shutil.copyfile'
                    .format(local=local_tarball_name, remote=remote_tarball_name))
                log('Copying {local} to {remote} using xbmcvfs.copy'
                    .format(local=local_tarball_name, remote=remote_tarball_name))
                _ = xbmcvfs.copy(local_tarball_name, remote_tarball_name)
                xbmc.sleep(300)
                success = os.path.isfile(remote_tarball_name)

Modified Code

            if not success:
                log('Failed to copy {local} to {remote} using shutil.copyfile'
                    .format(local=local_tarball_name, remote=remote_tarball_name))
                log('Copying {local} to {remote} using xbmcvfs.copy'
                    .format(local=local_tarball_name, remote=remote_tarball_name))
                _ = xbmcvfs.copy(local_tarball_name, remote_tarball_name)
                xbmc.sleep(300)
                success = xbmcvfs.exists(remote_tarball_name)

Now my logs show that the backup was successful, and it now also honours the number of backup tarballs and deletes the old ones.

2024-07-21 13:04:08.659 T:3016    debug <general>: script.module.osmcsetting.updates[osmc_backups] : local tarball name: /tmp/osmc_backup/OSMCBACKUP_2024_07_21_13_03_30.tar.gz 
2024-07-21 13:04:08.659 T:3016    debug <general>: script.module.osmcsetting.updates[osmc_backups] : local tarball exists: True 
2024-07-21 13:04:08.659 T:3016    debug <general>: script.module.osmcsetting.updates[osmc_backups] : remote tarball name: smb://nas01/osmc/backups/OSMCBACKUP_2024_07_21_13_03_30.tar.gz 
2024-07-21 13:04:08.659 T:3016    debug <general>: script.module.osmcsetting.updates[osmc_backups] : remote tarball exists: False 
2024-07-21 13:04:08.659 T:3016    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Copying /tmp/osmc_backup/OSMCBACKUP_2024_07_21_13_03_30.tar.gz to smb://nas01/osmc/backups/OSMCBACKUP_2024_07_21_13_03_30.tar.gz using shutil.copyfile 
2024-07-21 13:04:08.659 T:3016    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Failed to copy /tmp/osmc_backup/OSMCBACKUP_2024_07_21_13_03_30.tar.gz to smb://nas01/osmc/backups/OSMCBACKUP_2024_07_21_13_03_30.tar.gz using shutil.copyfile 
2024-07-21 13:04:08.659 T:3016    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Copying /tmp/osmc_backup/OSMCBACKUP_2024_07_21_13_03_30.tar.gz to smb://nas01/osmc/backups/OSMCBACKUP_2024_07_21_13_03_30.tar.gz using xbmcvfs.copy 
2024-07-21 13:04:08.660 T:3016  warning <general>: SMBFile::OpenForWrite() called with overwriting enabled! - smb://USERNAME:PASSWORD@192.168.0.160/osmc/backups/OSMCBACKUP_2024_07_21_13_03_30.tar.gz
2024-07-21 13:04:10.258 T:3016    debug <general>: CSMBFile::Close closing fd 10000
2024-07-21 13:04:10.562 T:3016    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Backup file successfully transferred 
2024-07-21 13:04:10.581 T:3016    debug <general>: script.module.osmcsetting.updates[osmc_backups] : Removing these files: ['OSMCBACKUP_2024_05_21_04_15_10.tar.gz', 'OSMCBACKUP_2024_05_22_04_15_09.tar.gz', 'OSMCBACKUP_2024_05_23_04_15_10.tar.gz'] 
2024-07-21 13:04:10.582 T:3016    debug <general>: script.module.osmcsetting.updates[service_entry] : update_args - {'percent': 100, 'heading': 'OSMC Backup', 'message': 'Removing old backup file: OSMCBACKUP_2024_05_21_04_15_10.tar.gz'} 
2024-07-21 13:04:10.586 T:3016    debug <general>: script.module.osmcsetting.updates[service_entry] : update_args - {'percent': 100, 'heading': 'OSMC Backup', 'message': 'Removing old backup file: OSMCBACKUP_2024_05_22_04_15_09.tar.gz'} 
2024-07-21 13:04:10.591 T:3016    debug <general>: script.module.osmcsetting.updates[service_entry] : update_args - {'percent': 100, 'heading': 'OSMC Backup', 'message': 'Removing old backup file: OSMCBACKUP_2024_05_23_04_15_10.tar.gz'}

sam_nazarko · 21 July 2024 17:43

Nice – we’ll check this out.

CC @anxdpanic

Sam

Lewpy · 24 July 2024 06:23

I got to grips with Github and submitted a pull request for the change, and modified the other instances under @anxdpanic guidance

digitalHUGHES · 9 August 2024 01:09

I just recently realized backups stopped working for me so I’m looking forward to the fix. They seem to package up fine but then something fails on the copying to network storage. Hopefully this is the same issue. Thanks for the hard work!

darwindesign · 9 August 2024 01:43

You might try deleting ~/.kodi/temp/* and rebooting first to see if that fixes your issue but if you want the fix faster you can try the Kodi v21 test build which includes this fix…

Lewpy · 31 August 2024 08:09

Have you updated to the new August release with Kodi 21.1? The backup fix is in there and is still working for me (regular 4am backups each night to NAS), and it would be good to know if it has helped others with the issue too