Speed improvements for music lib changes

Hi,
Apologies if this has been covered elsewhere, but is the new V significantly faster than the 4K+ when detecting additions to a music library?
My main lib is approx 1.7tb, all flac files (~65k), & if I do a full scan, to detect changes, it takes hours to run.

Obviously I can narrow it down, but then need to identify which sub-libs (eg artists) I’ve added a new album into, to then do context-sub-scans.
thanks

Yes, library scraping is improved. The eMMc is faster and the CPU is faster so both IO and scraping speed will be improved.

Someone did some benchmarks a while ago: OSMC Vero V Review - #215 by jeff2, although this was using a specific video add-on.

thanks Sam :+1:

You are welcome.
I remember you were unsure of how to backup and restore from the existing Vero. We can advise on this when you receive your device if you have any problems.

Hours seems a lot. A scan of my music library with 43,564 songs takes ~10-20 min depending on the device on a shared MySQL library and the files stored on a network share. Are you connected via wireless or have some bottleneck in your network that might be slowing things down? Kodi is slow for large music library scans to be sure, and the initial library build takes ages, but litteral hours for an update scan doesn’t seem right on any device that OSMC run on.

1 Like

It depends if he’s doing a full scan each time, but agree.

I’ve added a new album to the lib (has approx 64k tracks in total). Started a full scan just over an hour ago, & it’s maybe 45% done (doing the “L’s” now).

Connected via ethernet, with the flac lib on a single drive on the network.

Can’t remember exactly but there were advancedsettings options to configure scraping and different hashing options if I recall.

just to share my experience

  • 22k flacs on a Synology DS223, nfs
  • update library without new files, all connected via ethernet

Vero 4k+: 56 seconds
Vero V: 46 seconds

tried another Vero 4k+ but this time connected via Wifi to an AVM Mesh network: 133 seconds

Seeing the extreme large number (not even reaching 45% with over 1 hour runtime), it suggests to me, there is something odd like

  • something is broken with the SQLite db if it does not scale
  • the used NAS could be the bottleneck (what brand, model, share method is used)
  • something permanently changes the flacs on the NAS (perhaps only the modification date) and lets re-scrape all flacs by that on every update library run

Just to set expectations right: For sure the Vero V gives you an improvement … but with that extreme runtimes for an update library of the music db, the real bottleneck should be understood and corrected first.

1 Like

Hi - that’s useful info. To clarify, it’s not NAS, just a WD Elements 4tb shared drive on Win 11 m/c. The files don’t get updated, only new ones added, & are arranged as artist/album/track hierarchy.

Not sure what you mean by SQLite, I just have a “standard” installation, don’t have any custom / additional features in place (other than a single entry in advancedsettings (changes m/c name → IP).

Any ideas how I could investigate further (bearing in mind my knowledge of Linux,etc is minimal…)?

To get a better understanding of the problem you are experiencing we need more information from you. The best way to get this information is for you to upload logs that demonstrate your problem. You can learn more about how to submit a useful support request here.

Depending on the used skin you have to set the settings-level to standard or higher, in summary:

  • enable debug logging at settings->system->logging

  • reboot the OSMC device twice(!)

  • reproduce the issue (start a music update library and let it run 30-60 seconds)

  • while that update library still runs, upload the log set (all configs and logs!) either using the Log Uploader method within the My OSMC menu in the GUI or the ssh method invoking command grab-logs -A

  • publish the provided URL from the log set upload, here

Let us know if you have questions to this debug collection or you need more details.

OSMC skin screenshot:

ok, here’s the log:

https://paste.osmc.tv/elabixexod

thanks!

ok, this might point to the root cause for the behaviour you see

2024-07-15 07:32:10.348 T:3228    debug <general>: DoScan Scanning dir 'smb://Acer-mb/flac/' as not in the database
2024-07-15 07:32:10.509 T:3228    debug <general>: DoScan Scanning dir 'smb://Acer-mb/flac/4 Non Blondes/' as not in the database
2024-07-15 07:32:10.904 T:3228    debug <general>: DoScan Scanning dir 'smb://Acer-mb/flac/4 Non Blondes/Bigger, Better, Faster, More!/' as not in the database

For EVERY folder the scraper finds and touches, we see the message that this folder is not in the database means everything has to completely scraped (again).

You’re using the generic.album.scraper and generic.artitst.scraper (I never used) and which makes sense if you have MusicBrainz ids tags in your flacs.

Still not sure, why this happens at your system and why it is configured as currently found. Perhaps, I can continue this evening MEST but all others are welcome to help.

1 Like

Thanks Jim, as there have been no changes to the lib since the last full scrape, baffled as to why everything is flagged as not in library?

The folder “smb://Acer-mb/flac” is my main music lib, so everything in there I’d expect to be already in the Vero /Kodi db.

I see a path substitution active in a local advancedseettings.xml file, with mapping

====================== advancedsettings.xml =================== C7hKmH1p
<advancedsettings>
 <pathsubstitution>
  <substitute>
    <from>smb://Acer-mb/</from>
    <to>smb://192.168.1.99</to>
  </substitute>
 </pathsubstitution>
</advancedsettings> 

What is the history and intention for this?

I was having issues with Vero not recognising the DNS naming, so Darwin suggested I have this in place.

However, I was having the same performance issues prior to adding this setting.

It is working as you can see in the logs that it is substituting the IP address in while accessing files. I do wonder if the DNS issues were maybe causing an issue. It is kind of hard to say from that log as the scan only encompases around a minute of scan time and I don’t personally see too much there that I find all that insightful. Are you sure these albums it is adding in were actually already in the music library before you told it to scan? If you had told Kodi to clean the music library while you were having DNS problems there could have been removals from that. I do notice that there isn’t the usual messages about skipping folders due to no change. A typical update not taking ages is from it not having to look at most files. Kodi stores a folders modification date in its database and then during the scan it compares that to what the current value for the folder is and skips looking at the files if it doesn’t see a change. Your log isn’t showing that happening.

I think I’d be tempted to just remove all my music sources, delete the music database from the file system just in case there had been any corruption, and then reboot and configure the sources again but use the IP address this time (the substitution in advancedsettings doesn’t need to be removed) since if your starting over there is no point in not just using the more stable source path. The initial scan of the library is going to take a very long time, but updates should hopefully happen fast after that.

1 Like

thanks (again!) guys for all your help. I can try the above, ie full clear-down & re-define / re-build, but can you advise how I delete the existing music lib please?

(Assuming I’ll use Putty to connect),

thanks!

Sure, assuming you logged in as user osmc via ssh (using a putty client, etc.)

  1. sudo systemctl stop mediacenter
  2. rm /home/osmc/.kodi/userdata/Database/MyMusic*.db
  3. sudo systemctl start mediacenter

Personally, I would always avoid storing IP addresses in a DHCP environment. If DNS resolution does not work, the nameserver problem must be solved and not a solution “worked around” it. The file with these replacement details is located at /home/osmc/.kodi/userdata/advancedsettings.xml … I would remove it (as step 2.b after removing the music db and before restarting the mediacenter).

@mickb How far have you come with it?