Kodi improvements upstream is not pursuing

Feature Requests
1

I am a heavy user of OSMC and my database is quite massive, around 1000 TV Shows and 12.000 films. This is in the very limit of a 1GB Raspberry PI (with compressed ZRAM swap). Kodi is very wasteful of resources. For example, it request all the data from the database and then create the internal structures to represent that data (so you use double of memory), instead of reading a row at a time using a database cursor/iterator. That change would HALVE the memory required by Kodi.

There are lots and lots of low hanging fruits, both in terms of memory consumption and performance. But Kodi core team seems not to be interested in pursuing this area. I wonder if OSMC would be more receptive, since OSMC runs in small devices (although a PI 4 with 8 GB is not small!).

I know that OSMC has local patched for the Kodi it uses. Is there a procedure to request the inclusion of new patches, patches already send upstream but not integrated yet (if ever)?.

Just an example, if you feel interest in the area:

Query plan for “season_view” is AWFULLY SLOW #22279

1 Like
2

It’s Christmas - things will be a bit slower. I haven’t seen any reason why this won’t be investigated thoroughly. It looks like the PR was only opened 20 hours ago.

I am a Team Kodi member and will mention it in the correct channels and get you some feedback on this

Sam

2 Likes
3

I am trying now the github way. In fact I am been trying to discuss some changes in kodi for years, with no success. For instance, I think some non critical components in kodi could be rewritten in Python, lowering the entry barrier for new people at not a significant cost since kodi already includes a Python interpreter inside for the addons.

These architectural changes can not be discussed as “issues” in github.

Maybe I was never able to find the right interlocutor to bounce my ideas with…

I am an old guy. I miss mailing lists so much…

1 Like
4

I think most people with a library this large end up running a MySQL db on their file server which is usually better equipped to handle this load. Not that it wouldn’t be great if Kodi’s db wasn’t more efficient, but if your end goal is to not have to wait thirty seconds for a show to display, this would be a way to make that happen.

5

The point, @darwindesign , is that kodi brings in big chunks of the database for many requests, so memory usage is proportional to database size. The database engine doesn’t matter. For instance, in many cases, Kodi requests tons of data to the database and then it does the filtering in C++, instead of delegating that filtering to the database engine and getting only what it needs.

All the database management is quite… ugly in current Kodi. Changing the DB engine is not a cure, because Kodi is not using it efficiently.

I think the main use of a MySQL db is to keep different Kodi instances running the same database (when a film is watched, all devices see it as watched). I don’t think DB size is the motivation.

Of course could be that MySQL query planner could do a better job optimizing this particular request. It was just an example. I have tens if not hundred of other “inefficiencies”.

There are tons of low hanging fruits around. I have NOT found the right venue to discuss them so far :frowning:

2 Likes
6

This has already been done to some extent — for example scrapers are now written in Python.

I’d suggest posting on the Kodi forums.

Sam

7

I second this. With a collection that size, you are much better off on a DB running on a server that has a bit more “oomph” available for it. It’s also better for when you have multiple clients.

8

The trees are hiding the forest. You are getting lost in the details.

My point is that Kodi uses the database in a quite inefficient way. That inefficiency resides in Kodi, not in the database engine. For instance, for many operations, the memory usage in Kodi is proportional to the number of elements in the database. This is not solved changing the database.

I can provide tons of examples. I can provide measurements. Do not ignore my warnings just because your use case is not my use case.

Check the link I posted. Understand the issue. Get the enlightenment when you get the fact that the change I propose increases performance x15. Yes, maybe in this particular case MySQL is able to do the optimization by itself, but this is only an example of MANY and most of them can be traced back to “Kodi should do better”.

I can provide more examples of performance and memory waste if interested. Choose the flavor you want, I have it.

For example:

To get a full film listing Kodi asks the database for the details of ALL the films at once. It load that response for the database in RAM (In my case ~150 MB of RAM just burn) and then Kodi iterates over that data in RAM to build 12.000 objects to display in a screen with at most 12 banners per screen.

Al least Kodi should build those 12.000 display objects (a big waste by itself) while iterating the response from the database, instead of loading all the response in RAM at once. That would save ~150MB of RAM in my current database. That simple change: instead of reading the DB response at once and then create the structs in memory, create the structs incrementally from each row fetch.

This is not dependent of the DB engine used but the way Kodi uses the engine. Changing the engine doesn’t bring back those ~150 MB wasted.

I am suffering those inefficiencies because of my database size, true, but those inefficiencies are real, they are hitting somebody real (me) and that sad soul has invested quite a few hours analyzing the issue and proposing a solution or, at least, a “hack”.

Of course, a valid answer would be “you are way out of the DB size sweet spot and we don’t care about such an extreme use case”. That would be unfortunate, but life sucks and time is short.

1 Like
9

Nothing is getting lost. Kodi has an inefficient database, this has been common knowledge for a long time. As of yet, and after many years, nobody has stepped up to offer to change this over on the Kodi side (or maybe they had and ran into issues that were beyond what they could, or wanted to deal with). There is even a person who years ago posted changes on the Kodi forum of how someone could modify their Kodi MySQL database to speed it up and even made a GitHub for it. Why did this never make it into Kodi? Who knows, but as of right now, with Kodi in its current state, SQLite seems to work less optimally with very large libraries so I thought I would offer this bit of unsolicited advice even if it was slightly off topic. Now that I understand that you take offence to such behavior I offer my sincerest apologies.

10

It would be a pity if people misunderstand each other over this. The OP has highlighted shortcomings in the way the database is used and a problem identifying the right forum to discuss ‘strategic’ issues about Kodi development. Both very valid.

Kodi 20 has had, they tell us, thousands of commits since Matrix but I doubt the average user will see much of a difference. And as we start on Kodi 21 - where is the conversation about what the goals are for this new version? It might be a good time to suggest a bottom-up re-write of db handling.

My guess is that there are database experts who could do a lot better job with the db which has probably ‘just growed’ over the years. But they don’t take any interest in supporting media players. Meanwhile there are C++ devs working hard on Kodi who are more comfortable with that language and tend to treat the db as a flat file interface. If I’m wrong, can we identify the Team Kodi member who takes the most interest in the db and get a conversation going?

2 Likes
11

I use OSMC on RPi4 with remote MySQL on a Windows PC and have seen it getting slower over the years when getting lists.
Yesterday it was extra slow and I had time to log in over SSH and run TOP and I saw that Kodi was using 100% CPU for over 2 minutes … which would seem to match the description from @jcea
If there was a simple and relatively risk free change that could dramatically reduce that I would be keen to try it.

1 Like
12

Sorry, I beg your pardon. English is not my native language and I usually sound like I am angry. Not my intention. Sorry

1 Like
13

I’ve raised this upstream in Kodi Slack as I said I would.

2 Likes
14

If you can produce a patch against Kodi v19.x, I can make a test build available for all supported platforms for OSMC users.

Sam

1 Like
15

Using MySQL would totally take you down the rabbit hole. Not many months ago I was told by this very OSMC team that using the external SQL sever was always considered experimental, so if you use it you are on your own.
Kodi creates the databases with different (smaller) column lengths then fails then leaks memory, then crashes, but noone wants to fix this.
If you fix the DB table columns manually it’s still not perfect.
Some scrapers which come bundled with Kodi and also with OSMC are not graceful with inserting data, will still cause memory leaks and frequent crashes. I was told to go and complain to the scraper developers.
IMHO the unsafe / known defective scrapers shoud be removed, if noone wants to fix them. Or at least marked unsafe / deprecated, and the new safe ones used by default. I learned then that (Python) in the name is supposed to be understood as this is the new, safer one, use this - hard to decode for normal users.
This is the sad state of affairs AFAICS, rant off.

16

I would also sign up for testing this, I also have a large DB (although not that large) :pray:

17

Wow I thought I had a big collection at over 1000 films. I guess not.

Even at 1000 films I found the library situation annoying to work with. Which is why I just use the file system, if you use a third party image downloader you can use that to download all the art work in to the folders. Then when you navigate using the file system using certain skins, it can utilise that art work on the fly. This way there is no need to maintain the database. Although I admit it does present the films in a nice way.

18

I use Kodi/OSMC as a playback device only and don’t keep any media local nor do I mirror the DB or the contents to Kodi. Some of the media in my library needs transcoding on the fly and low-end ARM systems aren’t up to that task.

It wasn’t always that way, until I setup multiple playback devices around the house for music, photos, and video playback. At that point, having a central server made more sense, as keeping content updated across so many different devices and remembering what resolutions and types of content each could handle was a negative W-A-F.

This can be accomplished a few different ways. Kodi has a DNLA controller/renderer which can connect to any DLNA server. That’s the easy way, but DLNA organization kinda sucks - big time. Who came up with their list of virtual directories?

Many Media Servers on other platforms, say Jellyfin, also have a few different Kodi addons available. We tend to use a Jellyfin controller on an old $50 Amazon tablet or smartphone to tell the DLNA renderer on Kodi to play media from the DLNA Server built-into jellyfin. There’s also a Play-To option, if you allow remote applications control over Kodi. OSMC or Kodi are the same in these aspects. Tablets are nice to search for content, much easier than our MCE remote, push that slow, 4 yr old tablet still has life, even if it is blocked from all internet access (privacy) 99.9999% of the time.

Just saying, lots of options, thanks to the existing flexibility.

I miss listsrvs too.

19

I’m disappearing until the New Year.

If @jcea makes a patch available, I’ll produce relevant test builds for all platforms

Cheers

Sam

2 Likes
20

Unfortunately I haven’t seen a patch for this yet, so can’t produce any test builds so we can evaluate the results

Sam