I like to turn on all the information and art scraping I can get to have a better choice for my shows and movies.
I have been experiencing crashes on Kodi 19.3 that did not happen with Kodi 18.x and MariaDB 10.5: turns out upon scraping:
c06 and maybe c11 columns of the tvshow table, as well as
c08 and maybe c20 columns of the movie table
have to be modified from TEXT to MEDIUMTEXT type, similar to this: ALTER TABLE MyVideos116.tvshow MODIFY COLUMN c06 mediumtext DEFAULT NULL NULL; ALTER TABLE MyVideos116.tvshow MODIFY COLUMN c11 mediumtext DEFAULT NULL NULL; ALTER TABLE MyVideos116.movie MODIFY COLUMN c08 mediumtext DEFAULT NULL NULL; ALTER TABLE MyVideos116.movie MODIFY COLUMN c20 mediumtext DEFAULT NULL NULL;
(I have actually used HeidiSQL for this)
First BUG: even if some or many SQL UPDATEs fail OSMC should not crash with a sad face, but handle the errors gracefully. This is a MAJOR stability issue, my updated Vero 4k+ crashes multiple time, used to be much more stable on the older Kodi version.
Quick and dirty: OSMC could carry a patch after DB creation if DB is MySQL (MariaDB). This way I would not have to re-patch these tables every time there is a DB version upgrade (which I assume also truncates these fields upon upgrade. Which may cause problems because URLs are parsed out of these i believeā¦
One better would be to handle these differences in the back-end as some Kodi forum posts suggest it, and try to upstream this change.
Even after the fixes I see significant CPU and memory use during scanning and eventually get a āSad faceā during library update (maybe some other errors are also not handled gracefuly?):
This is all dependent on Kodiās code responsible for the database. OSMC is not at is not responsible for itās function or any bugs. Iād you wish to request the issues be addressed by the maintainers of the code responsible then you should report your issues and findings on the Kodi forums.
We believe this is a Kodi issue or a specific Kodi function and as such you should look at addressing the issue there. If you do believe that this is an OSMC specific issue, please let us know.
Excuse me?
1: this is not (AFAIK) an open source project where you can just point me at upstream: you did take an open source project for free (AFAIK) but you sold me your hardware-software combined multimedia player product for profit! (actually two of them)
2: you have modified Kodi from the gui to the deepest layers, how shoud I know where is it wrong? I just see the complete box freezing up rather often. Go, figure, if itās any of the components you chose to incorporate - fix it or use something else.
3: I have actually provided a fix for the database related problem. You shoud carry it take it for integration to your vendor. Not that you say your product does not work with mySQL / MariaDB: currently the opposite is documented.
4: as I (a) descrcribed (b) screenshot and (c) provided debug logs for: even after the DB updates are fixed something else is still going wrong when updating the library, taking kswapd to 90%+ CPU and then the complete system ends up sad face. Please advise how to try to fix it or what additional information do you need?
MySQL support in Kodi after a decade is still considered to be experimental and is not used by default out of the box.
Which scraper are you using?
Do you have a link to such a post so we can check this?
Regarding the crash from your logs. The device seems to be running out of memory and Kodi is terminated due to OOM
Feb 04 10:24:47 osmc2 kernel: Out of memory: Kill process 3522 (kodi.bin) score 594 or sacrifice child
Feb 04 10:24:47 osmc2 kernel: Killed process 3522 (kodi.bin) total-vm:1850464kB, anon-rss:1208340kB, file-rss:0kB, shmem-rss:0kB
Feb 04 10:24:47 osmc2 kernel: oom_reaper: reaped process 3522 (kodi.bin), now anon-rss:4kB, file-rss:0kB, shmem-rss:0kB
You have a number of autofs shares. Iām wondering if that could be a cause.
Letās focus on the OOM, I will dig up the DB fix suggestion tomorrow.
Yes, I have many file systems and the NFS paths are substituted with local paths mounted by autofs.
But how could this accumulate memory used in kodi.bin, growing up to use 1.2-1.4GB RAM?
What kind of memory leak in kodi.bongets heavily activated during library updates but otherwise let the player play videos all day?
Any chance there is no proper free-up after the many failed curl requests? Or the failed thumbnail generations?
I am surprised this is not visible from the logs. The UHDmovies-J source that I scanned for new content to reproduce sad face by OOM uses Universal Movie Scraper.
It always scrapes a few more movies then dies again.
I donāt understand the relevance of this link @darwindesign: even in my first message I coped the alter table fixes frome this link, except I also added two other columns that may overrun.
But at least you dug up the sentence I referred to and @sam_nazarko asked about:
You can rewrite the SQL in the DB backend. This is already done for other small differences between MySQL and SQLite.
We were already past this with @sam_nazarko yesterday, but let me state it for the third time: the SQL UPDATEs are fixed by the table alterations I have already provided, but the OSMC box STILL goes OOM and kills the runaway process, that is kodi.bin, even after the DB is fixed.
If you had read through the entire bug report you should have seen that the Kodi team are aware of the issue and have decided to NOT alter the table sizes but instead address it on the scraper side such that they donāt try to dump an excessive amount of data in a field in the first place. You should have also seen that this was addressed with the TVDB and TMDB python scrapers only so if you have an issue with a single show/movie that is causing a scraping issue the fix would be toā¦
Use the TVDB or TMDB python scraper for at least the problematic show/movie
Use some program like Tiny Media Manager to make a nfo file that doesnāt have the excessive amount of links causing a problem.
Take the issue up with the maintainers of the Universal Movie Scraper add-on since that is where the Kodi team decided this issue needed to be addressed.
What you shouldnāt do is press a downstream distro to make a change to an upstream project that runs counter to how the upstream project has already decided to address the issue.
It may also be possible that if you scrape on a PC this may get around your problematic show and the memory issue since you have already altered the table size.
The DB column lengths are no longer causing any errors. And update library still runs into Sad Face. That is the unsolved issue to focus on. Do I really need to create a new topic for you to stop dwelling on the resolved errors? Any idea what may still be causing the OOM, after there are no more SQL errors? Is there any way to profile the memory usage of kodi.bin to pinpoint which component is ballooning? What is not releasing the memory after each movie scraped?
OSMC sold me two Vero 4k player products (hardware with software) it is not an opensource distribution that can just say go and bother the upstream projects (BTW including choosing the hardware memory to accommodate running the htpc software of your choice). Or do you mean to say they purchased product has no support and OSMC is an independent project?
You identified the issue as you scanning in content that has more information that what Kodi was able to handle without issue. It seems that now now assume that because you altered your db to handle more info that this information will not cause any issues with any other part of the scraping process. Change your scraper to one of the python ones and see if the out of memory issue goes away. If it does then you have your answer.
Obviously OSMC provides support for things that are under their purview. The Vero is a piece of hardware that runs an open source OS designed mainly to run a particular piece of open source software (Kodi). Like any other project of this type there are different groups of people who maintain different pieces of the system. In the case of OSMC the parts of Kodi that are tweaked are mostly to do with audio and video playback as they are platform specific. The parts of Kodi that deal with the database and scraping are not being modified by OSMC at all and are the exact same code that is being used by Kodi on every other platform.
That is incorrect: it is not Kodi that had issue with ācontent that has more informationā. Instead it is MariaDB that was configured to not be able to fit into the designated column. Kodi, with the same column type (TEXT) on SQLite handles the same amount of information just fine (See Implementation Limits For SQLite ā 10^9 bytes). So can we rather focus on the Kodi OOM please?
Is the Universal Movie Scraper (by Team Kodi BTW) not working / not supported in OSMC? If so why not disable / blacklist / warn about it at least?
BTW is this scraper not using python? What is the difference, can I read up on why others are preferred by OSMC? (My current preference is just based on which scrapes the most information, actors, images, fanart, ratings etc. for a given movie / TV show.)
If OSMC is an opensource project, can you please point me at your open source (githut etc.)?
I still am reasonably sure I did purchase my Vero from OSMC: Store - OSMC - if this is not the forum to ask for support for it (because this is only for the opensource OSMC projectā¦) then please point me to the Vero4k+ support!
What exactly do you mean by the āpurviewā of OSMC? Kodi is one of the many (although central) component provided in every version of the OSMC download packages. Why any part of that package is not in your āpurviewā?
Also, OSMC does not seem to be limited to be tweaking āmostly audio and video playbackā as it has a complete customer UI, config, update, log-grab etc. so I still think you do touch everything from the UI down to the drivers, but I will check once you point me to the OSMC code.
This way I can also take a look at the DB and scraping code of OSMC, although even if you do not change it I assume there can be interactions with other parts of the code (say thumbnail generation) that can still OOM.
Do you have a list of all the things that you feel like not supporting? MariaDB? Universal Movie Scraper? All Kodi issues? Anything else?
Change the scraper to TMDB for the source and see if the issue goes away. If it does and you want to go back to using the Universal Movie Scraper then you would need to bring up the issue on Kodiās forum or via a bug report on their Github. The āpythonā name is just how they call out the distinction between the old versions of the scrapers and the new ones. There is no scraper that is āpreferred by OSMCā as that is not something that OSMC touches. I think that currently the Universal Movie Scraper is a bit more prone to problems than the others.
OSMC is on Github just like Kodi is. OSMC is an OS that runs Kodi. Kodi is a separate entity. The custom UI is just a skin (add-on) and isnāt a modification to Kodi itself and the customer can use any skin they want that is available for Kodi. The only parts of Kodiās config that are not using Kodiās stock have to deal with platform specific audio and video. The Kodi part of updates, which are add-ons, are coming from OSMC but only as a mirror service to take the load off of Kodiās servers (binary add-onās excepted as they are platform specific).
MariaDB is currently the recommended MySQL backend according to Kodiās wiki but as already pointed out earlier in this thread Kodi has always considered MySQL to be experimental and it has never gotten much love. That being said I have been using Kodi this way for over a decade and issues have been few and far between. The OSMC devs didnāt write, and donāt maintain the Universal Movie Scraper so no, OSMC doesnāt provide support for it. There does exist the people who did, and do, and you are free to discuss your issues with them on Kodiās forum. Generally speaking non platform specific Kodi issues should be taken up over at Kodi. The community on this forum will frequently try to assist people with Kodi specific issues on this forum anyway when it is feasible. This often is just a matter of providing workarounds or tweaks to configurations so there is no need for involvement of any programmers. When it comes to something like a particular scraper having an issue and the person wanting that particular scraper to stop having that issue then that is something that the person/s who are maintaining said scraper need to be contacted about.
