Vero 4k+: Linux Kernel 4.9 is end-of-life

New hardware? That sounds exiting. Can you share anything? Is that something that will replace the Vero 4K + or will both be available, SoC etc.? :slight_smile:

It will be a complete replacement – apart from those that have a business use case and still want Vero 4K / 4K +.

4 Likes

Found this thread, and while the name of the thread is about Vero 4k+ @sam_nazarko already references Vero V, which in the meantime I have ordered and started using, but to my dismay it is still using 4.9.269, not AMLogic’s 6.1 based kernel:


osmc@fiver:~$ uname -a
Linux fiver 4.9.269-44-osmc #1 SMP PREEMPT Thu Nov 30 02:59:52 UTC 2023 aarch64 GNU/Linux

The most unsettling part is that by now not only the 4.9 LTS kernel, but also the 4.14 is out of support since two days ago: Linux 4.14 LTS Reaches End-Of-Life After Six Years - Phoronix
…and the last 4.x LTS kernel will also out of support within the year: The Linux Kernel Archives - Releases

I am also not entertained by the arguments of “it still runs fine and the users do not care about the version” because all the missing security updates do matter. 4.9.269 was released close to 3 years ago Linux kernel 4.9.269 released — Linux Kernel Announce and while I assume Sam & co are trying to backport some fixes I would bet it is not rid of all the vulnerabilities that are fixed in the active LTS kernels (6.1 is one of them).

I know about the 5 year support commitment for Vero 4k and 4k+ already being expired, but is there an ETA to upgrade at least the brand new Vero V to a supported and secure LTS kernel version (for example the mentioned 6.1)?

If it would happen for the 4k+ as well I would be even happier as I have two of those too, and I prefer not to throw them away :sweat_smile: But I don’t want them to end up as Tor exit nodes on my private network…

Hi Peter,

The latest supported AMLogic kernel with full media functionality (important for us obviously) is 5.15.
We are already working on it and it will be available on Vero V.

This kernel is still maintained by us for matters of security, maintenance and improvements. It is still maintained by AMLogic.

Red Hat for example are still supporting, shipping and maintaining Linux 3.10 until June 2024 in an official capacity as part of their RHEL7 commitments.

Our initial plan is to move Vero V to a 5.15 kernel soon, and we’ll have test threads with that in due course. 2024 is also the year where we plan to move to a full 64-bit environment, so there’s a lot of changes ahead.

With regards to Vero 4K and 4K +, we have managed to get support upstreamed so they can run on a mainline kernel indefinitely if a customer wishes to do so. There won’t be feature parity but they can indeed have a modern software stack even after we stop supporting the device.

That’s concerning. I haven’t had any reports of OSMC Vero devices being compromised due to kernel exploits. Do you have any references to this or is it just speculation?

Are you publicly exposing your device to the internet?

Thanks

Sam

Hi @sam_nazarko,

Thanks for the fantastic fast response, as always, and thanks for confirming a kernel to get the Vero V back in “official support” is in the pipeline already!

I was assuming you are trying to keep up with the security patches, however I am sure both the community responsible for the LTS kernels (probably working for corporations which have a stake in those :wink:) and RedHat has more resources not to miss anything.
But I have not heard any specific case of a Vero being exploited, and I am not exposing it to the internet - just a turn of phase we usually use with the tor exit node (e.g. I don’t want my smart fridge become a tor exit node).

With a supported LTS you will also have to carry much less backports! :crossed_fingers: Happy to join and support the beta testing once open.

I care less about the 64 bit environment, but should probably streamline things similarly as I see many distros are considering dropping 32-bit libs / compatibility.

I have one more question like Colombo turning back from the door: can we hope for Python 3.11, or only when it comes with whatever debian major release makes it default?
I have read wonders about the speedups and any help is appreciated looking at how slow library scanning and scraping is when setting up a new devices with a library of thousands of movies and tens of thousands of episodes…

It’s a lot of work. At the time we started working on Vero V our options were the existing 4.9 kernel or 5.4. AMLogic were meant to release their latest Android (shared kernel) SDK based on 5.15 at the end of last year.

We have access to those sources but the stack is still immature and I expect them to finalise a release around March. Frustratingly, a lot has changed around the kernel build system for AMLogic to meet Android’s GKI requirements and there’s a fair bit of work to do around that. As you can imagine, there is a lot of vendor specific code in this tree so it’s not a case of just fetching a new kernel and building it. We will need to port over all of our improvements; fixes (and no doubt new bugs on the way); 3D support etc.

So while a 5.15 release is on the horizon, I expect it will be in a testing phase for some time with public testing probably occurring around the middle of this year. This should lay the foundation for the future of Vero V. The decision to use 4.9 for the Vero V wasn’t taken likely. It is not even officially supported by AMLogic for this device. But we chose to do this so that users would have an identical stack to Vero 4K / 4K + and feature parity. In short, if it works on Vero 4K / 4K +, it will work on Vero V. The idea was to launch a device without regressions.

Red Hat has a few billion dollars in revenue per year and customers willing to pay large sums of money for long term support in environments that can’t change or be upgraded easily. You’ve purchased a Vero 4K + for a one off fee of £99.

With that said, we keep an eye on anything important and backport it. If your device is not exposed to the Internet then I would not be particularly concerned about security risks. The biggest risks to your device would be in userland (i.e. sshd vulnerability), but we are on a stable and supported distribution.

Plan is to do this to mainly streamline Docker support and stop the conflicts that sometimes occur with an arm64 kernel and 32-bit userland.

We are also going to migrate to Debian Bookworm with the 64-bit change. This will bring Python 3.11.

I’m not sure if you’ve tried Vero V yet but if you have you will appreciate the significantly faster scraping speeds that it has over Vero 4K +.

Cheers

Sam

1 Like

Hi Sam,

As I said I have 2x 4k+ and just received my Vero V this week, still scanning the files into the library.

Perceived speed:
UI is snappier even when used with a 4k UI, but frankly scraping does not seem any bit faster than the 4k+, it is sluggish and freezes into the process in less than 24 hours time.
This is even though I am trying to use the modern, default, stable, supported scrapers: “The Movie Database Python” and “TMDb TV Shows”. (These were the defaults with Vero V as shipped, and earlier I was advised here that the Universal Scrapers are not great.)

(I did not want to migrate the databases from the Vero 4k+es as it was discussed here earlier that the MariaDB is not officially supported and also crashed into scanning even with the DB columns altered to be similar to the SQLite DB, so I am assuming a clean device best goes with a fresh library re-scraped from scratch.)

CPU temperature is 85C, not sure if this can cause instability.

Cheers,
P

Unless you specifically want a 4K UI, I would keep it at 1080p. We recently added support for the 4K UI, but it can cause some problems with whitelisting / refresh rate adjustment. It’s still a bit experimental.

Others have reported quite significant speedups.
Kodi modernised scrapers recently and to make them more adaptable, devolved them out of the core (C++) and in to Python add-ons. That obviously has some significant performance implications, but as a long term strategy it does make sense.

You should be able to use Kodi’s export and import facilities to manage this.

That’s within range and more than acceptable

Sam

I prefer to use 4k UI because I have a large screen and 1080 scaled up to 65" shows visually “smeary” images. Of course even on 4k some scraped images have compression artifacts, but at least the good ones look great.

As for scraping I plan to keep stopping and restarting the scraping till it completes :man_shrugging:

Cheers,
P

Perhaps a separate thread with some logs would help us identify the freezing issue; it would be good to know if the whole device is locking up or if it’s just Kodi that becomes unresponsive.

There could also be a corrupt file that is causing Kodi to crash. Unfortunately Kodi isn’t very tolerant of such files and doesn’t handle them gracefully.

Neither, just the scanning stops at a title and is stuck there. If I stop scanning and restart the top right message alternates between the stuck title and the scraping thread that is making progress.

If I get fed up I will post logs into a new thread, thanks, just that it will create loads of logs till it gets stuck. Is there a configuration of debug logging which only focuses on the scraping and library?

Not off the top of my head and I’d suspect not, particularly as it’s now Python based and somewhat devolved from core.

You can happily run 4K’s, V’s, RPi’s, Kodi on Windows, *nix, Android, Fire OS, etc. all off the exact same MariaDB at the same time. I know this from first hand experience. I’m assuming you are referring to when I stated that MySQL is considered experimental by Kodi. It is, but that doesn’t mean that it doesn’t work or someone should avoid using it. I also stated that I had been running Kodi this way for over a decade now. If you already have a Maria setup for Vero 4K’s I don’t see where there should be any reason not to just use that same database. If your using a SQLite db on a different Kodi install (regardless of the OS it is sitting on) there is no reason why you couldn’t just transfer that as well instead of starting from scratch. Really the only userdata files that should potentially be an issue transferring from one device to the next is guisettings.xml and any binary add-ons assuming that the file paths used are universal (ie a network path).

I appreciate the work. And I know it’s a lot. But I do not agree with this statement.

Yes, exposing a Vero to the internet ist probably not a good idea for most use-cases. But in the modern age of IPv6, more and more devices are “exposed to the Internet”, possibly without users being aware of this.

In the Linux kernel, vulnerabilities enabling remote code execution from the outside or in the network stack are rare, but they do exist. But more dangerous to Vero instances are probably privilege escalation bugs usable after (not very unlikely) RCE bugs in Kodi - a huge project with many moving parts and plugins et al. And there are lots of privesc bugs fixed in pretty much every release.

I doubt AMLogic and/or OSMC backport every CVE fix in the Linux kernel. But from a pure infosec POV, that would be the correct thing to do.

Now of course, the question is: Is it more work to

  1. fork a kernel, patch it with vendor hardware code and backport CVE fixes, or
  2. base yourself on upstream kernel.org releases and port vendor hardware code to every release.

You have made your decision on this. But can you help us understand what that means?

Is there a source code repository with the “security improvements” you backport to it that we can view, subscribe - and possibly contribute to?

Thanks.

How is that going to happen? Any typical ISP router will block external to internal traffic unless allowed.

2 Likes

We do not enable IPv6 out of the box by default.

Relying on NAT as a form of firewall is bad; but routers and ISPs that support IPv6 do not typically make every address on the LAN publicly routable for this specific reason. And from experience (at least here in the UK) when you make them publicly routable there are usually inbound traffic management rules set by default.

Can you reference one since 4.9.269 that we need to look in to?

From your post history and enquiries about Debian Bullseye, and recently Bookworm, I know you are a fan of newer / shinier / greater.

Low hanging security fruit would be the default SSH password; the ability to escalate privileges as root from Kodi.

I am sure you use Kodi add-ons on your device. Have you checked their source code to make sure they are not malicious; do you check them when they auto update? I’m playing devil’s advocate because you are picking what I would consider to be one of the least viable attack vectors for a device and amplifying it significantly.

We backport what is relevant.
You could also note that as you move to newer kernels, new bugs and vulnerabilities are introduced.

It’s impossible to do 2 and have any form of modern hardware, even with x86_x64.

Every device with media acceleration has a downstream kernel with downstream patches. Any hardware that is mainlined is going to be old.

The good news for you is that your device is getting old. The 4.9 kernel will be the last release for Vero 4K + and unfortunately it won’t see the new 5.15 kernel. From a user and functionality perspective, that won’t particularly matter.

Vero 4K + has been supported with the mainline kernel since early 2022.

See

There isn’t feature parity with the OSMC kernel, but if you want the latest and greatest you can indeed run it on OSMC. As the device starts to reach its age we don’t want to abandon users, so we took the necessary steps for it to work on a mainline kernel and be maintainable for the forseeable future. That way it can be repurposed for other roles, perhaps as a server or NAS; even if it never achieves feature parity in terms of media playback capabilities.

All source code is public on GitHub.

Feel free to suggest that this might be true for a majority. But it is definitely not true for all users.

A quick websearch finds many users who have their IPv6 exposed by their routers: https://www.reddit.com/r/HomeNetworking/comments/huj960/is_my_random_home_router_exposing_my_ipv6/

There are many reasons for this. Some routers don’t have IPv6 firewalling enabled. Some expose IPv6 by default. Other users enable global reachability for one device and don’t realize this may affect all devices.

Also, many network engineers regard NAT as a bug and global reachability of IPv6 as feature, the way the internet was designed and is supposed to be.

All this shows: In the modern age of IPv6, it is just not feasible to assume that “your device is not exposed to the Internet” - let alone design your security decisions on that assumption.

Well – for now, a Vero definitely won’t be through IPv6…

Exposing things to the internet accidentally happens more often than not, which is why we prompt whether a user wants to enable SSH during initial setup for example.

I am still not aware of any actively exploitable kernel CVEs via network >4.9.269 that wouldn’t be covered.

The kernel doesn’t have any exposed interface to the network by default unless using NFS booting; GDB for debugging or an NFS server.

Regardless – as shown above, you can boot a mainline kernel today and run it on your device if you wish to do so.

Linux 4.9.269 was released on 2021-05-22. So it’s safe to assume that pretty much all CVEs issued for Linux after that date are not fixed in 4.9.269.
There have been at least 306 CVEs in 2022, and 288 CVEs in 2023: Information on source package linux
If you can say, I would be interested to see how many of these were backported. (But please don’t regard this as “pushy”, I am in no position to “demand” otherwise, I’m just sincerely curious :slight_smile:

I am convinced most Linux developers and maintainers would argue it’s impossible to do #1 and backport a significant amount of CVE fixes.

I acknowledge that you have to choose between two imperfect options requiring lots of work, and that you feel forced to chose hardware features over security patches. But it’s good to be clear and open about this.