Am hoping to help pinpoint the problem with this skin. It applies just fine but the first reboot after leads to a sadface bootloop. The only way to get the system back to normal is to edit the guisettings.xml through SSH to put the system back to confluence. I can’t seem to find anything in the logs except for a segmentation fault. The creator of the skin is willing to work with distribution owners:
As publishers and curators to the referencing skin, “Arctic Zephyr: From Ashes” we will provide additional information which was not included in the original postings by @h4xdaplanet. Through various reports and a multitude of testings it has been determined via system log stack traces a “Segmentation Fault” occurs under very specific conditions outlined below:
The platform is a Raspberry Pi
The skin is “Arctic Zephyr: From Ashes”
The Operating System is either OSMC, OpenELEC, or LibreELEC
It appears that in the case of OSMC, the operating system is actually a stripped down version of the Debian platform running Kodi Jarvis. The full version of the Debian platform running Kodi Jarvis runs the skin and Kodi without issue.
Stack traces have linked the issue to the following file: ld-linux.so
This is the dynamic loader or dynamic linker for Linux which of course can be modified and reconfigured by distributions and depending on the distribution may in fact be further configurable by the end user.
It is possible that OSMC, LibreELEC, and OpenELEC are simply missing key packages which are included in the main distributions they are built upon which is causing the kernel to panic on boot.
We will attempt to contact the developers of OSMC more directly to affect a resolution to this issue.
If other users are using “Arctic Zephyr: From Ashes,” we are monitoring this thread on OSMC and ask that system logs be provided to further the investigation. Please note that it will ultimately be up to the OSMC developers on whether or not this gets fixed but it is clear that this issue is effecting a very specific subset of users running on the OSMC platform.
ld-linux.so is provided by libc6, which is taken from upstream Debian Jessie. We have not customised this in any way. All OSMC packages that are built downstream are built against a matching libc6.
Let us know how we can replicate the problem. If you can reproduce this problem yourself, please try install a debug build:
sudo apt-get install rbp1-mediacenter-debug-osmc (or rbp2 where appropriate)
And use GDB to generate a backtrace.
What are the dependencies of your skin? We have not heard of problems with other skins, so it’s unlikely to be a missing library. Unless your skin directly depends on a library which is dlopened(), Kodi would not launch if a library was missing, and ldd would show the missing library.
I was not aware that this skin was causing a kernel panic. I’m not sure how that can be – but please take a photo of the panic so we can see what is going on.
I am an OSMC user with raspberry pi 3. I am using arctic zephyr but if I change to arctic zephyr from ashes mod I get the sad face and the reboot looping. I am not an advanced user but I can provide you every log you need if you explain me where is it or how I can get it.
@sam_nazarko We do not normally use the Raspberry Pi in our testing of the skin releases and are dependent on community feedback for this information including required logs. We curate the skin as a service to the community and it was handed off in late 2015 (when it was known as “Arctic Zephyr: Exploded”) to us and subsequently renamed “Arctic Zephyr: From Ashes” at its initial release. Please note that we are referring to the regular “dormer” line (3.x) version and not “danvers” (4.x) as the latter is intended for Krypton and no where near ready.
The following is an outline of the “addons.xml” file for “Arctic Zephyr: From Ashes:”
I would highly recommend simply downloading the CDN version (most people prefer this route or the Emby Repo route). Do note, as outlined in Arctic Zephyr: Generations (Jarvis / Leia) [3.4.70 / Coming Soon] you will in fact need to install "Script.Duration" before installing the skin.
All reports have indicated that all which is required is to set “Arctic Zephyr: From Ashes” as the Kodi skin and reboot to cause the kernel panic which indicates a “Segmentation Fault” having occurred.
@DBMandrake we have not asked in detail what processor platform is being used for those running Debian and From Ashes. It is however known that From Ashes will run on a multitude of ARM devices running Android although this may too be considered an “Apples to Oranges” comparison.
Based on analytical data, From Ashes is ran on tens of thousands of devices and it has been reported that the Debian based “Raspbian” is capable of running From Ashes without causing a kernel panic.
The replication of the issue has been described and is apparently fairly simple to emulate. What is concerning is how specific of a subset of users the issue is actually having an effect on.
I’d strongly recommend you start doing so. The user base is massive, and after Windows and Android, I believe it is the most popular Kodi platform. You describe this problem to be present in OpenELEC, LibreELEC and OSMC, which is almost the entirety of the Kodi userbase on this device.
I haven’t seen any reports of a kernel panic. If there is a kernel panic, you should see a full stack trace on screen. We would need a photo of this, and then debugging the problem would be a lot easier. Are you sure you mean a kernel panic? I think you may just be referring to the ‘sad face’ screen, which means that Kodi could not start properly. It does not mean the kernel has panicked however.
If the system kernel panics, then it will not be possible to retrieve logs in the same session (they would need to restart), as their system would have completely crashed. They would also not be able to log in to get logs, as the skin would try and load each time their device booted.
A segfault does not necessarily indicate a kernel panic.
Was it running the same version of Kodi (16.x)?
Does the version includuing @popcornmix’s backports tree?
When was this version compiled?
Your skin certainly depends on a large number of add-ons. I suspect you have inherited a lot of these dependencies, but it means that one of the add-ons that are depended on may be the cause of the issue, instead of the skin itself.
This suggests either an incompatible add-on version is present, or another add-on has compatibility issues with your skin or an add-on that your skin depends on.
I’d strongly recommend you start doing so. The user base is massive, and after Windows and Android, I believe it is the most popular Kodi platform. You describe this problem to be present in OpenELEC, LibreELEC and OSMC, which is almost the entirety of the Kodi userbase on this device. [/quote]
Indeed, with the uptick of people using Raspberry Pis I signed off on several variations of the Raspberry Pi’s to be bulk ordered (in particular, the Zeros are perfect little controller boards for something we’re working on in the datacenter but naturally the 3s as well). I also personally ordered a Raspberry Pi 3 which will of course arrive before any bulk orders with a preload of OSMC.
I haven’t seen any reports of a kernel panic. If there is a kernel panic, you should see a full stack trace on screen. We would need a photo of this, and then debugging the problem would be a lot easier. Are you sure you mean a kernel panic? I think you may just be referring to the ‘sad face’ screen, which means that Kodi could not start properly. It does not mean the kernel has panicked however.
If the system kernel panics, then it will not be possible to retrieve logs in the same session (they would need to restart), as their system would have completely crashed. They would also not be able to log in to get logs, as the skin would try and load each time their device booted.
A segfault does not necessarily indicate a kernel panic. [/quote]
Reports initially started out as the “sad face screen” which, as you indicated, just simply means Kodi could not start but apparently the only way to break the boot loop cycle was and is to do something as drastic as a reload of OSMC while others have reported some success in gaining access to the system via SSH and resetting Kodi's theme to Confluence. Others have of course simply connected the card to a computer and mounted it as rw to pull the logs and and revert Kodi back to Confluence.
Someone provided a system log file of what happens when From Ashes is ran as the default skin with all conditions for the kernel to panic met:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0098d3a0 in CWinEventsLinux::MessagePump() ()
[Current thread is 1 (Thread 0x7502b000 (LWP 623))
....
Thread 1 (Thread 0x7502b000 (LWP 623)):
#0 0x0098d3a0 in CWinEventsLinux::MessagePump() ()
#1 0x76fa2fb4 in ?? () from /lib/ld-linux-armhf.so.3
Backtrace stopped: previous frame identical to this frame (corrupt stack?)]
Naturally, I will ask for further logs and a Raspberry Pi 3 with OSMC is on the way so I can see for myself as well. My area of expertise when it comes to Linux is in enterprise linux operating systems such as RHEL / CentOS / CloudLinux and other various derivatives and not non-enterprise variants like Ubuntu and Debian although a case could be argued for both distributions meeting the qualifications of the enterprise.
Does the version includuing @popcornmix’s backports tree?
When was this version compiled?
Your skin certainly depends on a large number of add-ons. I suspect you have inherited a lot of these dependencies, but it means that one of the add-ons that are depended on may be the cause of the issue, instead of the skin itself. [/quote]
I am uncertain as to whether or not the users who tried Raspbian ever used @popcornmmix’s backbports tree or not but there has been confirmation that Raspbian with Kodi 16.1 and Kodi 17 Alpha work fine. Do note that the Kodi 17 Alpha version of this skin is missing DialogConfirm.xml and users have simply “hacked that in there” until the June 18th update. It was the only missing piece for the “From Ashes (Danvers Preview)” to run on Kodi 17 Krypton.
I will ask for further details about the Raspbian setups but from my understanding it’s the latest stable version available.
As to the large number of addons required (I completely agree with you; it can take low power systems up to 2 minutes to install From Ashes depending on load and internet connection), 8 are required by de facto as the base Arctic Zephyr skin requires them. The other 11 are required for the enhanced features of Arctic Zephyr: From Ashes including its integration with Emby. With the Summer 2016 Update, this number will become 12 (“Script.Duration” was mistakenly dropped from the requirements). Of that, only one was added after “Arctic Zephyr: Exploded” became “Arctic Zephyr: From Ashes:”
Every other change which has been made since From Ashes’ release last year has been in the confines of Kodi itself such as Autocompletion Keyboard or using enhancements to the Skin Helper Service which is used by virtually every skin to provide additional functionality.
To me, this would also suggest the possibility that there is an incompatibility between one or more components of the skin and OSMC once you take into account all factoring criteria such as but not limited to what other platforms if any does this issue occur on. The only answers are LibreELEC and OpenELEC. However, as stated, that’s not the only criteria.
We identified OSMC as being the most popular after Windows, OS X, and Linux (all distros), so we, as stated, are willing to work with you all to find an appropriate resolution. I personally should have a Pi in my hands in a few days as I bought one for myself separate from the bulk orders.
In the interim, we’re dependent on user reports and logs nor is there any way for us to realtime change aspects of the environment like we can in Windows, OS X, and Linux. We’re fairly certain what users are reporting, based on the description and logs, are in fact a kernel panic but until we see it for ourselves there is no way to be 100% certain.
As always, we will continue watching this thread and will post updates in responses or after we have at least one rPi 3 for testing and can report in further detail.
This is not a kernel panic. If you can log in via SSH, the kernel has also not panicked.
Kodi has segfaulted. This isn’t good. We recently noticed an issue where Kodi segfaulted in an add-on due to updating an element in a list that was no longer there. Installing a debug build, and then running the backtrace again would make it more obvious what the issue is. I have put some instructions for this above.
I recommend you grab the latest version from Download - OSMC so you can make sure you’re on a level playing field.
Things are similar because the focus is Kodi. I suspect we will find an issue that will apply to Kodi rather than the underlying OS.
If I had to hedge a bet randomly, I would be wary of script.skinshortcuts. I have noticed some problems with this add-on causing crashes before. We currently ship a version as our OSMC skin depends on it. It may be possible that it needs updating, but it would not explain the issue on LE/OE (I don’t believe they bundle it).
When you get your device let me know and I will give you some guidance. I will also try and find some time to work on it.
This is not a kernel panic. If you can log in via SSH, the kernel has also not panicked.
Kodi has segfaulted. This isn’t good. We recently noticed an issue where Kodi segfaulted in an add-on due to updating an element in a list that was no longer there. Installing a debug build, and then running the backtrace again would make it more obvious what the issue is. I have put some instructions for this above. [/quote]
@sam_nazarko it’s great that my Raspberry Pi 3 came. It’s a wonderful device; I naturally ordered my personal one with the “media package” which just gives you a few extra things like a wireless keyboard and mouse, SD Card with OSMC which I subsequently flashed to the latest version, cables, and all those other goodies. Even though it included a case I decided to also buy myself the Raspberry Foundation case (the pink and white one). As you had suggested, it’s a wonderful little device and now that I have it in front of me I’m no longer flying blind.
You are quite correct, OSMC is not panicking but Kodi is most definitely crashing. Usually, when we encounter Seg Faults, it’s a kernel panic but I come from the view point of highly virtualized machines running VMWare Enterprise. Given that I was able to press the left arrow key to get to the shell login screen from the sad face screen without a reboot or "rescue mode" we can obviously rule out a kernel panic.
I am pleased to see that a tool already exists to quickly format a new version of OSMC. This is rather convenient. I was expecting to have to build it on my Mac before moving it to SD.
[quote=“sam_nazarko, post:11, topic:17042, full:true”]
Things are similar because the focus is Kodi. I suspect we will find an issue that will apply to Kodi rather than the underlying OS. [/quote]
With this, I do want to provide the kodi.log which starts off with the error on line 19:
13:06:01 5327.163574 T:1956389808 ERROR: DBus: Error org.freedesktop.DBus.Error.ServiceUnknown - The name org.freedesktop.UPower was not provided by any .service files
I know that AZ:FA uses a newer version of “script.skinshortcuts” than what’s available in Kodi’s repository and what’s shipped on OSMC. The logic is that it will use Emby's Repo version of "script.skinshortcuts" before falling back to the Kodi Repo version as Emby’s version is always ahead. However, nothing really indicated from the Kodi log file that “script.skinshortcuts” was or is having a problem.
Only one other error occurs in kodi.log before the “sad face” and it is something that is universal to all platforms so I would also highly doubt it being the culprit:
Also, a warning which doesn't occur with other platforms as far as reports we have received does happen under OSMC:
13:06:03 5329.317383 T:1956389808 WARNING: JSONRPC: Could not parse type "Setting.Details.SettingList"
Now, I haven’t had time to review the OSMC logs but after discovering that OSMC doesn’t keep the logs in /var/logs or other traditional locations I found a help page that described using the command grab-logs -A -C to save a copy of all the logs to /boot/uploadlog.txt. As I did say, I haven’t had the change to go through them all yet but I have included the entirety as well.
Perhaps now that we have more information we can shed some light onto this and reach a fix. I know there are a lot of people waiting for the “green light” on using the rPi 3 while others moved to NUCs.
I can tell you of one item that can be ruled out. The From Ashes Audio Suite. It was installed right after getting through OSMC's setup process. I really can't stand Confluence's sounds so the system was up for a good hour with the From Asues Audio Suite running.
I did not realize that I had not enabled Debug Level Logging in that log until you had said something. While I haven’t had the chance to read the log myself, since the issue is so extraordinarily easy to replicate I went ahead and set debug and restarted the rPi3 so it would get stuck in a “sad face” crash loop.
Please note that in debug level logging, Kodi will usually present two messages which can safely be ignored as the tags are correct and were even rechecked by multiple third-parties for verification. They did not however appear until the First or Second Generation Color Enhancers:
If this is of useful information to you @sam_nazarko, the order for which we heard about it was OSMC Crashing → OSMC Crashing with some Skins on Jarvis → OSMC Crashing with Arctic Zephyr: From Ashes (effecting only rPi3 in every report we received).
The crash has extended to now being reproducible on every version of Arctic Zephyr: From Ashes including those which previously worked.
Many have pointed out, unlike it’s counterparts, AZ:FA is not an easy skin to run; particularly for low powered devices. Installs can take up to 2 minutes if it’s a clean install and it can drive up CPU and RAM usage but people have this skin running on many low powered devices without issues including the Amazon Fire Stick (or Amazon Stick - it’s something I haven’t hard of; I just know it’s like a Chromecast but made by Amazon and has local capabilities where you can directly load files such as Kodi).
We have noticed a potential crash with the hardware mouse pointer improvements for Raspberry Pi. These improvements are present in OSMC, OpenELEC and LibreELEC.
We will keep monitoring this thread for future updates in regards to this issue while @popcornmix looks into it. We haven’t contacted the other distributions about it as OSMC seemed to be the “most put together” of the group but will also endeavour to do so.
Arctic Zephyr: From Ashes’ “Dormer” line (3.x - Jarvis) is reaching “Long Term Support” status with the next update as we introduce the preview of Arctic Zephyr: From Ashes Anniversary Edition “Danvers” line (4.x - Krypton) as “Preview Availability.”
Based on estimations and discussions, it seems the Krypton beta will hit around the Autumn but it would be nice if for the Jarvis users they had a working solution.
If there is something you think we can do, as I said, we’ll be watching this thread and you can always PM directly.
@popcornmix was able to identify and resolve an issue with the hardware mouse pointer improvements:
I have now added these changes and compiled a test build. I plan to include this change in an update on Sunday. If you would like to test these changes in advance, you can do so by running the following commands:
You can test these changes in advance by running the following commands:
After this is done, I recommend you change the line back in /etc/apt/sources.list, or you will continue to get OSMC updates as they are pushed, ahead of any monthly update, which have no guarantee of stability.
This is excellent news to hear that @popcornmix was able to find and resolve the issue within what appears to be the XBMC code. Fortunately, I am off Friday - Monday unless I need to actually do anything or need to sign a multitude of documents so I was able to test out the fix as proposed on GitHub and described here. In addition, I sent several copies over to be tested on a group of Raspberry Pi 3’s using a different SD Card from the one the one that loads our proprietary 64bit Operating System for what we intend to use them for (a long with the Zeros which don’t come into play here) and can confirm that the entire group (12) plus the personal one I bought reacted in the same way and Arctic Zephyr: From Ashes boots and runs on OSMC.
The group of 12 are still cycling through automated reboots (they’ll do that for another hour at 6 minute intervals) for a total of 10 reboots but I’ve already rebooted my personal one enough times to know it’s consistent and does correct the issue.
This was tested on the upcoming Summer 2016 Update for Arctic Zephyr: From Ashes. I’m not sure whether or not the OSMC controls to the Pi are a Widget or not but we might patch that into the skin for Jarvis even as it enters "Long Terms Support (LTS) status in favor of the Preview AvailabilityArctic Zephyr: From Ashes Anniversary Edition (codenamed “Danvers”) for Krypton.
I don’t know when you all intend to start moving into previews and alphas for Krypton but when you do, we will be watching as we can get behind supporting the OSMC Project.
From the Preview Builds of AZ:FA Summer 2016 Update and OSMC:
@sam_nazarko and @NortheBridge - Thank you so much for working together. You guys really help make this community so valuable. Looking forward to Sunday’s release.
