Vero 4k random hangs

Hi Sam,

I’ve been testing my vero4k over the last week and can confirm the issue still remains
What I’ve tried:

  1. External USB powered HDD + running Deluge
  2. External USB powered HDD + no Deluge
  3. No External HDD + Deluge
  4. No External HDD + no Deluge
  5. Tried all above with external HDD with it’s own power supply
    So I’m still experiencing random hangs unfortunately and the nature of these hangs appear to be random.

Also I noticed that sometimes during playback screen goes black and sound disappears for couple seconds. Not sure if it’s related though =)

Thanks.

Hi

Thanks for the detailed reply.

That doesn’t sound right at all. As you’ll likely need to return your hardware to us for replacement under warranty, I’d recommend a complete installation from osmc.tv/download. If you continue to experience problems after this, please send an email to support@osmc.tv so we can make the necessary arrangements

Sam

Just checked right after screen went black and I can this in kodi.log

20:50:44.015 T:3367027456 WARNING: CRenderManager::WaitForBuffer - timeout waiting for buffer

We would need to see full debug logs when this happens to have an idea of what’s happening. Snippets don’t help

If you’re able to access your device during these blackouts, it’s probably not a hardware problem as the device isn’t locking up

Ok I’ll enable full debug and will send you the logs next time I face this problem

Had another blackout during playback today
https://paste.osmc.tv/upuvufotes
Balckout happened around 16:20

Also I believe the hangs are caused by overheating. I’m running stress to confirm this now.

Stress is a synthetic benchmark.

Are you running a lot of background services?

The blackout occurs due to a change in frame rate by the looks of it. Live TV program?

Sam

I’m running Deluged, Samba and SSH.

I was playing back recorded TV program. Mediainfo: http://paste.osmc.io/azuyecezip.tex

I have “Adjust display refresh rate” set to “On start / stop”. Same issue happens with “Always”

Does it always go black in the same spot?

I think the black screen issue is when the decoder struggles with changes in frame rate. You can test this by disabling amcodec temporarily to rule out a HW issue.

Re. hangs: did you ever try without the HDD attached?

I’m starting to think the Vero 4K is probably OK.

Sam

I was able to reproduce this with both internal storage and UBS attached drives (4 different models including Samsung T3 SSD)

  1. Start playback (pretty much any file)
  2. connect via ssh and run ‘stress --cpu 8 --io 4 --vm 2 --vm-bytes 128M’ command
  3. takes 2-10 minutes to hang Vero 4K

Note: I was able to run stress alone without playing movies for aroung 30 minutes without any hangs

I don’t like using programs creating artificial stress situations since often you see symptoms which are not the one you want to inspect and you start hunting phantom problems.

Nevertheless I tried your test setup WITHOUT Deluged installed, running a a Full HD mkv with DTS-HD MA (passthrough) and the example parms for stress like you did.

It runs now since 38 mins without any problem. I see in “top” 14 stress processes running and taking an overall CPU utilization of constant 100% (60%user, 40% sys), around 700 MB used memory, 200 MB free.
With the actual stretch development repository you get a kernel which allows you to run iotop. While my test running, I see nearly constant throughput about 1.5 MB/s from the Filecache/HDD.
This is just to compare your test scenario and circumstances with mine in case you want to repeat it.

So, the good question is: Was this Deluged stuff running while your test? And if yes, what happens if you deactivate it completely or even deinstall it and repeat the test?

You should see nearly the same CPU utilization, same memory demand and behavior like here.
Btw. the video playback and stress processes were still running without any issue while writing this last line … :wink:

Hi JimKnopf,

I agree with your point about creating artificial stress situations however this was
done to be able to consistently reproduce the issue as I was suspecting that this was caused by high CPU usage (which Deluge is generating pretty well =) )
So with stress I don’t have to wait for hours before my Vero hangs.

Just to confirm, I eliminated most of other factors that could contribute to the problem:

  1. disconnected all usb devices
  2. connected Vero 4K power directly to wall socket.
  3. stopped deluge
  4. started 1080p mkv from internal storage
  5. started stress as per my post above
  6. room temperature ~20C
  7. Vero4k is not covered and there’s nothing nearby that would generate heat.

As I mentioned earlier it takes minutes for my Vero to hang.

I also experienced hangs couple times when copying medium sized files(5-7GB) to internal storage from USB attached storage as well as over the network (stress/deluge weren’t running). I didn’t check CPU usage/temperature in those cases I’m afraid.

So just to reiterate:

  • This happens with both USB attached (usb powered and with power supply) and internal storage
  • With and without deluge running(however deluge is a good way to hang vero4K quicker)
  • Happens when playing content over network
  • Happens when copying files to intenal storage without any stress tests or deluge or movies running
  • reinstalling osmc from scratch doesn’t eliminate this problem
  • Happens when not playing any content
  • common factor is high CPU load

I still hope this is a software/firmware problem as this otherwise is a brilliant little device.

Hope, you mean + 20°C or is it in a permafrost area? :wink:

I can offer you to let run one of the video materials on my machine . With that we would know whether it’s related to the video or isolated to your box. This should be negotiated via PM in this forum. On the other hand you state it also occurs “when not palying any content”.

Another idea is to deactivate all scraping activities on your box, see your post and logs at Vero 4k random hangs - #11 by paestra. At timestamp 15:27:46.566 T:4056937216 something weird/strange happened but it is not clear whether this IMDB access has anything to do with the issues you’re suffering.

For any business solution @sam_nazarko should make a comment.

1 Like

It’s +20C of course =)

Yes, this happened with a number of different files as well as without any files being played

There was no hang during that time so I believe this is not related

From the mentioned log one can see to box was still alive long past 16:20 … that leads me to the idea to ask about your TV. I assume the Vero is directly attached to a TV. Have you thought about switching off the TV while hang wait 1-2 minutes and switch it on again? What happens then?

I can probably introduce a crash with cpuburn too; but god knows why you’d do that.

I’ll add some temperature governing; but I don’t want to cripple performance for people that don’t do this regularly.

With that said, some increased governing over frequencies is probably needed. There was a guy running sabnzbd+ and other services and he found increased temperatures which caused occasional freezes during playback.

If you had a fresh install with just OSMC I’m confident you wouldn’t experience these problems.

Cheers

Sam

Don’t get me wrong, I’m not trying to find a problem where there isn’t one. And I’m also not trying to crash the system deliberately :slightly_smiling_face:
I do believe however that downloading a single torrent file while watching a movie is not too much to ask from this device ← this was the initial problem I faced before I started digging deeper.
The same setup was working fine on my RPI3. There were no crashes whatsoever.

Please let me know however if you think that Vero 4K is not fit for my use case.

I’m willing to test further and provide whatever information necessary for you guys to figure out what’s going on.

I don’t think a single torrent would be a problem at all.

Obviously however you are adding something that will contend for bandwidth, IO and CPU time. If you were seeding a large torrent, you’d probably notice problems with streaming.

The Ethernet on the Vero 4K isn’t connected via USB however; so this isn’t as problematic as torrenting via Ethernet and saving to a USB disk would be on a Pi.

Transmission tends to be more lightweight (you can do more at once) than Deluge; and the OSMC version has some performance improvements.

If you believe it’s overheating, then you could try changing the max scaling frequency temporarily to see if this is the case. There are still a couple of corner cases where the device can overheat because the governor doesn’t kick in quickly enough.

However, if the device is indeed overheating, then it would become unresponsive completely (no SSH or ping response).

The hangs sound different, and this could depend on customisations made to the system. top will let you see processes that are starved (high IO wait) for example.

Kodi’s render thread will exhibit skips if it is switched away. You may find that renicing the torrent client process resolves this. If it works, you can set a nice value in the systemd unit itself.

Even better would be to run: sudo systemctl restart mediacenter and see if Kodi comes back. That will determine if it’s a Kodi crash or not.

Also just to confirm: the device isn’t in a cabinet or anything sitting on top of it.

At the moment there are a lot of variables, and we need to narrow things down a bit so we can work out what the issue is (it will no doubt be software / configuration related)

Sam

Probably I didn’t make myself clear earlier in the thread…
The device hangs dead showing black/grey/blue/you-name-it screen
No response over network, neither does it respond to remote control.
Nothing in log files after boot
The only thing that brings it back is unplug/plug power.

You need to put some monitoring in place:

1 Add a simple top command to run every minute via cron. (If cron isn’t there already, install it from the app store or through apt-get.) Run crontab -e to edit the cron table then add the following line:

* * * * * /usr/bin/top -b -n1 | /usr/bin/head -5 >> /home/osmc/monitor.out 2>&1

2 Write your system journal to “disk”. (It’s normally just kept in memory.) To do this run:

sudo mkdir /var/log/journal
sudo systemctl restart systemd-journald

(The system journals can become difficult to read because the Vero4K doesn’t have a battery-backed clock, but in the event of a system crash, we should be able to figure out the latest date/time.)