Apt-get upgrade fails (from RC1 to RC2)

mfarmwald · 15 April 2015 22:56

I’m tried this twice (on two different raspberry pi 2’s running RC1):

sudo apt-get update
sudo apt-get upgrade

It works for a while & hangs at:

Preparing to unpack …/armv7-connman-osmc_1.2.5_armhf.deb …

After that, I have to manually use the OSMC installer on the micro sd card.

ActionA · 16 April 2015 00:10

try
sudo apt-get dist-upgrade

mfarmwald · 16 April 2015 00:33

sudo apt-get dist-upgrade

Fails in exactly the same way and place.

DBMandrake · 16 April 2015 10:33

Log files please. Without those we can only guess.

If Kodi is still working use the log uploader to upload all your logs, if not upload the two APT logs with:

paste-log /var/log/apt/term.log
paste-log /var/log/apt/history.log

mfarmwald · 16 April 2015 15:55

This happened on 3 RPIs (both 1 & 2). After it hung, I was unable to ssh in again - actually I could ssh in, but the normal credentials wouldn’t allow me in (I tried osmc/osmc and pi/raspberry.)

So, I gave up and reinstalled OSMC RC2 using the OSMC Installer.

mfarmwald · 19 April 2015 23:03

Tried to “apt-get upgrade” on osmc rc2. It fails in the same place.

Preparing to unpack …/openssh-client_1%3a6.7p1-5_armhf.deb …
Unpacking openssh-client (1:6.7p1-5) over (1:6.7p1-3) …
Preparing to unpack …/armv6l-connman-osmc_1.2.6_armhf.deb …

I think it’s a bug in connman, see the following link:

I’m surprised that everyone who uses ssh isn’t seeing the same problem…

fzinken · 20 April 2015 00:14

When doing upgrades via SSH it always makes sense to use screen to not cut the branch you are sitting on

mfarmwald · 20 April 2015 00:18

I’ve got nearly 20 pi’s spread out and not easily accessible. If I can’t use SSH/apt-get to update, I’ll have to switch to another distribution.

The whole point of ssh is to be able to do it remotely.

It looks to me like it’s been fixed in the main Debian source, but hasn’t propagated to OSMC yet.

fzinken · 20 April 2015 00:20

Well my point was you can most likely use SSH just use “screen” then even if connman breaks the network the upgrade continues in the background.

mfarmwald · 20 April 2015 00:59

Sorry, I thought you mean the tv screen that the pi was attached to.

I searched for linux screen and now understand your comment. It would help, but it seems better if the connman stuff could get updated to the latest debian fix, so that that a straight ssh update would work correctly.

I did notice that the pi that went unresponsive eventually came back (with a differently ip address) and seems to work ok, so that the upgrade probably finished properly.

I’d offer to help, but I have no idea how the tool chain/etc for OSMC works.

Thanks.

sam_nazarko · 20 April 2015 06:43

Give us a chance to fix this first, please.

Can you show me where – we’re actually providing our own ConnMan package and we are based on 1.2.8 at the moment.

S

DBMandrake · 20 April 2015 07:02

We don’t use the Debian conman package, we have our own package with our own systemd service and our own package scripts so we will not be affected by the above referenced bug.

I will go over the package scripts again to see if I can spot any bugs however upgrading of the conman package and checking networking comes back up afterwards has been fairly thoroughly tested, with this being the first reported instance of this issue…

I have upgraded the connman package many times via ssh and I never use screen, nor should you need to.

We really need logs to be able to catch this - in particular the apt logs. These logs are persistent so can be uploaded after a reboot.

PS that bug report says the fix was included in connman 1.26 - we already have 1.28 which is the latest version which will include that fix, so this is not your issue.

DBMandrake · 20 April 2015 12:37

That’s your issue right there.

To upgrade connman we must stop the service (therefore disabling networking) before dpkg can extract the new version of connman. Then after files are extracted and replaced it will be restarted again. This is done by these two debian package scripts:

github.com

osmc/osmc/blob/master/package/connman-osmc/files/DEBIAN/preinst

#!/bin/bash

install_divert ()
{
	if ! dpkg-divert --listpackage "$2" | grep -q "$DPKG_MAINTSCRIPT_PACKAGE"; then
		dpkg-divert --add --package "$DPKG_MAINTSCRIPT_PACKAGE" --rename --divert "$1" "$2"
		EXIT=$?; if [ $EXIT -ne 0 ]; then exit $EXIT; fi
	fi
}

if [ -z "$DPKG_MAINTSCRIPT_PACKAGE" ]; then exit 1; fi
if [ "$1" = "upgrade" ]; then touch "/var/run/${DPKG_MAINTSCRIPT_PACKAGE}_upgrade"; else rm -f "/var/run/${DPKG_MAINTSCRIPT_PACKAGE}_upgrade"; fi
if [ "$1" = "install" -o "$1" = "upgrade" ]; then

	# install all diverts needed by this version of the package:
	# install_divert PATH_DIV PATH_ORIG

	sleep 1
	kill -9 $(pidof connmand) >/dev/null 2>&1
	systemctl stop connman.service >/dev/null 2>&1

This file has been truncated. show original

github.com

osmc/osmc/blob/master/package/connman-osmc/files/DEBIAN/postinst

#!/bin/bash

remove_divert ()
{
	if dpkg-divert --listpackage "$2" | grep -q "$DPKG_MAINTSCRIPT_PACKAGE"; then
		dpkg-divert --remove --package "$DPKG_MAINTSCRIPT_PACKAGE" --rename --divert "$1" "$2"
		EXIT=$?; if [ $EXIT -ne 0 ]; then exit $EXIT; fi
	fi
}

if [ -z "$DPKG_MAINTSCRIPT_PACKAGE" ]; then exit 1; fi
if [ "$1" = "configure" ]; then

	# remove only diverts that prior versions of this package added which are no longer needed:
	# remove_divert PATH_DIV PATH_ORIG

	if [ -e "/var/run/${DPKG_MAINTSCRIPT_PACKAGE}_upgrade" ] && [ -n "$2" ] && dpkg --compare-versions "$2" lt "1.2.9-3"; then

		# once only disable of ipv6 on all previously configured interfaces
		for file in /var/lib/connman/*/settings; do

This file has been truncated. show original

If you’re just updating the connman package alone typically this means the network would be down for about 10 seconds, however if a lot of packages are being upgraded in addition to connman it could be much longer, maybe a couple of minutes. As long as you don’t press a key on your ssh session it should resume when the connection comes back up.

The real issue though is that your Pi got a different IP address from your DHCP server - we have no choice but to shut down connman and restart it to upgrade it, however we have no control over the IP address assigned by the DHCP server so it entirely possible for this IP address to be different thus effectively break the ssh connection.

Connman already asks for the previous IP address specifically (dhcp client hint) when acquiring an address and MOST DHCP servers will honour this request unless it would cause an IP address conflict with another device. (Unlikely in such a short period of time)

The fact that you got a different IP address when disconnecting for probably less than a minute means either your DHCP server is not very well behaved, or that you have multiple DHCP servers on the same network allocating different ranges. (Thus your IP address might bounce between a couple of different addresses)

Not a whole lot that we can do about this I’m afraid. The good news is that APT is smart enough to keep installing if the controlling terminal (eg the ssh session) dies - this means that if you were to log in to the local console and run top you would see that APT is still actually running and will finish the entire dist-upgrade, provided you don’t go pulling the power plug thinking it’s hung…

So if you seem to lose the network connection during an upgrade give it 5-10 minutes to complete then try to connect again, keeping in mind that it might be on a different IP address depending on your DHCP setup.

mfarmwald · 20 April 2015 18:03

Thanks. This was very helpful and a very quick response.

I’ll make all of my pi’s static DHCP (I was planning to do that anyway.)

Again, thanks for the quick response.