Certificate verification failures

I’m getting CERTIFICATE_VERIFY_FAILED when e.g. using youtube-dl:

$ python2 /usr/local/bin/youtube-dl -s https://devstreaming-cdn.apple.com/videos/wwdc/2019/722l6blhn0efespfgx/722/722_hd_introducing_combine.mp4?dl=1
[generic] 722_hd_introducing_combine: Requesting header
WARNING: Could not send HEAD request to https://devstreaming-cdn.apple.com/videos/wwdc/2019/722l6blhn0efespfgx/722/722_hd_introducing_combine.mp4?dl=1: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
[generic] 722_hd_introducing_combine: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)> (caused by URLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)'),))

Is there a certificate store that needs to be updated?

The resource is accessible on other systems.

Not sure if relevant:

$ apt show ca-certificates
Package: ca-certificates
Version: 20200601~deb9u1
Installed-Size: 380
Maintainer: Michael Shuler <michael@pbandjelly.org>
Architecture: all
Depends: openssl (>= 1.0.0), debconf (>= 0.5) | debconf-2.0
Enhances: openssl
Breaks: ca-certificates-java (<< 20121112+nmu1)
Description-en: Common CA certificates
 Contains the certificate authorities shipped with Mozilla's browser to allow
 SSL-based applications to check for the authenticity of SSL connections.
 .
 Please note that Debian can neither confirm nor deny whether the
 certificate authorities whose certificates are included in this package
 have in any way been audited for trustworthiness or RFC 3647 compliance.
 Full responsibility to assess them belongs to the local system
 administrator.
Description-md5: e867d2a359bea1800b5bff209fc65bd1
Multi-Arch: foreign
Tag: protocol::ssl, role::app-data, security::authentication
Section: misc
Priority: optional
Filename: pool/main/c/ca-certificates/ca-certificates_20200601~deb9u1_all.deb
Size: 159692
MD5sum: c91a91f7b3b5141d6e7eafc6eb2412f5
SHA256: d6fc9275de84bdf598b3703a50935dc69d2e80105317a44f6cba465c073f88f9

That looks fine (ca-certificates being installed). What’s the output of:

wget --server-response  -O /dev/null https://google.com

You can use no-check-certificate for now with youtube-dl in case there is some temporary issue with the certificate.

$ wget --server-response  -O /dev/null https://google.com
--2020-09-18 16:23:08--  https://google.com/
Slår upp google.com (google.com)... 216.58.207.206, 2a00:1450:400f:80b::200e
Ansluter till google.com (google.com)|216.58.207.206|:443... ansluten.
HTTP-begäran skickad, väntar på svar... 
  HTTP/1.1 301 Moved Permanently
  Location: https://www.google.com/
  Content-Type: text/html; charset=UTF-8
  Date: Fri, 18 Sep 2020 14:23:08 GMT
  Expires: Sun, 18 Oct 2020 14:23:08 GMT
  Cache-Control: public, max-age=2592000
  Server: gws
  Content-Length: 220
  X-XSS-Protection: 0
  X-Frame-Options: SAMEORIGIN
  Alt-Svc: h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Adress: https://www.google.com/ [följer]
--2020-09-18 16:23:08--  https://www.google.com/
Slår upp www.google.com (www.google.com)... 216.58.211.132, 2a00:1450:400f:80a::2004
Ansluter till www.google.com (www.google.com)|216.58.211.132|:443... ansluten.
HTTP-begäran skickad, väntar på svar... 
  HTTP/1.1 200 OK
  Date: Fri, 18 Sep 2020 14:23:08 GMT
  Expires: -1
  Cache-Control: private, max-age=0
  Content-Type: text/html; charset=ISO-8859-1
  P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
  Server: gws
  X-XSS-Protection: 0
  X-Frame-Options: SAMEORIGIN
  Set-Cookie: NID=204=MwXITMuiAp-DzTP-Lvj-9WZWo2YrRE7KqPZsL2Ww21M55z0kUfzbeuLErCTGjhLR0A5-jCOSKYgdhVr3RshKIlJFIXxzs-QDKAkbRYSYBvxHcUFMrpQfNM1ZAIYN0D5r07TjVqbWG1WCqP3UCstNBgonfOY-DxZa-moP9BoBnRo; expires=Sat, 20-Mar-2021 14:23:08 GMT; path=/; domain=.google.com; HttpOnly
  Alt-Svc: h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
  Accept-Ranges: none
  Vary: Accept-Encoding
  Transfer-Encoding: chunked
Längd: ospecifierat [text/html]
Sparar till: "/dev/null"

/dev/null                                                      [ <=>                                                                                                                                     ]  12,93K  --.-KB/s    in 0,001s  

2020-09-18 16:23:08 (10,2 MB/s) - "/dev/null" sparades [13244]

It looks like HTTPS is working on your system as expected.
You could run: sudo update-ca-certificates for good measure.

Ok, thanks! I guess it’s some local problem. So far it has only manifested in python (including Kodi addons).

I tried sudo update-ca-certificates but it still fails verification.

This is reported a few times on the youtube-dl GitHub issue tracker.

It’s possible a Python dependency is not installed; so while the certificates are present on the system, they can’t be used yet due to a missing Python package.

Nice of you to look it up! I did look at their similar issues, but didn’t come across a recent one. I’ll take another look.

Thanks for you replies. You’re lightning fast.

Wget rejects the cert as well:

$ wget --server-response  -O /dev/null https://devstreaming-cdn.apple.com
--2020-09-18 16:44:29--  https://devstreaming-cdn.apple.com/
Slår upp devstreaming-cdn.apple.com (devstreaming-cdn.apple.com)... 17.253.39.206, 17.253.39.202, 2a01:b740:a08:f000::3, ...
Ansluter till devstreaming-cdn.apple.com (devstreaming-cdn.apple.com)|17.253.39.206|:443... ansluten.
FEL: Certifikatet för "devstreaming-cdn.apple.com" är inte pålitligt.
FEL: Certifikatet för "devstreaming-cdn.apple.com" saknar en känd utfärdare.

Wget doesn’t appear to depend on python.

Does this change anything?

It works here.

So maybe one of their CDN endpoints doesn’t have the correct certificate.
My guess is that the issue will resolve itself in a few hours.

Thanks for trying it out!

Got it. Can’t be helped. Good to hear it wasn’t anything on OSMC’s end.

Thanks for your time.

Just making a note that the last command fails repeatedly on OSMC and succeeds repeatedly on macOS 10.14 and Ubuntu 18.04 from the same IP. Not sure if that is indicative of anything. I’ll try again tomorrow.

Tried a few tests. Pi4, Pi3, Vero4 fail.
Ubuntu 20, OpenSuse work.

Looks like a issue with an outdated CA root certificate. Even if you re-install the ca certs bundle it still given an issue.

As a test I downloaded the latest cacerts.pem from the curl website and pointed to that and the connection is successful.

With the failure:

* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate

Then with the latest CA cert:

wget https://curl.haxx.se/ca/cacert.pem
curl -v --cacert cacert.pem https://devstreaming-cdn.apple.com
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=devstreaming-cdn.apple.com; OU=management:idms.group.665035; O=Apple Inc.; ST=California; C=US
*  start date: Mar 11 21:30:01 2019 GMT
*  expire date: Apr  9 21:30:01 2021 GMT
*  subjectAltName: host "devstreaming-cdn.apple.com" matched cert's "devstreaming-cdn.apple.com"
*  issuer: CN=Apple IST CA 8 - G1; OU=Certification Authority; O=Apple Inc.; C=US
***  SSL certificate verify ok.**

You will have to wait for the CA certs to be updated upstream by Debian.

1 Like

Interesting.

Except it worked for Sam. Something doesn’t add up.

Version 20190110 of ca-certificates, which is the version on stable Debian buster, still works. I supect that’s the one Sam used.

I ran a strace on the curl command above and it produced these lines:

stat64("/etc/ssl/certs/116bf586.0", 0xffe4d300) = -1 ENOENT (No such file or directory)
stat64("/etc/ssl/certs/e3d20ab2.0", 0xffe4d300) = -1 ENOENT (No such file or directory)

The first hash points to GeoTrust_Primary_Certification_Authority_-_G2.pem
The second hash can’t be found.

The GeoTrust certificate doesn’t appear in the 20200601 package.

Indeed, tested on Buster.

We will have a Buster testing thread up by tomorrow evening. We don’t anticipate many problems, but the more testers the merrier.

Sam

  1. The problem has already been noted here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962596

  2. You’ll pick up the 20200601 version if buster-updates is in your sources.list, which is likely to be the case on OSMC. Debian -- Package Search Results -- ca-certificates

1 Like

Furthermore, last time we had a major Debian-specific certificate
verification issue, we discovered that Debian is not actually capable
of restoring previously-removed certificates without manual user
intervention, see
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=743339. That means
that even once these certificates are restored, users who have already
updated to the affected version of ca-certificates will suffer
permanently broken certificate verification unless they have found this
bug report and know to take manual intervention, because the
certificates will remain disabled locally.

That sounds reassuring

Just to be clear, does this mean that OSMC can restore the cert with an update even though the Debian bug report thread claims that Debian can’t?

Debian controls the contents of its packages. OSMC uses whatever certificates are provided from Debian’s repository.

You can try to reinstall the older ca-certificates package, which is still available:

sudo apt-get install --reinstall ca-certificates=20190110

It worked successfully on my Pi3, though that’s no guarantee of success on your box and, though it’s a downgrade, is the version currently provided for Debian Buster/stable. Debian -- Package Search Results -- ca-certificates

As I understood my quoted excerpt from the bug thread the issue is not with what certificates Debian provide, and that it won’t work even when the certificate is reintroduced into the package, since if that were the case then there would be no problem to speak of.

Do you mean that you received certificate errors, then downgraded, and the problem was gone? Or that you succeeded in downgrading the package?