Not scraping TVDB https since update

Hi,

Log uploaded to: https://paste.osmc.tv/poxovesinu

Unable to add new shows or episodes since an update on 8 October.

Getting errors on TVDB scraping (extract from uploaded log for context):

2020-10-12 05:59:42.833 T:3938620128   DEBUG: VideoInfoScanner: Scanning dir 'nfs://192.168.1.19/raid/data/Audio_video/8.SickRage/neXt (2020)/' as not in the database
2020-10-12 05:59:42.833 T:3938620128   DEBUG: VideoInfoScanner: Found episode match nfs://192.168.1.19/raid/data/Audio_video/8.SickRage/neXt (2020)/neXt.2020_S01/neXt.2020_S01E01_FILE.#1_1080p.WEB.DL.x264.mkv (s1e1) [s([0-9]+)[ ._x-]*e([0-9]+(?:(?:[a-i]|\.[1-9])(?![0-9]))?)([^\\/]*)$]
2020-10-12 05:59:42.842 T:3938620128   DEBUG: GetEpisodeList: Searching 'https://api.thetvdb.com/login?{"apikey":"6aa6e4ecae5b56e9644f6a303c0739b6","id":359350}|Content-Type=application/json' using The TVDB scraper (file: '/home/osmc/.kodi/addons/metadata.tvdb.com', content: 'tvshows', version: '3.2.8')
2020-10-12 05:59:42.857 T:3938620128   DEBUG: CurlFile::ParseAndCorrectUrl() adding custom header option 'Content-Type: application/json'
2020-10-12 05:59:42.857 T:3938620128   DEBUG: CurlFile::Open(0xdde010e8) https://api.thetvdb.com/login
2020-10-12 05:59:42.857 T:3938620128    INFO: easy_acquire - Created session to https://api.thetvdb.com
2020-10-12 05:59:44.265 T:3938620128   ERROR: CCurlFile::FillBuffer - Failed: HTTP returned error 401
2020-10-12 05:59:44.265 T:3938620128   ERROR: CCurlFile::Open failed with code 401 for https://api.thetvdb.com/login:
2020-10-12 05:59:44.265 T:3938620128   ERROR: Run: Unable to parse web site
2020-10-12 05:59:44.267 T:3938620128 WARNING: No information found for item 'nfs://192.168.1.19/raid/data/Audio_video/8.SickRage/neXt (2020)/', it won't be added to the library.
2020-10-12 05:59:44.269 T:3938620128   DEBUG: VideoInfoScanner: No (new) information was found in dir nfs://192.168.1.19/raid/data/Audio_video/8.SickRage/neXt (2020)/

I use SickRage to generate the tvshow.nfo file that has the url in it:
neXt (2020)/tvshow.nfo

    <episodeguide>
        <url cache="auth.json" post="yes">https://api.thetvdb.com/login?{&amp;quot;apikey&amp;quot;:&amp;quot;6aa6e4ecae5b56e9644f6a303c0739b6&amp;quot;,&amp;quot;id&amp;quot;:359350}|Content-Type=application/json</url>
    </episodeguide>

To produce the error from the Ver4k+ interface:

Videos → Files → → neXt (2020) → Scan for new content

I get this message for other shows.videos, and even if I do a full scan rather than just an individual series scan.

On 6th October I installed ‘jq’ via apt-get:
sudo apt-get install jq

New episodes were added to Kodi after this date.

On the 8th of October, I did an update:

sudo apt-get update
sudo apt-get dist-upgrade

Since that date (8th) no new episodes for any TV show have been added to Kodi, even though I have received several new ones.

The install logs (/var/log/apt/history.log) for that time show:

Start-Date: 2020-10-06  19:27:51
Commandline: /usr/bin/apt-get-real install jq
Requested-By: osmc (1000)
Install: jq:armhf (1.5+dfsg-1.3), libjq1:armhf (1.5+dfsg-1.3, automatic), libonig4:armhf (6.1.3-2, automatic)
End-Date: 2020-10-06  19:27:54

Start-Date: 2020-10-08  22:38:57
Commandline: /usr/bin/apt-get-real dist-upgrade
Requested-By: osmc (1000)
Upgrade: perl-base:armhf (5.24.1-3+deb9u6, 5.24.1-3+deb9u7), libcomerr2:armhf (1.43.4-2+deb9u1, 1.43.4-2+deb9u2), libisccfg140:armhf (1:9.10.3.dfsg.P4-12.3+deb9u6, 1:9.10.3.dfsg.P4-12.3+deb9u7), libcups2:armhf (2.2.1-8+deb9u5, 2.2.1-8+deb9u6), libdbus-1-3:armhf (1.10.28-0+deb9u1, 1.10.32-0+deb9u1), libcurl3:armhf (7.52.1-5+deb9u10, 7.52.1-5+deb9u12), libssh-4:armhf (0.7.3-2+deb9u2, 0.7.3-2+deb9u3), python-imaging:armhf (4.0.0-4+deb9u1, 4.0.0-4+deb9u2), dbus:armhf (1.10.28-0+deb9u1, 1.10.32-0+deb9u1), python2.7-minimal:armhf (2.7.13-2+deb9u3, 2.7.13-2+deb9u4), libsqlite3-0:armhf (3.16.2-5+deb9u1, 3.16.2-5+deb9u2), bind9-host:armhf (1:9.10.3.dfsg.P4-12.3+deb9u6, 1:9.10.3.dfsg.P4-12.3+deb9u7), e2fsprogs:armhf (1.43.4-2+deb9u1, 1.43.4-2+deb9u2), libpython2.7:armhf (2.7.13-2+deb9u3, 2.7.13-2+deb9u4), python2.7:armhf (2.7.13-2+deb9u3, 2.7.13-2+deb9u4), libisc160:armhf (1:9.10.3.dfsg.P4-12.3+deb9u6, 1:9.10.3.dfsg.P4-12.3+deb9u7), e2fslibs:armhf (1.43.4-2+deb9u1, 1.43.4-2+deb9u2), libss2:armhf (1.43.4-2+deb9u1, 1.43.4-2+deb9u2), liblwres141:armhf (1:9.10.3.dfsg.P4-12.3+deb9u6, 1:9.10.3.dfsg.P4-12.3+deb9u7), wpasupplicant:armhf (2:2.4-1+deb9u4, 2:2.4-1+deb9u7), libdns162:armhf (1:9.10.3.dfsg.P4-12.3+deb9u6, 1:9.10.3.dfsg.P4-12.3+deb9u7), libmariadbclient18:armhf (10.1.44-0+deb9u1, 10.1.45-0+deb9u1), libxml2:armhf (2.9.4+dfsg1-2.2+deb9u2, 2.9.4+dfsg1-2.2+deb9u3), libpython2.7-minimal:armhf (2.7.13-2+deb9u3, 2.7.13-2+deb9u4), libgnutls30:armhf (3.5.8-5+deb9u4, 3.5.8-5+deb9u5), libnss3:armhf (2:3.26.2-1.1+deb9u1, 2:3.26.2-1.1+deb9u2), libisccc140:armhf (1:9.10.3.dfsg.P4-12.3+deb9u6, 1:9.10.3.dfsg.P4-12.3+deb9u7), libbind9-140:armhf (1:9.10.3.dfsg.P4-12.3+deb9u6, 1:9.10.3.dfsg.P4-12.3+deb9u7), libpython2.7-stdlib:armhf (2.7.13-2+deb9u3, 2.7.13-2+deb9u4), curl:armhf (7.52.1-5+deb9u10, 7.52.1-5+deb9u12), libjpeg62-turbo:armhf (1:1.5.1-2, 1:1.5.1-2+deb9u1), python-pil:armhf (4.0.0-4+deb9u1, 4.0.0-4+deb9u2), nfs-common:armhf (1:1.3.4-2.1, 1:1.3.4-2.1+deb9u1), libssl1.0.2:armhf (1.0.2u-1~deb9u1, 1.0.2u-1~deb9u2), base-files:armhf (9.9+deb9u12, 9.9+deb9u13), tzdata:armhf (2019c-0+deb9u1, 2020a-0+deb9u1)

Is there anything I’ve left out?

Edited to add: only ‘urls’ using the https://api.thetvdb.com/login? (the V3 API?) url are failing, those shows added a while ago that are using the older API, http://thetvdb.com/api/, seem to be working.

This sounds like a issue you would need to take up with SickRage. I assume if you brought up a full screen info on one of these shows, clicked refresh, scan all, ignore local files, then scraping for that show would work again.

Indeed, the TVDB site is giving you a 401 “unauthorized access” for the line:

2020-10-12 05:59:44.293 T:3938620128 DEBUG: GetEpisodeList: Searching 'https://api.thetvdb.com/login?{&quot;apikey&quot;:&quot;6aa6e4ecae5b56e9644f6a303c0739b6&quot;,&quot;id&quot;:359350}|Content-Type=application/json' using The TVDB scraper (file: '/home/osmc/.kodi/addons/metadata.tvdb.com', content: 'tvshows', version: '3.2.8')

If I run it from the command line (slightly altered), I get this:

osmc@osmc:~$ curl -d '{"apikey":"6aa6e4ecae5b56e9644f6a303c0739b6", "id":359350}' -H 'Content-Type=application/json' https://api.thetvdb.com/login
{"Error":"Bad Content-Type or charset, expected 'application/json'"}osmc@osmc:~$

Changing Content-Type=application/json to Content-Type:application/json returns a token, which I assume is the correct response:

osmc@osmc:~$ curl -d '{"apikey":"6aa6e4ecae5b56e9644f6a303c0739b6", "id":359350}' -H 'Content-Type:application/json' https://api.thetvdb.com/login
{"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MDMxMzk0OTgsImlkIjoic2lja2NoaWxsIiwib3JpZ19pYXQiOjE2MDI1MzQ2OTh9.dH-DFiIoKk7GIXHCXbEZZKT9PeyJwAOxFN141YZk9llQyfW0yTfhNhKsrY1YQWbmun5jTtFERgn70V5zoUZuMxe7h71U3esoo7ARonRiaTpUXbiDMyrx-HWkODBgBsIfAhuApAgf2_Dcs_ISBzSK-53mn4GX49BArPXu-3B26w-tyr1YPn3b_jWKMxH7E5Bvo3-Vo1GD164IIsE2wWEvAc_CFWf6Q8a3irvPv5AtM2x1bxaCBtaDVfU_A3Hj9tzqvyh0O6rW7LwtPG1VOyYW7crPAWiu329woLfd106Hu5Hw747WVwY-ShCL2J8AIrAxGf9bt0mQn0fpd3bW7wCSpw"}osmc@osmc:~$ 

So I’m not sure what’s going on here. Perhaps @darwindesign has a better idea.

Of course, this doesn’t explain why you get a 401 “unauthorized” error. :wink: Wikipedia does mention:

Note: Some sites incorrectly issue HTTP 401 when an IP address is banned from the website (usually the website domain) and that specific address is refused permission to access a website.