• 1(current)
  • 2
  • 3
  • 4
  • 5
  • 11
v17 TVDB scraper failing login to thetvdb API suddenly
#1
All of a sudden in the last two days (with no changes to Kodi or the scraper), the TVDB scraper stopped working.

This is Kodi v17.6 running on Linux (Opensuse Leap). TVDB scraper 3.0.1

Looking at the logs, it seems the initial call to login to pass the API Key for kodi to get the token for further calls is failing with SSL error 35.


16:56:22.506 T:140371273099008   DEBUG: scraper: CreateSearchUrl returned <url post="yes" cache="auth.json">https://api.thetvdb.com/login?{"apikey":"439DFEBA9D3059C6"}|Content-Type=application/json</url>
16:56:22.506 T:140371273099008   DEBUG: CurlFile:: ParseAndCorrectUrl() adding custom header option 'Content-Type: application/json'
16:56:22.506 T:140371273099008   DEBUG: CurlFile::Open(0x7fab20a92e30) https://api.thetvdb.com/login
16:56:22.506 T:140371273099008    INFO: easy_aquire - Created session to https://api.thetvdb.com
16:56:22.618 T:140371273099008   ERROR: CCurlFile::FillBuffer - Failed: SSL connect error(35)
16:56:22.618 T:140371273099008   ERROR: CCurlFile::Open failed with code 0 for https://api.thetvdb.com/login|Content-Ty...ation/json
16:56:22.618 T:140371273099008   ERROR: Run: Unable to parse web site


The curl to the API from command line on the same machine works fine and gets the token

%  curl -d '{"apikey":"439DFEBA9D3059C6"}' -H "Content-Type: application/json" https://api.thetvdb.com/login
{
  "token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1MjA1NjEyMjUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTIwNDc0ODI1fQ.zF-DzODZGeKKTqS1EU1p3WOWfo1rFfhoa5pSmy6mlAB94z4GoiL0PSsrcIZCRJZ64R0f01qjg8VddP16lKdmpbVWxCqhWM21RQY5r5WG-N-mKPIzAJIqlbqf9csnpzANBrB0uUmsblMaKKWZfM-mpq0_vGhpxZ0EP_pgvDk72YfYuKNzY-viADDa3EdJXhEo3GVcpc1CkKvp2MMKIozALQLvdYchxwMrvO-TupQ0HiJSzVzH3GsaUEYxfVZ5w6wcys1zj9MkIN8GFhrtJN4ZXSxNGX8beQO9qxXBt3BetN3RF2h-Lf_GhVGaGGbnd2Fm9mmpd2_3wNsRau1qrQprIQ"
}


Any idea what may have changed underneath?
Reply
#2
This is odd. Cannot even download the TVDB add-on to re-install

18:19:38.527 T:140370201089792   DEBUG: CurlFile::Open(0x7fab205aebb0) http://mirrors.kodi.tv/addons/krypton/me...-3.0.1.zip
18:19:39.145 T:140370201089792   ERROR: CCurlFile::FillBuffer - Failed: SSL connect error(35)
18:19:39.145 T:140370201089792   DEBUG: CCurlFile::Open - effective URL: <https://mirror.de.leaseweb.net/xbmc/addons/krypton/metadata.tvdb.com/metadata.tvdb.com-3.0.1.zip>
18:19:39.146 T:140370201089792   ERROR: CAddonInstallJob[metadata.tvdb.com]: MD5 mismatch after download. Expected 514413076525b702fbd3e45bcf708b34, was D41D8CD98F00B204E9800998ECF8427E
Reply
#3
Please stop posting snippets that you think are important, and post the whole log so we can get the bigger picture.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#4
Never mind. I will debug on my own and fix it.

When I post whole logs (after redacting all the identifying info in the path names, etc), people look at it just to see if there are illegal add-ons and when they find there are none, don't respond to the problem.

No response even when I debug the whole problem and post the solution to the problem.

For example,
https://forum.kodi.tv/showthread.php?tid...id=2702379

I don't think there is any additional "whole picture" here because the problem is so localized and modular.

Sorry to bother you guys.
Reply
#5
You are sort of correct.

The first thing we analyse is the presence of Banned Add-ons. That is no secret.

Once it is confirmed that your system is clean, then you become eligible for support.

Not all problems can be solved by all members and Team Members. We simply provide ASSISTANCE and do not, in any way, advertise that we provide guaranteed fixes for all your problems.

There are just too many variables in setups, hardware, add-ons, platform, other installed software, updates, drivers... the list is almost endless. But when one person, out of around 50million users reports a problem, it is specific to their setup.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#6
For other people who might encounter this problem, the root cause of this is a mess created by the use of < Python3 in addons like the TVDB addon. This is not something you find in the debug log.

Linux distributions need a minimum version of 16.2 for python-openssl (or python-pyOpenSSL) and minimum version of  1.7.1 of pything-cryptography to avoid a bug in earlier versions. Many linux distributions discontinued support for Python 2 and haven't updated it to these minimum versions of these libraries but should be available in your repository.

Other symptoms include high CPU usage and inability of Kodi to exit as these built-in addons don't quit and keep writing to .xsession-errors creating a very large file.

Updating your Linux installation to the above minimum libraries should fix this problem.

Not sure what suddenly triggered this, perhaps TVDB auto updated itself to use Python scrapers.

These kinds of Kodi gotchas are on the increase, unfortunately.
Reply
#7
@Common Man

Thank you for posting your findings.

Not something that I am knowledgeable in, but maybe @olympia might be able to comment.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#8
TVDB scraper is NOT a python scraper.

You seem to be quite knowledgeable and smart enough. Equipped with strong self confidence to insult Kodi contributors and team members willing to assist you. So please, please contribute and help us making things better and smoother, so others will not be as disappointed as you are.
Reply
#9
Just to leave additional information here on why this problem happened all of a sudden:

The change that created this problem is that thetvdb.com started redirecting the http url to https url which failed with the 2.7 python libraries linked to Kodi because of bugs in the two dependent libraries mentioned above

ldd /usr/lib64/kodi/kodi.bin | grep py
        libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0

This is also why the addon download itself above failed when directed to a https repository url.

tvdb scraper itself has no code, presumably it uses some common code for scrapers which eventually leads to calls to the pyCurl library which then uses the Python library linked above and that fails unless the later crypto and openssh libraries are used. The actual mechanism is irrelevant to fix this problem.

Addressing the sarcastic and passive aggressive comment from Olympia above:
These forums should really be moved to a stackexchange forum so people can help each other. Avoid having to deal with snarky Kodi staff acting passive-aggressive and penalizing people who don't use illegal addons by making them jump through hoops unnecessarily or simply accusing them of using illegal streaming even when there are perfectly legal situations (see my thread on the stutter of death problem for this obnoxious behavior). So don't expect people to have much respect for what has happened to these forums in the last two years or the Kodi team. It takes two to tango. Don't want to get involved in that mess of a situation.
Reply
#10
This is affecting Windows as well, I'm guessing if I manually fix this same pyCurl issue it will work again.

In windows it hangs in the background even when the scraper is cancelled.  Exiting Kodi it goes away but then it requires a knifing from Task Manager.  Attempting to scrape again results in same.  Tried latest nightly since I was already running a previous one from last month / no change.
Reply
#11
May or may not be the same problem. If it is the same problem I discovered above, it should fail right at the beginning since it cannot even login to thetvdb.com using the redirected https url. So it will fail immediately with an error message. If it is doing the scan for a few items and hanging at some point, then it is a different problem.

Moreover, I believe the python libraries on Windows come with Kodi and so should have working versions. I could be wrong about this.

I have found several scenarios where it may hang because it doesn't seem to time out for certain error conditions.

1. If you remove a particular source you have added which it had scanned earlier (and so has the data in the database) and scan again, it will likely hang. The only solution to this is to delete and recreate the library or add that source back.
2. If there is an earlier scanned source that is a network mounted file system (SMB or NFS) which isn't available for whatever reason when you scan, it may hang for the same reasons as above. The SMB version updates have created a problem for Kodi handling of SMB shares. You can detect if this is the problem by trying to mount a new source with SMB (if you are using Samba/Windows shares) and trying to browse it under Video->Files. If it hangs there, then it is a Samba access problem.
3. There is a smaller likelihood of it hanging if one of the items being scanned returns data it doesn't like from the tvdb site. This one is difficult to locate. See if there are any database update errors in the log.

If I were to guess, it is likely 1 or 2 above.

Try cleaning the library from Settings-> Media->Library and see if that completes. If it doesn't, then it is likely 1 or 2 above. If it completes, it is possible that the next scan will succeed.

Good luck.
Reply
#12
having issues with thetvdb as well in windows on leia nightly, here's a snippet below and the log file, i don't have debug enabled atm I will enable though and post another log if necessary

edit: I have also been removing/adding sources and rebuilding as Common Man was alluding the problem may be related to, as I recently changed the password on my smb source. Even after deleting the passwords.xml file I could no longer connect by IP address and was forced to use the name as a source instead (another strange yet unrelated issue).

https://drive.google.com/file/d/1W-KB7q-...sp=sharing

small snippet:

Code:
08:55:51.423 T:7032   ERROR: script.module.thetvdb --> Exception in get_data --> ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='api.thetvdb.com', port=443): Read timed out. (read timeout=20)",),)
08:55:52.176 T:12088   ERROR: CCurlFile::FillBuffer - Failed: Timeout was reached(28)
08:55:52.176 T:12088   ERROR: CCurlFile::Open failed with code 0 for https://api.thetvdb.com/episodes/372785|...anguage=en
08:55:52.176 T:12088   ERROR: ADDON::CScraper::Run: Unable to parse web site
08:55:53.982 T:6232   ERROR: script.module.thetvdb --> Exception in get_data --> ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='api.thetvdb.com', port=443): Read timed out. (read timeout=20)",),)
08:56:49.625 T:12088   ERROR: Previous line repeats 7 times.
08:56:49.625 T:12088   ERROR: CCurlFile::FillBuffer - Failed: Timeout was reached(28)
08:56:49.625 T:12088   ERROR: CCurlFile::Open failed with code 0 for https://api.thetvdb.com/episodes/4266033...anguage=en
08:56:49.625 T:12088   ERROR: ADDON::CScraper::Run: Unable to parse web site
leia nightly / nvidia shield / win 10 64-bit | 16gb ddr4 | intel i7-7700k | titan x pascal
Reply
#13
same here, having issues contacting the tvdb see the log. 

https://pastebin.com/kJC0nrJ3
How to post a debug log ; MacOS acces the hidden userdata folder ; How to post a question ; How to fix gatekeeper issues
Reply
#14
The last two seem more like a problem with tvdb service. Could be temporary. See if it persists.
Reply
#15
http://www.isitdownrightnow.com/api.thetvdb.com.html

check that first. if the api is down the scraper won't connect.  known issue on their end since v2.0.
Reply
  • 1(current)
  • 2
  • 3
  • 4
  • 5
  • 11

Logout Mark Read Team Forum Stats Members Help
TVDB scraper failing login to thetvdb API suddenly0