Guest - Testers are needed for the reworked CDateTime core component. See... https://forum.kodi.tv/showthread.php?tid=378981 (September 29) x
  • 1
  • 5
  • 6
  • 7(current)
  • 8
  • 9
  • 47
Release TMDb TV Show scraper (Python - Default Matrix Scraper)
#91
@Karlson2k

Can you click on the below link and tell us what you see. Is the metadata in Russian or English

https://api.tmdb.org/3/tv/70384?api_key=...ru,en,null
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#92
(2020-11-16, 21:26)Hebotsuki Wrote:
(2020-11-15, 05:34)pkscout Wrote:
(2020-11-14, 20:58)Hebotsuki Wrote: It's not working for me either. According to Travis from TheMoviedb, some new fileds have been added to the credits in order to a future support for translated person names. That's probably what broke the scraper. I guess the scraper developpers are already aware of the problem.
Just FYI that there is an update available from the repo now for this. If you don't get it automatically, you can force check for updates, and that should download it.
Thanks, @pkscout. I installed the latest version on leia and seems to work fine: actors, episodes, fallback to English works with titles and plot with no translation, etc. I don't see any missing information. I only have one question: which writing credits are scraped for the episodes? I tested a couple of TV shows and one of them doesn't scrape the writing credits, but in TMDb there are "screenplay" and "storyboard" for the writing credits. The other one only scraped the "writer" credits, but no the "stroryboard" writing credit, which is in TMDb too.

Anyway, thanks a lot for the work!
Writer is the only type of crew that is added in Kodi. That's to keep it consistent with the movie scraper. I can look at adding the other ones, but it's a bit of a whack-a-mole problem. Anytime another kind of writer is added, I'd have to update the scraper.  I'll take another look though. There might be a way to do it using "department" instead of roles.
Reply
#93
@Karlson2k

We think we have found the problem, but it will take some time to fix.

In the meantime, can you Refresh the tv show with the new scraper (no need to refresh the episodes when asked). This will update the episodeguide URL and allow you to scrape with the new scraper.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#94
Hi, I found what I think is another issue with the episode groups scraping. It seems like the scraper maps groups onto season numbers by just numbering them 1 to N, in the order in which the they were returned by the API. While this generally works fine with most groupings, where the groups are just seasons 1 to N or volumes 1 to N, it breaks on some shows that still have a "specials" grouping. If your specials group is at the beginning it ends up being season #1 and every other season/volume is off-by-one, or it ends up being some extra later season. For an example of this see https://www.themoviedb.org/tv/2418-the-v...219bbfa30e which has a Specials group ordered as #0 but which the scraper will number as "season 8".

I think the original TMDB scraper used the "order" field as the season number, which generally doesn't have this problem; most of the time it matches the sorted order anyway, and in cases where it doesn't it's an editable element of the data that's can be fixed.
Reply
#95
(2020-11-17, 08:10)jads Wrote: Hi, I found what I think is another issue with the episode groups scraping
Hmmm, yes I see what you mean.

You can spot it in the log on line 970 and 1809... https://paste.kodi.tv/osezudimov.kodi
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#96
I did not manage under any circumstances to paste the log on any of the two mentioned sites, but i did upload it to dropbox. Link: https://www.dropbox.com/s/dbz3bo6gun11rlj/kodi.log?dl=0

I have 3 shows that don't work, Fear the Walking Dead, Las Fierbinti and The Walking Dead, please have a look maybe you can find the issue, thank you in advance!
Reply
#97
@hellcry

You are using an old version of the scraper. Before I look at the issue, please update to v1.1.23 by downloading the zip file and installing via zip... https://github.com/xbmc/metadata.tvshows.../tree/leia

There have been quite a few updated in the last two days and one or two more to come, and once all problems have been fixed it will be pushed to the repo, but that will be a couple of days still.

Let us know if you still have problems.

@Karlson2k
The link above for v1.1.23 is the fix for your problem.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#98
(2020-11-17, 09:46)Karellen Wrote: @hellcry

You are using an old version of the scraper. Before I look at the issue, please update to v1.1.23 by downloading the zip file and installing via zip... https://github.com/xbmc/metadata.tvshows.../tree/leia

There have been quite a few updated in the last two days and one or two more to come, and once all problems have been fixed it will be pushed to the repo, but that will be a couple of days still.

Let us know if you still have problems.

@Karlson2k
The link above for v1.1.23 is the fix for your problem.

Thank you very much, I did the zip install and I can confirm all is working great.
Reply
#99
(2020-11-17, 08:10)jads Wrote: Hi, I found what I think is another issue with the episode groups scraping. It seems like the scraper maps groups onto season numbers by just numbering them 1 to N, in the order in which the they were returned by the API. While this generally works fine with most groupings, where the groups are just seasons 1 to N or volumes 1 to N, it breaks on some shows that still have a "specials" grouping. If your specials group is at the beginning it ends up being season #1 and every other season/volume is off-by-one, or it ends up being some extra later season. For an example of this see https://www.themoviedb.org/tv/2418-the-v...219bbfa30e which has a Specials group ordered as #0 but which the scraper will number as "season 8".

I think the original TMDB scraper used the "order" field as the season number, which generally doesn't have this problem; most of the time it matches the sorted order anyway, and in cases where it doesn't it's an editable element of the data that's can be fixed.

Thank you for pointing this out. Yes, the seasons are being done sequentially.  All the shows I tested with didn't have specials in the episode grouping, so it worked. I have an update for you to test that is not yet part of the main branch that uses the order field instead of the sequence counting.  I tested with Venture Bros, Firefly, and Money Heist, and it seems to work as expected.

Matrix: https://github.com/xbmc/metadata.tvshows...-groupings
Leia: https://github.com/xbmc/metadata.tvshows...pings-leia

Just as a note, I didn't test the Leia code, but unless I made a typo, it is the same as the Matrix code (which I did test) and should work.
Reply
(2020-11-17, 01:48)Karellen Wrote: @Karlson2k

Can you click on the below link and tell us what you see. Is the metadata in Russian or English

https://api.tmdb.org/3/tv/70384?api_key=...ru,en,null
It's Russian.
(2020-11-17, 09:46)Karellen Wrote: @hellcry

You are using an old version of the scraper. Before I look at the issue, please update to v1.1.23 by downloading the zip file and installing via zip... https://github.com/xbmc/metadata.tvshows.../tree/leia

There have been quite a few updated in the last two days and one or two more to come, and once all problems have been fixed it will be pushed to the repo, but that will be a couple of days still.

Let us know if you still have problems.

@Karlson2k
The link above for v1.1.23 is the fix for your problem.
I've tested 1.1.24
Works fined, picked-up correctly data from non-python scraper.
https://paste.kodi.tv/sohidemitu.kodi

Two things to note:
  1. For single episode the scraper downloaded information for two seasons in two languages. Looks suboptimal.
  2. After every
    Code:
    DEBUG: [metadata.tvshows.themoviedb.org.python (1.1.24)]: Called addon with params: ['plugin://metadata.tvshows.themoviedb.org.python/', '42', '...
    there is matching
    Code:
    WARNING: Attempt to use invalid handle 42
     with the same number.
Reply
(2020-11-17, 12:28)pkscout Wrote: Matrix: https://github.com/xbmc/metadata.tvshows...-groupings
Leia: https://github.com/xbmc/metadata.tvshows...pings-leia

Just as a note, I didn't test the Leia code, but unless I made a typo, it is the same as the Matrix code (which I did test) and should work.

Tested on Leia, works fine as well.
https://paste.kodi.tv/imijupuruk.kodi 
(addon.xml still states that it's 1.1.24, while actually it's 1.1.25)
Warnings about "invalid handle" are still present. 

Even when refreshed info without data update, there is "invalid handle" warning: https://paste.kodi.tv/oyazolujef.kodi
Reply
@pkscout, why this scraper put temporally files to ".kodi/userdata/addon_data/metadata.tvshows.themoviedb.org.python/cache" instead of standard location ".kodi/temp/scrapers/metadata.tvshows.themoviedb.org.python"?
I have ".kodi/temp" mounted on RAM disk to avoid SSD trashing and speedup data processing and now it's required to use some tricks to move this scraper temp files to RAM drive.
Reply
(2020-11-17, 12:28)pkscout Wrote: Thank you for pointing this out. Yes, the seasons are being done sequentially.  All the shows I tested with didn't have specials in the episode grouping, so it worked. I have an update for you to test that is not yet part of the main branch that uses the order field instead of the sequence counting.  I tested with Venture Bros, Firefly, and Money Heist, and it seems to work as expected.

Matrix: https://github.com/xbmc/metadata.tvshows...-groupings
Leia: https://github.com/xbmc/metadata.tvshows...pings-leia

Just as a note, I didn't test the Leia code, but unless I made a typo, it is the same as the Matrix code (which I did test) and should work.

I tried running a copy of the Leia code and it seems to do the right thing on my shows now. I don't have a Matrix instance set up so I can't (quickly) test that one myself.

Just as an aside, it might also be safer to sort the episodes list by the 'order' key before sequentially numbering them. While I haven't seen any cases where the API returns episodes in a different order (and unlike with the season groups, you can't explicitly set or edit the order #) I'm not sure it's safe to assume the API will actually return the episode list in a particular order.
Reply
(2020-11-17, 13:02)Karlson2k Wrote: I've tested 1.1.24
Works fined, picked-up correctly data from non-python scraper.
https://paste.kodi.tv/sohidemitu.kodi

Two things to note:
  1. For single episode the scraper downloaded information for two seasons in two languages. Looks suboptimal.
  2. After every
    Code:
    DEBUG: [metadata.tvshows.themoviedb.org.python (1.1.24)]: Called addon with params: ['plugin://metadata.tvshows.themoviedb.org.python/', '42', '...
    there is matching
    Code:
    WARNING: Attempt to use invalid handle 42
     with the same number.
I know #1 feels suboptimal, but because of the order of operations for Kodi and the way the TMDb API works, it's actually better to get the season information for a show when the show is loaded than to wait until the episode is loaded to do the season.  Also I need the seasons if a show has a custom episode grouping, and that happens early in the process as well.  For #2 I have no idea where that handle is being generated, but I'm like 98% sure it's not in the scraper code, so it might be whatever core process calls the scraper.
Reply
(2020-11-17, 13:59)Karlson2k Wrote: @pkscout, why this scraper put temporally files to ".kodi/userdata/addon_data/metadata.tvshows.themoviedb.org.python/cache" instead of standard location ".kodi/temp/scrapers/metadata.tvshows.themoviedb.org.python"?
I have ".kodi/temp" mounted on RAM disk to avoid SSD trashing and speedup data processing and now it's required to use some tricks to move this scraper temp files to RAM drive.
I'm not familiar with the temp location.  I use a python call to get the addon_data location and then store the addon's data there. If there is some other Python call that will return the path to that temp directory on every platform, I'd be happy to change it. I'd rather not start doing platform specific code though.  As an aside, I'm using the same location the TV Maze scraper uses to store it's cache files.
Reply
  • 1
  • 5
  • 6
  • 7(current)
  • 8
  • 9
  • 47

Logout Mark Read Team Forum Stats Members Help
TMDb TV Show scraper (Python - Default Matrix Scraper)0