Login at Kodi Home

tr0nllam · 2020-10-01, 01:51

I'm loving the addition of the Universal TV Scaper, but I have found a couple issues.

1) It doesn't properly scrape DVD Ordered episodes. I mostly use a mixture of TVDB and IMDB and it's able to correctly identify the DVD ordered episode, but it doesn't scrape the DVD Season and DVD Episode info, so the episode shows up as unnumbered. It also isn't scraping Display Season or Display Episode for Specials.

2) Is there a way to scrape IMDB episode cast lists for Specials? I've noticed that even though Specials obviously don't match up on IMDB with the TVDB episode ordering, TMM is able to scrape the IMDB rating for those episodes. However, it seems unable to grab the cast.

**Karellen** · (This post was last modified: 2020-10-01, 02:01 by Karellen.)

(2020-10-01, 01:51)tr0nllam Wrote: It doesn't properly scrape DVD Ordered episodes. I mostly use a mixture of TVDB and IMDB and it's able to correctly identify the DVD ordered episode, but it doesn't scrape the DVD Season and DVD Episode info, so the episode shows up as unnumbered. It also isn't scraping Display Season or Display Episode for Specials.

DVD Orders at TVDB are partly broken. Won't be fixed until they release v4 API.
https://forum.kodi.tv/showthread.php?tid...pid2942013

(2020-10-01, 01:51)tr0nllam Wrote: It also isn't scraping Display Season or Display Episode for Specials.

Does that information exist at TVDB for those specials?

tr0nllam · 2020-10-01, 02:12

(2020-10-01, 01:59)Karellen Wrote:
(2020-10-01, 01:51)tr0nllam Wrote: It doesn't properly scrape DVD Ordered episodes. I mostly use a mixture of TVDB and IMDB and it's able to correctly identify the DVD ordered episode, but it doesn't scrape the DVD Season and DVD Episode info, so the episode shows up as unnumbered. It also isn't scraping Display Season or Display Episode for Specials.
DVD Orders at TVDB are partly broken. Won't be fixed until they release v4 API.
https://forum.kodi.tv/showthread.php?tid...pid2942013

(2020-10-01, 01:51)tr0nllam Wrote: It also isn't scraping Display Season or Display Episode for Specials.
Does that information exist at TVDB for those specials?

1) If I use the TVDB Scraper on the episodes in question, it does scrape the DVD Season and Episode field. However, when using the Universal TV Scraper those fields are not filled.

2) The specials I tested do have the Display Special/Episode fields filled in on TVDB. If I use the TVDB scraper for those episodes, the Display Season and Display Episode information is scraped, but it shows up as DVD Season/DVD Episode instead. If I use the Universal TV Scraper, neither field is scraped.

**mlaggner** · (This post was last modified: 2020-10-01, 07:49 by mlaggner.)

- there was a bug in the universal scraper - that's why tmm could only get the "normal" season/episode number. Will be fixed in the next version
- matching between tvdb/tmdb/imdb is done via IDs on show level and season/episode number on episode level... if the season/episode numbers between tvdb and imdb do not match, we're kinda out of luck. I may have a look if there is another possibility to match this

EDIT: could you give me a detailed example to analyze the problem?

tr0nllam · 2020-10-01, 08:44

(2020-10-01, 07:47)mlaggner Wrote: - there was a bug in the universal scraper - that's why tmm could only get the "normal" season/episode number. Will be fixed in the next version
- matching between tvdb/tmdb/imdb is done via IDs on show level and season/episode number on episode level... if the season/episode numbers between tvdb and imdb do not match, we're kinda out of luck. I may have a look if there is another possibility to match this

EDIT: could you give me a detailed example to analyze the problem?

The Doctor Who (2005) Specials are a good example. When "The Waters of Mars (S00E14)" is scraped with the Universal TV Scraper, it successfully retrieves the IMDB rating, but the cast is absent because that episode is essentially uncategorized on IMDB. Then, you have an episode like 'The Snowmen (S00E66)" which is listen as S07E07 on IMDB, but TMM is unable to retrieve the cast.

Similar issues crop up when a season on IMDB counts a two-part episode as one, while TVDB lists it as two. That will cause the entire season to be screwed up because the episode numbers are off by one. For example, Seinfeld Season 6 episode "The Highlights of 100" is listed as a two-part episode on TVDB, but IMDB lists it as a single episode. So, every episode after that is one episode off for all IMDB info, except the rating.

I just realized the reason I get correct IMDB episode ratings, but not the other IMDB info in these situations is because I chose Trakt as my episode ratings scraper. That ends up scraping all 4 ratings (TVDB, TMDB, Trakt, and IMDB) and manages to give the correct data for the episode. However, if I switch the episode ratings to IMDB, it gives the wrong info.

**mlaggner** · 2020-10-01, 08:51

(2020-10-01, 08:44)tr0nllam Wrote: I just realized the reason I get correct IMDB episode ratings, but not the other IMDB info in these situations is because I chose Trakt as my episode ratings scraper. That ends up scraping all 4 ratings (TVDB, TMDB, Trakt, and IMDB) and manages to give the correct data for the episode. However, if I switch the episode ratings to IMDB, it gives the wrong info.

that is no more needed just for ratings Wink

we use the official rating exports from IMDB to mix in IMDB ratings where possible (= where an IMDB id is available, because to mix that in we only need the IMDB id of the episode).. in my mind there is no more need for the universal scraper just to get ratings..

nevertheless, I will have a look at the other issues, but if IMDB lists the episodes different than TVDB/Trakt/... no software is able to get the real data Tongue

**Karellen** · 2020-10-01, 08:53

(2020-10-01, 08:44)tr0nllam Wrote: The Doctor Who (2005) Specials are a good example. When "The Waters of Mars (S00E14)" is scraped with the Universal TV Scraper, it successfully retrieves the IMDB rating, but the cast is absent because that episode is essentially uncategorized on IMDB.

What is the importance of scraping info from IMDB? What does IMDB give you that TVDB or TheMovieDB cannot give you?

tr0nllam · 2020-10-01, 09:04

(2020-10-01, 08:53)Karellen Wrote:
(2020-10-01, 08:44)tr0nllam Wrote: The Doctor Who (2005) Specials are a good example. When "The Waters of Mars (S00E14)" is scraped with the Universal TV Scraper, it successfully retrieves the IMDB rating, but the cast is absent because that episode is essentially uncategorized on IMDB.
What is the importance of scraping info from IMDB? What does IMDB give you that TVDB or TheMovieDB cannot give you?

I mainly just use it for the episode cast lists and ratings. IMDB has much more detailed cast lists for individual episodes.

**mlaggner** · 2020-10-01, 09:21

I've just written an article about the ratings mixin - https://www.tinymediamanager.org/blog/imdb-ratings/

tr0nllam · 2020-10-01, 09:24

(2020-10-01, 08:51)mlaggner Wrote:
(2020-10-01, 08:44)tr0nllam Wrote: I just realized the reason I get correct IMDB episode ratings, but not the other IMDB info in these situations is because I chose Trakt as my episode ratings scraper. That ends up scraping all 4 ratings (TVDB, TMDB, Trakt, and IMDB) and manages to give the correct data for the episode. However, if I switch the episode ratings to IMDB, it gives the wrong info.

that is no more needed just for ratings we use the official rating exports from IMDB to mix in IMDB ratings where possible (= where an IMDB id is available, because to mix that in we only need the IMDB id of the episode).. in my mind there is no more need for the universal scraper just to get ratings..

nevertheless, I will have a look at the other issues, but if IMDB lists the episodes different than TVDB/Trakt/... no software is able to get the real data

I kind of figured there was no solution to the different episode number issue.

I'm also having issues with Java Heap Space errors when using the Universal TV Scraper. Specifically, shows with a large number of episodes are causing the problem even when I've increased Max Memory to 1024 MB. I switched over from Ember Media Manager, so I'm re-scraping everything and it's causing a lot of problems.

**mlaggner** · 2020-10-01, 09:25

(2020-10-01, 09:24)tr0nllam Wrote: I'm also having issues with Java Heap Space errors when using the Universal TV Scraper. Specifically, shows with a large number of episodes are causing the problem even when I've increased Max Memory to 1024 MB.

I will have a look at this in the evening

tr0nllam · 2020-10-04, 02:02

The latest nightly corrects the issue with the Universal TV Scraper being unable to scrape DVD Season/Episode, but I'm still having some problems.

If I scrape a show with DVD Order using "scrape episode metadata", TMM correctly identifies the episodes, but will not scrape the IMDB info for the episode. However, if I choose to edit the episode manually and hit the scrape button, it works perfectly. Also, sometimes a second scrape with "scrape episode metadata" option causes it to work.

Also, is it a design choice to have Specials episode order listed as DVD Season/Episode rather than Display Season/Episode? As of now, TMM requires you to mark all your specials as DVD Order if you want them to be placed in their proper order within the series.

**mlaggner** · 2020-10-04, 10:18

I've just made some smaller optimizations in the universal TV show scraper (regarding the memory usage)

could you post an exact example according the DVD order? I do not fully understand where the problem should be...

tr0nllam · 2020-10-04, 11:26

(2020-10-04, 10:18)mlaggner Wrote: I've just made some smaller optimizations in the universal TV show scraper (regarding the memory usage)

could you post an exact example according the DVD order? I do not fully understand where the problem should be...

If I scrape Wonderfalls with DVD Order, it will scrape the the TVDB info (title, aired, etc.) but it does not scrape the ratings (trakt) or cast (IMDB). This seems to be universal with all shows using DVD Order. However, once you scrape it (using scrape episode metadata) for a second time, it fills in the missing data for most of the other shows I tried. However, with Wonderfalls, I had to go to edit episode and hit the scrape button for each episode in order for it to retrieve the missing data.

tr0nllam · 2020-10-05, 11:34

Doing more testing has confirmed the issue. Basically, if you set DVD Order, Universal TV Scraper will not scrape IMDB info from the scrape episode metadata option in the menu. However, for some reason, if you go into edit episode and then click the scrape button, the correct IMDB info will populate the NFO still using the Universal TV Scraper. I've tested this with multiple shows and other than a few random episodes that work from the start, most require manually scraping from the edit episode box to get all the data.