• 1
  • 12
  • 13
  • 14(current)
  • 15
  • 16
  • 42
Release TheMovieDB - TV Show scraper (XML)
The config file confused me for the opposite reason in this log, I could see instances where it was cached, since it's loaded for both GetCast and GetEpisodeArt.  But they're both within one episode.

It's like each episode is treated as a new scan, starting fresh (except that the episodeguide must persist throughout).  That's why I suggest adding a cachepersistence term.
Reply
(2018-11-22, 11:40)scudlee Wrote: If you look in the addon.xml for metadata.tvdb.com, you'll see a part that says  cachepersistence="00:15".  Try adding that to the equivalent spot in the addon.xml for metadata.tvshows.themoviedb.org.  (You will need to stop and restart Kodi.) 
@scudlee thanks a lot for this further suggestion! Now everything work smoothly and very fast even with parsing nfo files (X-RateLimit-Remaining didn't go lower than 27)!

Are you going to udpate both tmdb.xml and addon.xml so that they will be in the next Kodi 18 nightly? I would even suggest to change the version number of the addon to 3.1.0.

Thanks again you all for the support!
Light IMDb Ratings Update - Keep updated the IMDb ratings for your movies and TV shows.
In case you found useful my work, feel free to offer me a cappuccino!
Reply
@olympia

As this has now become the default TV Show scraper, I have been using it a big more and comment as follows...

1. I have noticed that when scraping episodes, only one episode thumb link is scraped. Is it possible to scrape all available thumbs for the episode?
My reason...
TheMovieDB allows multiple episode images at different resolutions, unlike TVDB that allows only one image. If you are unfortunate enough to scrape the 400x300 image added by their TVDB-Bot, then it is impossible to change the thumb to a better image within the Choose Art dialogue.

2. No IMDB ID's are scraped. Could they be scraped, along with IMDB ratings? Similar to what TVDB does? The ID's are available for each episode and tv show.

Thanks
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
The multiple episode thumbs were sacrificed as a result of the previous issues (they did used to be scraped), since they're not present in the season lists, only in the individual episode pages.  Including artwork at the season level only supplies the season art.
Similarly for the episode external ids, including ids at the season level only pulls external season ids, not for each episode.

Not sure of ways around that that don't negate the benefits of the fewer API calls.  Outside of proper rate-limiting, that is.
Reply
Oh, right. Anything at the episode level becomes problematic.

Is one option to enable "enhanced episode scraping" (which would include additional episode artwork, IMDB Id's and IMDB Ratings) in the scraper possible? This will then apply rate limiting and a slower scrape for those that prefer the additional info?
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
@scudlee, just to know if you had the occasion to commit the changes to addon.xml and tmdb.xml on github...

Have a great weekend!
Light IMDb Ratings Update - Keep updated the IMDb ratings for your movies and TV shows.
In case you found useful my work, feel free to offer me a cappuccino!
Reply
A few more tricks to reducing the API calls in this thread if interested: Specifically
Quote:There's lots of ways to do multiple requests in one by using 
append_to_response
.
There's some basics like so:
https://api.themoviedb.org/3/tv/1402?api...ges,videos https://api.themoviedb.org/3/tv/1402/sea...ges,videos
But if you know that a season exists, you can even do this (get season 1 and 2 in a single call):
https://api.themoviedb.org/3/tv/1402?api...1,season/2
So it kind of all just depends on the specific use case and need.

https://www.themoviedb.org/talk/5bf54610...4f6e0980fb
Reply
@docwra Sadly that doesn't really help for a couple of reasons.  I mean, maybe it could help, but it would take some work.

The first issue, as Travis hints at, is you need to know the seasons exist before you can append them, so you would need to fetch the unappended file first to get the seasons to then get the appended version with all the seasons.
But that's fine.  That would just take the number of calls from the number of seasons to a flat 2.  Not bad.

...Except, there's a limit of 20 appends allowed, so anything over 20 seasons would just fail.  There would need to be a mechanism to split the seasons up, 20 into this file, the next 20 into another. 
Doable, but might need xslt to handle the maths.  The number of calls would go up by 1 for every multiple of 20 or so.

Actually, it might be worse than that, since the season images would need getting too, so it would likely be split every 10 seasons (2 appends per season).

If it was unlimited appends, then it would definitely make sense, but the added complexity of having to split things up might not be worth it, especially when not even @axlt2002 is getting close to the rate-limit any more.

@axlt2002 The changes were pushed a few days ago, the addon should have updated automatically by now.
Reply
(2018-11-25, 15:29)scudlee Wrote: @axlt2002 The changes were pushed a few days ago, the addon should have updated automatically by now.
@scudlee thanks a lot for your feedback. I have just downloaded the latest nightly of Kodi 18 but the scraper is still at its old state...
Light IMDb Ratings Update - Keep updated the IMDb ratings for your movies and TV shows.
In case you found useful my work, feel free to offer me a cappuccino!
Reply
(2018-11-25, 22:25)axlt2002 Wrote: @scudlee thanks a lot for your feedback. I have just downloaded the latest nightly of Kodi 18 but the scraper is still at its old state... 

Hah... Yeah... About that... Link (My bad.)

Still, the version that comes with a nightly build will likely be 3.0.6 until someone (probably Martijn) manually syncs it to the one in the repo.
But it should automatically update to the newest version.
Reply
@scudlee

I can't seem to figure out why I am getting no Actors information. I have tried v3.0.6 and v3.1.0. Something changed at TMDB?

This is the show... https://www.themoviedb.org/tv/60743-cons...uage=en-US

The log line 1004... https://paste.kodi.tv/iyaputoniv.kodi
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
@Karellen
Not sure if it's a change at themoviedb or some random glitch, but for some reason the Constantine credits don't include the "gender" field, which the other couple of shows I checked still did have, and in fact the episode guest stars for Constantine do have it as well.
When I scraped with 3.1.0 I did get the episode credits on Constantine, although your log shows 3.0.6 failing on both counts, which might have been why I changed it in the first place for 3.1.0 (I made that change a while ago, so I forgot about it).

If it's just a random glitch, I don't feel any urgency, so here's a quick pastebin fix that makes the gender field optional:
https://pastebin.com/knUeM7Vw

However this also includes a little something extra (just for you Wink). 
I took your request for the extra episode thumbnails, and combined it with my random musings on the append_to_response to fetch the episode art in batches of about 10 episodes each per season (it groups them on the tens digit, so 1-9, 10-19, etc.)  I was able to do it that way without using xslt.  It also means I could add the external ids on as well and still be under the 20 append limit (I haven't done that yet, though).

Obviously that's going to add a few more requests per scan.  But a full scan would also spend 9(ish) episodes using the cache before the next request, so hopefully it should keep the rate-limit at bay. 
I managed a full scan of One Piece (19 seasons, 838 episodes) with 127 cached files, and never hit the limit.
Reply
Hi @scudlee

Thank you for that fix Smile

I tested it and confirm the Actors are now downloaded and there are multiple remote images in the Choose Art screen.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
Folks:

Noob Alert.

I have used XBMC during the 2009-2011 on a PC and then I went overseas.  When we were moving back, I was confronted with boxes and boxes of shiny movies we had collected to be moved and double that veriety of Western and Indian Music CD/VCD/DVD.  It rekindled my search to put everything on a box, this time on a nas connected to a Fire TV or perhaps a cheaper Android Box or something running libreelec.  

I know that Kodi will scrape movie and store the artwork, plots, posters (NFO and JPG files) in its own secret userdata location. But earlier I used to store this in individual movie folders that stored the ISO or MKV file (Had used EmberMedia then).

I need a clarification:

Kodi will first check the individual movie folder for the NFO and art file and will inhale this (into userdata) from these folders before it goes scouting around the internet to scrape for the movie. So if I keep my catelog updated with these art works, then rebuilding images will be so much easier for Kodi.

Am I correct?  Please enlighten.

Many thanks.
Reply
(2018-11-28, 20:28)agarg Wrote: Am I correct?  Please enlighten.
Yes you are correct.

Kodi will always search for local info first. When it does not find it, then the scraper gets invoked to go online.

As long as you correctly name and save the artwork and nfo files, kodi will find them.

See links in My Signature below. Some might be useful to you.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
  • 1
  • 12
  • 13
  • 14(current)
  • 15
  • 16
  • 42

Logout Mark Read Team Forum Stats Members Help
TheMovieDB - TV Show scraper (XML)2