[Release] Universal Movie Scraper

  Thread Rating:
  • 7 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
maxxxmagnum Offline
Junior Member
Posts: 2
Joined: Aug 2016
Reputation: 0
Post: #1186
I'm a newbie to Kodi and so far with mixed level of happiness Sad I searched the forums and resolved some beginner's issues, but here's a few I couldn't find the answers to.

My main goal was to backup all of my DVDs (approx. 700 titles) to ISO format. In the past I used a media box that worked great, but had been discontinued and does not support larger internal HDs. KDlinks A300 seemed like a good choice, however...

1. Universal Movie Scraper downloads most of the titles and covers art successfully, however, this applies only to the majority of popular movies. Any older, obscure, rare title... good luck with that. Incorrect title, incorrect or no cover art.

2. TV shows or mini series are a special pain. Example - complete Monty Python's Flying Circus (14 DVD set). Virtually impossible to retrieve correct info from available databases if you rip the DVD set as ISO (each contains several episodes). Naming the DVDs as per some of the suggestions on the forum yielded no better results. Kieslowski's Decalogue (or Dekalog) same problems. I looked them up on thetvdb.com and tried to rename them accordingly but to no avail.

3. Some multi-DVD movies such as Kieslowski's Red, White and Blue consistently downloaded incorrect info for one of the movies (I think it was Blue).

4. The best results I got was entering movie info manually using the Imdb tt00xxxxx code which is specific for a given movie. Needless to say this is extremely tedious when you have a lot of movies. Any changes to the database (purging it), and you have to start all over again for each one of those titles.

The list of problems doesn't stop here, but I have to so I don't irritate the forum too much (LOL). I was wondering if it's possible to customize movie scraper to download from sites such as amazon? I use several Windows based programs for cataloging DVDs and CDs and always had very good results with amazon. Both the titles and cover art are downloaded correctly in overwhelming majority of cases.

For my old media box I used free program called ThumGen to download info and covers. It had extensive list of sites to choose from.

To rename the ripped DVD ISOs I use Filebot, Zeeb and Movie Renamer.

Will be very grateful for any suggestions.
find quote
nickr Online
Team-Kodi Member
Posts: 15,638
Joined: May 2009
Reputation: 320
Location: Christchurch NZ
Post: #1187
Well if obscure movies aren't on TMDB, you should add them.

Also (and I know you won't thank me for this suggestion), but ripping as mkv's using makemkv will preserve quality, while making individual files for each movie or episode, making naming and metadata a lot easier. It can be done from your isos.

If I have helped you or increased your knowledge, click the 'thank user' button to give thanks :) (People with less than 20 posts won't see the "Thank you" button.)
find quote
maxxxmagnum Offline
Junior Member
Posts: 2
Joined: Aug 2016
Reputation: 0
Post: #1188
(2016-08-30 08:38)nickr Wrote:  Well if obscure movies aren't on TMDB, you should add them.

Also (and I know you won't thank me for this suggestion), but ripping as mkv's using makemkv will preserve quality, while making individual files for each movie or episode, making naming and metadata a lot easier. It can be done from your isos.

My scraper is set up to pull movie info from IMDB. Some of the titles are there, but the scraper doesn't see them or incorrectly identifies them. I tried TMDB, but even info contained there is not always correctly downloaded which makes me think there's an intrinsic coding issue with the scraper itself. Kodi had been around for a while, so I'm surprised these scrapers haven't been perfected like some Windows or Mac based programs.

Thank you, I've seen those suggestions for mkv on other threads, but some of us prefer to keep DVDs as ISO files. Especially those with more than one disc, special features etc. Besides, conversion to mkv is much more time consuming, although in the end it does save space. Most of the DVD to ISO takes no more than 15 min. With cost of storage so low nowadays, space is certainly a non-issue.

I still believe that adding ability to pull info from other sources such as amazon would be a better solution if possible.
find quote
jadedcyborg Offline
Junior Member
Posts: 13
Joined: Jun 2013
Reputation: 0
Post: #1189
Scraping with Rotten Tomatoes seems to be broken for a few titles but most works flawlessly. Seems to be an issue with the OMDb API.

EDIT: For example:

Code:
$ curl -s 'http://www.omdbapi.com/?i=tt3110958&r=json&tomatoes=true' | python -m json.tool
{
    "Actors": "Jesse Eisenberg, Mark Ruffalo, Woody Harrelson, Dave Franco",
    "Awards": "3 nominations.",
    "BoxOffice": "N/A",
    "Country": "USA, China, UK, Canada",
    "DVD": "N/A",
    "Director": "Jon M. Chu",
    "Genre": "Action, Adventure, Comedy",
    "Language": "English, Mandarin",
    "Metascore": "46",
    "Plot": "The Four Horsemen resurface and are forcibly recruited by a tech genius to pull off their most impossible heist yet.",
    "Poster": "http://ia.media-imdb.com/images/M/MV5BNzQ0NDgwODQ3NV5BMl5BanBnXkFtZTgwOTYxNjc2ODE@._V1_SX300.jpg",
    "Production": "Lionsgate",
    "Rated": "PG-13",
    "Released": "10 Jun 2016",
    "Response": "True",
    "Runtime": "129 min",
    "Title": "Now You See Me 2",
    "Type": "movie",
    "Website": "http://www.nowyouseeme.movie/",
    "Writer": "Ed Solomon (screenplay), Ed Solomon (story), Peter Chiarelli (story), Boaz Yakin (characters), Edward Ricourt (characters)",
    "Year": "2016",
    "imdbID": "tt3110958",
    "imdbRating": "6.9",
    "imdbVotes": "40,659",
    "tomatoConsensus": "N/A",
    "tomatoFresh": "N/A",
    "tomatoImage": "N/A",
    "tomatoMeter": "N/A",
    "tomatoRating": "N/A",
    "tomatoReviews": "N/A",
    "tomatoRotten": "N/A",
    "tomatoURL": "http://www.rottentomatoes.com/m/now_you_see_me_the_second_act/",
    "tomatoUserMeter": "N/A",
    "tomatoUserRating": "N/A",
    "tomatoUserReviews": "44947"
}
(This post was last modified: 2016-08-30 13:13 by jadedcyborg.)
find quote
nickr Online
Team-Kodi Member
Posts: 15,638
Joined: May 2009
Reputation: 320
Location: Christchurch NZ
Post: #1190
(2016-08-30 10:17)maxxxmagnum Wrote:  
(2016-08-30 08:38)nickr Wrote:  Well if obscure movies aren't on TMDB, you should add them.

Also (and I know you won't thank me for this suggestion), but ripping as mkv's using makemkv will preserve quality, while making individual files for each movie or episode, making naming and metadata a lot easier. It can be done from your isos.

My scraper is set up to pull movie info from IMDB. Some of the titles are there, but the scraper doesn't see them or incorrectly identifies them. I tried TMDB, but even info contained there is not always correctly downloaded which makes me think there's an intrinsic coding issue with the scraper itself. Kodi had been around for a while, so I'm surprised these scrapers haven't been perfected like some Windows or Mac based programs.

Thank you, I've seen those suggestions for mkv on other threads, but some of us prefer to keep DVDs as ISO files. Especially those with more than one disc, special features etc. Besides, conversion to mkv is much more time consuming, although in the end it does save space. Most of the DVD to ISO takes no more than 15 min. With cost of storage so low nowadays, space is certainly a non-issue.

I still believe that adding ability to pull info from other sources such as amazon would be a better solution if possible.
Actually makemkv won't take any longer, but hey in the end it's up to you. For stuff that just won't be recognised try an nfo file with just the IMDb URL in it.

If I have helped you or increased your knowledge, click the 'thank user' button to give thanks :) (People with less than 20 posts won't see the "Thank you" button.)
find quote
mihvel Offline
Junior Member
Posts: 18
Joined: Dec 2011
Reputation: 0
Post: #1191
Seems that scraper does not fetch Movie Country info from IMDB any more. Can it be fixed?
find quote
neurosis13 Offline
Posting Freak
Posts: 904
Joined: Jan 2009
Reputation: 23
Location: Germany
Post: #1192
(2016-09-20 00:38)mihvel Wrote:  Seems that scraper does not fetch Movie Country info from IMDB any more. Can it be fixed?

Same here, hoping for a fix too
find quote
jcea Offline
Junior Member
Posts: 16
Joined: May 2015
Reputation: 0
Post: #1193
Bad preview images from fanart.tv.

Images like http://assets.fanart.tv/fanart/movies/49...pg/preview used to work, but it doesn't anymore. See this IRC chat conversation:

https://chatlogs.metabrainz.org/brainzbo...948&page=1

Specially relevant: https://chatlogs.metabrainz.org/brainzbo...459&page=1

The official documentation: http://docs.fanarttv.apiary.io/#introduc...ge-preview
find quote
Kode Offline
fanart.tv admin
Posts: 763
Joined: Mar 2010
Reputation: 16
Post: #1194
More info on why this has just become a problem here: https://fanart.tv/2016/09/recent-api-issues/

Get and request your ClearLOGOs / ClearART / TV Thumbs / Season Thumbs / Music ClearLOGOs / cdART / Artist Backgrounds / CD Covers from fanart.tv

Severus Server Monitor - Self hosted application to monitor all your servers
find quote
Kode Offline
fanart.tv admin
Posts: 763
Joined: Mar 2010
Reputation: 16
Post: #1195
urls ending in /preview now 301 redirect to the correct url, probably still a good idea to update the urls though in the long run

Get and request your ClearLOGOs / ClearART / TV Thumbs / Season Thumbs / Music ClearLOGOs / cdART / Artist Backgrounds / CD Covers from fanart.tv

Severus Server Monitor - Self hosted application to monitor all your servers
find quote
olympia Offline
Team-Kodi Member
Posts: 2,695
Joined: May 2008
Reputation: 39
Post: #1196
@Kode - this is fixed. On the other hand everyone who scraped his/her library prior to adjusted scraper will need the redirect for sure. Or else they need to re-scrape everything with the new version of the scraper.
find quote
olympia Offline
Team-Kodi Member
Posts: 2,695
Joined: May 2008
Reputation: 39
Post: #1197
(2016-09-20 19:40)neurosis13 Wrote:  
(2016-09-20 00:38)mihvel Wrote:  Seems that scraper does not fetch Movie Country info from IMDB any more. Can it be fixed?

Same here, hoping for a fix too

Should be fixed with IMDb scraper library v2.8.7.
find quote
BugRaider84 Offline
Senior Member
Posts: 217
Joined: Jan 2013
Reputation: 0
Location: Tranquility Base, Moon
Post: #1198
(2016-09-27 23:51)olympia Wrote:  
(2016-09-20 19:40)neurosis13 Wrote:  
(2016-09-20 00:38)mihvel Wrote:  Seems that scraper does not fetch Movie Country info from IMDB any more. Can it be fixed?

Same here, hoping for a fix too

Should be fixed with IMDb scraper library v2.8.7.

Hi olympia,

where do I get v2.8.7 ?
I'm running Universal Movie Scraper 2.8.1 and I'm not getting any IMDb data.
Noticed it today and didn't scrape anything for like 5 days.
Before then everything was fine.

Thanks for your help in advance Smile
find quote
Post Reply