Solved Additional movie ratings scrape
#1
Would be great if TMM could scrape some additional movie ratings and be capable of displaying and exporting those info.

Specifically
- Tomatometer (critics' rating + n of reviews) and Audience score (audience rating + n of users) from Rotten Tomatoes
- Metascore from Metacritic

All these data are available from OMDb API (http://www.omdbapi.com).

As you may know, skin helper script (https://github.com/marcelveldt/script.sk...er.service) already supports this and there are some skins that can show these additional ratings (for instance http://forum.kodi.tv/showthread.php?tid=279036). But the problem is the script stores the data only in temporary cache and thus there's no way to extract or export the data for all movies in the collection.

If TMM can scrape and export these ratings, then the user could even rate movies by her own formula for average score.

There's already such a site that ranks movies by combined score: www.cinesift.com

Image

Thanks for considering this!
#2
Kinda hard right now :|
TMM was mainly designed for Kodi & NFO files.... and there we only could write/save ONE rating per movie.
To scrape multiple, and write only one with a custom formular is currently out-of-scope...
At least for a longer time.
I'm sorry.
tinyMediaManager - THE media manager of your choice :)
Wanna help translate TMM ?
Image
#3
That's too bad.
Then how about offering a choice for the rating provider?
Currently TMM scrapes only "audience score" without votes if you select rotten tomatoes scraper.
So let the user choose among "tomatometer", "audience score", or "metascore" in addition to the currently available IMDb & TMDb rating.
Then the user could export each kind of ratings by multiple scraping from different providers and aggregate those ratings to an external document.
#4
ok, we could add this as scraper configuration.
This will be the easier solution.
I'll add it to our todo-list... Smile

Edit:
Just tried it, but there's only the audience rating or the critics rating available via API Sad
http://developer.rottentomatoes.com/docs...Movie_Info
tinyMediaManager - THE media manager of your choice :)
Wanna help translate TMM ?
Image
#5
Probably there's also query volume limit on the rotten tomatoes API.
That's why I suggested using OMDb API instead, just for rating.

For instance, for The Godfather (1972), it returns all data we need:

http://www.omdbapi.com/?i=tt0068646&plot...atoes=true

- "Metascore":"100"
- "tomatoMeter":"99"
- "tomatoReviews":"84"
- "tomatoUserMeter":"98"
- "tomatoUserReviews":"728130"

Note that I used "IMDb ID" search instead of "title search" to avoid opening another can of worms.

If you add an option of rating provider under scraper settings or scrape metadata window, and search with IMDb ID, then probably it would work flawlessly in most cases.
#6
One more thing. I know Kodi currently supports only one rating.
But can't TMM just add those additional ratings to NFO files as custom tags?
Would it break anything when running Kodi from NFO files with those tags that will be just ignored?

The reason is if you want to aggregate tomatometer, audience score, and metascore in addition to IMDb rating, then you have to scrape several times using the same API.
Well, I can do that if it's only one time task, but if you think about new movies and updating those ratings, that's not something feasible to be done.

So if it's possible, I'd like to see an option of adding all additional ratings to custom tags to store and export.
#7
for scraping OMDB or cinesift, we have to develop an own scraper for that.
It's not that hard, but we're lacking on time....

Furthermore, inside TMM we also have only a single value for rating;
Kinda hard to change that now,... breaking API change, used in all plugins, used by TMM core, rework our "star" rating GUI...
Have to think about it...
tinyMediaManager - THE media manager of your choice :)
Wanna help translate TMM ?
Image
#8
I can see it's not that easy.
As I said, if the single rating with TMM can't be changed, please consider adding them as custom tags, not as rating.
Anyway thanks for replying. Hope to see this feature some day.
#9
Finally done this using Universal Movie Scraper and TMM. It took me several days to scrape, verity, and aggregate the four rating data in an excel sheet.
Here's what I've come to know through the process.

- TMM's Rotten Tomatoes scraper has some serious issues in detecting movies. Almost half of movies can't be identified. It can't use already known IMDb ID or TMDb ID, purely guessing out of filename. Furthermore, RT site seems to use US release date as movie year. Also after around one thousand movies, you keep getting 403 Forbidden error due to their API volume limit. So it's almost unusable.

- Universal Movie Scraper, Kodi addon, too can't use already known IMDb ID for searching. For this reason, scraping was taking hours and I was left with lots of un/mis-identified movies for each rating scrape. Fortunately it exports IMDb ID for each movie, which allows you to compare with your original IMDB rating database exported from TMM. After figuring out what those misidentified movies actually are, I had to rescrape each movie using Kodi. Yes, repeat the same process for each rating scrape to the point that you practically memorize the wrong/right movie title list.

- OMDb API, for some reason, does not return RT ratings for some movies (there's no pattern but especially for many recent movies). It amounted to about 8~9% of my collection, which I had to find ratings manually from the RT site.

Though I've finally got a nice rating comparison chart, keeping this data up to date is unthinkable. For new movies to my collection, probably I will add each rating manually to the spreadsheet.


Again, it would be awesome if TMM could scrape those ratings from OMDb API using IMDB ID. Then it won't take much time to scrape and there will be no misidentified movies. Also updating ratings could be done with just one click.

Regarding the single rating tag issue, I would suggest those additional ratings be written as custom tags something like this:

<metaScore>100</metaScore>
<tomatoMeter>99</tomatoMeter>
<tomatoReviews>84</tomatoReviews>
<tomatoUserMeter>98</tomatoUserMeter>
<tomatoUserReviews>728130</tomatoUserReviews>

Then the user could export all of them at once. If the user wants to use one of them for TMM rating display, renaming those tags to <rating> and <votes> tags (via bulk editing NFO files) would do. Of course, TMM could offer such rating selection function.
#10
tars:

Assume that you are able to scrape all this data. How would you display it in Kodi? what skin would you use?

thanks
#11
(2016-08-06, 14:25)Edworld Wrote: Assume that you are able to scrape all this data. How would you display it in Kodi? what skin would you use?

You don't need to scrape them just to display in Kodi. Skins with skin helper service instantly scrapes those rating for each movie.
Read my first post again which also mentions an example skin.
#12
Looks like OMDb API has recently made some changes. "tomatoes" parameter is no longer supported and without it the API returns all three ratings if available.

So with this query: http://www.omdbapi.com/?i=tt0068646

You get:

{"Title":"The Godfather","Year":"1972","Rated":"R","Released":"24 Mar 1972","Runtime":"175 min","Genre":"Crime, Drama","Director":"Francis Ford Coppola","Writer":"Mario Puzo (screenplay), Francis Ford Coppola (screenplay), Mario Puzo (novel)","Actors":"Marlon Brando, Al Pacino, James Caan, Richard S. Castellano","Plot":"The aging patriarch of an organized crime dynasty transfers control of his clandestine empire to his reluctant son.","Language":"English, Italian, Latin","Country":"USA","Awards":"Won 3 Oscars. Another 23 wins & 27 nominations.","Poster":"https://images-na.ssl-images-amazon.com/images/M/MV5BZTRmNjQ1ZDYtNDgzMy00OGE0LWE4[email protected]._V1_SX300.jpg","Ratings":[{"Source":"Internet Movie Database","Value":"9.2/10"},{"Source":"Rotten Tomatoes","Value":"99%"},{"Source":"Metacritic","Value":"100/100"}],"Metascore":"100","imdbRating":"9.2","imdbVotes":"1,219,320","imdbID":"tt0068646","Type":"movie","DVD":"09 Oct 2001","BoxOffice":"N/A","Production":"Paramount Pictures","Website":"http://www.thegodfather.com","Response":"True"}

Again it would be great if tmm can scrape Rotten Tomatoes and Metacritic ratings in addition to IMDb ratings. IMDb ratings (and its top 250 chart) increasingly becomes a joke, not to mention its own flaws due to its crowd-sourced nature, especially with the influx of super-overrated Indian and Turkish movies. It alone cannot offer much meaningful information any more. That's why more and more movie streaming (or selling) sites are trying to provide ratings from multiple sources.

Hope you guys consider adding this feature to the upcoming v3.
#13
yes, we consider this change - but that will be the last greater change for v3 since that breaks almost every interface (and we do not want to change scraper interfaces too early, since there won't be any chance to fix problems in this stage)
tinyMediaManager - THE media manager of your choice - available for Windows, macOS and Linux
Help us translate tinyMediaManager at Weblate | Translations at 66%
Found a bug or want to submit a feature request? Contact us at GitLab
#14
Wonderful!
Yep, I can wait. Thanks a lot!
#15
It took more than four years but I am really grateful to the devs for finally making this wish fulfilled!

Image



Logout Mark Read Team Forum Stats Members Help
Additional movie ratings scrape0
This forum uses Lukasz Tkacz MyBB addons.