Kodi Community Forum

Full Version: Ember Media Manager 1.4.8.0 ALPHA - Discussion Thread
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Spend TMDB some money to get better servers ;-)

Gesendet von meinem Nexus 5 mit Tapatalk
(2017-04-07, 19:11)DanCooper Wrote: [ -> ]
(2017-04-07, 18:04)axlt2002 Wrote: [ -> ]Hi Dan,

It is me again...Angel

I'm actually putting my hands on the Rate Update add-on in order to extend it to update also the TV Show Episodes rating and votes (actually, in fact, rating and votes are updated only at the TV Show level). The add-on uses IMDd site as source and thereore the knowledge of the IMDd id of the Movie/TV Show/Episode (e.g. tt3429842) is fundamental to have an efficient management, especially in case of huge collections. You would actually asking yourself what can EMM do in relation to this...well, as a starting point, EMM stores the IMDb id for all the mentioned contents; the point is that it uses "proprietary" tags.

The format used by Kodi to save the ids for a Movie is completely aligned with the one used by EMM:

Kodi format
Code:
<id>tt2245084</id>
<uniqueid type="unknown" default="true">tt2245084</uniqueid>

EMM format
Code:
<id>tt2245084</id>
<tmdb>12144</tmdb>

Passing to the TV Show and Episode the way how the ids are saved changes; while Kodi uses the same format of a Movie (id and uniqueid), EMM does as reported hereafter:

EMM format for TV Show
Code:
<id>305288</id>
<imdb>tt4574334</imdb>
<tmdb>66732</tmdb>

EMM format for Episode
Code:
<uniqueid>5468124</uniqueid>
<imdb>tt4593118</imdb>
<tmdb>1198665</tmdb>

Would it be possible to have implemented the same approach used for a Movie also for TV Show and Episode (i.e. use <id> to save the IMDb id and <tmdb> to save the TMDB id; the latter will be anyway ignored by Kodi)?

This request is because the add-on uses the field <id> retrieved from the Kodi Movie/TV Show libraries. At the first run, since the IMDb id is practically not available (being stored in the field <imdb> that Kodi does not store), it performs an exhaustive search on the IMDb site based on the TV Show title, taking some time, providing a window to select the correct TV Show and, finally, store in a local database the IMDb id paired with the <id>. You can imagine what this would be in case of Episodes...

I really hope you can make such modifications. I don't think it will imply some issue on EMM side. Please send me a PM in case you would like to share with me a test .exe of EMM so that I will be able to progress on the add-on code.

Looking forward for your collaboration.

Hi

I think that would not work proper in this way. There are some points that have to be changed for a perfect solution:

Kodi always use the ID of the selected scraper (depending on source settings) as <id>. In case of tv shows in most cases that whould be the TVDB ID, not the IMDB ID. For movies it is the IMDB ID, even if you use the TMDB scraper. I think that has been manually set in scraper since IMDB is the biggest and most popular source for movies.

Since Kodi 17 it's possible to save more than one ID for each movie/tv show/episode. There is a table named uniqueid that contains all known IDs. ATM Kodi save only one ID per media, also without any specification of the ID source (type it's always unknown). I'm already working on changes on Embers DB to save all the IDs as separate nodes in NFO like this:
Code:
<id>121361</id>
<uniqueid type="imdb">tt0944947</uniqueid>
<uniqueid type="tmdb">1399</uniqueid>
<uniqueid type="tvdb" default="true">121361</uniqueid>

I don't know why Kodi does not save the type even the Kodi scrapers has been used... I think that should be fixed.
Also there is no field in Kodis DB to set the field default. I've no idea how that will be worked.

Your addon should be use this table to check if there is an ID with type=imdb. If no ID has been found, run a search by title.


But the bigger and most crappy problem is that:
"SpongeBob SquarePants" as example has different episode numbering on TVDB/TMDB and IMDB:
TVDB & TMDB:
S01E01 Help Wanted 1999-05-01
S01E02 Reef Blower 1999-05-01
S01E03 Tea at the Treedome 1999-05-01

IMDB:
S01E01 Help Wanted / Reef Blower / Tea at the Treedome 1999-05-01

So there is no way to get a proper rating or other information from IMDB. For this reason TVDB has not saved the IMDB IDs for this epsiodes.
We have the same issue in Ember: if no IMDB is known, the IMDB scraper search be tv shows IMDB ID and search the episode by episode number on the IMDB website... with the result, that the result will no be mapped correctly and of this case will be wrong.


To summarize everything:
- I will use the new <uniqueid> node to save all known IDs
- for movies the <id> node will be the IMDB ID
- for tv shows the <id> node will be the TVDB ID as default

Thanks Dan for your usual availabilty and really clear way to explain the things. I understand now the rationale behind the actual behaviour of the add-on.

The only thing that is not completely celar to me is when you say "Your addon should be use this table to check if there is an ID with type=imdb"...are all the ids saved in the Kodi Library in Kodi 17? If I use the JSON command VideoLibrary.GetTVShows, Am I able to retrieve all of them or still only the "default" id? In fact the point is that the add-on checks the Kodi library and not the .nfo files.

More in general, I think that if a user makes use of EMM, he usually "combines" in some way the information coming from the different sites. In addition to that, he most probably (at least 99%) will not use an internal Kodi scraper afterwards...so that the value stored in <id> is quite not significant from a Kodi perspective. Therefore it should be available the possiblity to decide which id to use in <id> between the ones available. Do you think it is something that you can add as an option?

About the Episodes, the case you reported is without solution in fact; and we can not do anything for that. But having in general the possibility to use the IMDd id as the <id>, it will be really an advantage to search for them and retreive the interested data (please note that here we are talking about a specific task such as update the rating and votes only that is something that TVDB does not provide in a consistent number). The point here is also that if I have the Episode title in italian, is impossible to search for it on IMDb...the IMDB id is the solution...but if I can not retrieve it from the Kodi library, I'm just stuck! So that the question I asked in the previous paragraph...

Have a nice evening.
(2017-04-07, 23:02)axlt2002 Wrote: [ -> ]
(2017-04-07, 19:11)DanCooper Wrote: [ -> ]
(2017-04-07, 18:04)axlt2002 Wrote: [ -> ]Hi Dan,

It is me again...Angel

I'm actually putting my hands on the Rate Update add-on in order to extend it to update also the TV Show Episodes rating and votes (actually, in fact, rating and votes are updated only at the TV Show level). The add-on uses IMDd site as source and thereore the knowledge of the IMDd id of the Movie/TV Show/Episode (e.g. tt3429842) is fundamental to have an efficient management, especially in case of huge collections. You would actually asking yourself what can EMM do in relation to this...well, as a starting point, EMM stores the IMDb id for all the mentioned contents; the point is that it uses "proprietary" tags.

The format used by Kodi to save the ids for a Movie is completely aligned with the one used by EMM:

Kodi format
Code:
<id>tt2245084</id>
<uniqueid type="unknown" default="true">tt2245084</uniqueid>

EMM format
Code:
<id>tt2245084</id>
<tmdb>12144</tmdb>

Passing to the TV Show and Episode the way how the ids are saved changes; while Kodi uses the same format of a Movie (id and uniqueid), EMM does as reported hereafter:

EMM format for TV Show
Code:
<id>305288</id>
<imdb>tt4574334</imdb>
<tmdb>66732</tmdb>

EMM format for Episode
Code:
<uniqueid>5468124</uniqueid>
<imdb>tt4593118</imdb>
<tmdb>1198665</tmdb>

Would it be possible to have implemented the same approach used for a Movie also for TV Show and Episode (i.e. use <id> to save the IMDb id and <tmdb> to save the TMDB id; the latter will be anyway ignored by Kodi)?

This request is because the add-on uses the field <id> retrieved from the Kodi Movie/TV Show libraries. At the first run, since the IMDb id is practically not available (being stored in the field <imdb> that Kodi does not store), it performs an exhaustive search on the IMDb site based on the TV Show title, taking some time, providing a window to select the correct TV Show and, finally, store in a local database the IMDb id paired with the <id>. You can imagine what this would be in case of Episodes...

I really hope you can make such modifications. I don't think it will imply some issue on EMM side. Please send me a PM in case you would like to share with me a test .exe of EMM so that I will be able to progress on the add-on code.

Looking forward for your collaboration.

Hi

I think that would not work proper in this way. There are some points that have to be changed for a perfect solution:

Kodi always use the ID of the selected scraper (depending on source settings) as <id>. In case of tv shows in most cases that whould be the TVDB ID, not the IMDB ID. For movies it is the IMDB ID, even if you use the TMDB scraper. I think that has been manually set in scraper since IMDB is the biggest and most popular source for movies.

Since Kodi 17 it's possible to save more than one ID for each movie/tv show/episode. There is a table named uniqueid that contains all known IDs. ATM Kodi save only one ID per media, also without any specification of the ID source (type it's always unknown). I'm already working on changes on Embers DB to save all the IDs as separate nodes in NFO like this:
Code:
<id>121361</id>
<uniqueid type="imdb">tt0944947</uniqueid>
<uniqueid type="tmdb">1399</uniqueid>
<uniqueid type="tvdb" default="true">121361</uniqueid>

I don't know why Kodi does not save the type even the Kodi scrapers has been used... I think that should be fixed.
Also there is no field in Kodis DB to set the field default. I've no idea how that will be worked.

Your addon should be use this table to check if there is an ID with type=imdb. If no ID has been found, run a search by title.


But the bigger and most crappy problem is that:
"SpongeBob SquarePants" as example has different episode numbering on TVDB/TMDB and IMDB:
TVDB & TMDB:
S01E01 Help Wanted 1999-05-01
S01E02 Reef Blower 1999-05-01
S01E03 Tea at the Treedome 1999-05-01

IMDB:
S01E01 Help Wanted / Reef Blower / Tea at the Treedome 1999-05-01

So there is no way to get a proper rating or other information from IMDB. For this reason TVDB has not saved the IMDB IDs for this epsiodes.
We have the same issue in Ember: if no IMDB is known, the IMDB scraper search be tv shows IMDB ID and search the episode by episode number on the IMDB website... with the result, that the result will no be mapped correctly and of this case will be wrong.


To summarize everything:
- I will use the new <uniqueid> node to save all known IDs
- for movies the <id> node will be the IMDB ID
- for tv shows the <id> node will be the TVDB ID as default

Thanks Dan for your usual availabilty and really clear way to explain the things. I understand now the rationale behind the actual behaviour of the add-on.

The only thing that is not completely celar to me is when you say "Your addon should be use this table to check if there is an ID with type=imdb"...are all the ids saved in the Kodi Library in Kodi 17? If I use the JSON command VideoLibrary.GetTVShows, Am I able to retrieve all of them or still only the "default" id? In fact the point is that the add-on checks the Kodi library and not the .nfo files.

More in general, I think that if a user makes use of EMM, he usually "combines" in some way the information coming from the different sites. In addition to that, he most probably (at least 99%) will not use an internal Kodi scraper afterwards...so that the value stored in <id> is quite not significant from a Kodi perspective. Therefore it should be available the possiblity to decide which id to use in <id> between the ones available. Do you think it is something that you can add as an option?

About the Episodes, the case you reported is without solution in fact; and we can not do anything for that. But having in general the possibility to use the IMDd id as the <id>, it will be really an advantage to search for them and retreive the interested data (please note that here we are talking about a specific task such as update the rating and votes only that is something that TVDB does not provide in a consistent number). The point here is also that if I have the Episode title in italian, is impossible to search for it on IMDb...the IMDB id is the solution...but if I can not retrieve it from the Kodi library, I'm just stuck! So that the question I asked in the previous paragraph...

Have a nice evening.

Yes, the uniqueid table is a part of the Kodi DB. I've tested it with a new NFO that contains multiple <uniqueid> nodes:

Call:
Code:
http://localhost:8080/jsonrpc?request={"jsonrpc":"2.0","id":"1","method":"VideoLibrary.GetTVShowDetails", "params": {"properties": ["uniqueid"], "tvshowid":10}}

Result:
Code:
{"id":"1","jsonrpc":"2.0","result":{"tvshowdetails":{"label":"Game of Thrones","tvshowid":10,"uniqueid":{"imdb":"tt0944947","unknown":"121361"}}}}

As you can see, the API tells you all saved IDs.

EDIT: tested with Kodi 17.1.
(2017-04-07, 17:12)DanCooper Wrote: [ -> ]
(2017-04-07, 14:12)Bjur Wrote: [ -> ]The interesting thing is also, when I check task manager the memory runs to 100% 2 gb ember or something then goes back again and up again until the program starts.

How much memory is used just after start?

I have tried to start it up. Around 2.3 gb it restart the memory in taske manager and when it is started around 120 mb of memory
Hi,

Today I did my very first attempt at syncing Kodi from EMM.
I have an error but I think I know why and I'd like just to confirm it.
In EMM, I have 4 separate sources defined as follow:
Z:\TV Shows\French\
Z:\TV Shows\English\
O:\TV Shows\
P:\TV Shows\


In Kodi, it's different, I have defined 1 TV Shows source, with 4 media locations:
SMB://...
SMB://...
SMB://...
SMB://...
(each location goes to one of my TV show directory)
that single TV Show source is setup to use local media.

In EMM, I changed the poster of 1 TV show and tried to sync it.
it failed, with log file saying:
GetRemotePath: ""Z:\TV Shows\English\War and Peace\"" | Source not mapped!".

It seams that the different sources set up in EMM must exactly match the sources set up in Kodi. I can't have 1 source that points to multiple SMB paths.

Is that the reason?

thanks
(2017-04-09, 00:39)hatbrox Wrote: [ -> ]Hi,

Today I did my very first attempt at syncing Kodi from EMM.
I have an error but I think I know why and I'd like just to confirm it.
In EMM, I have 4 separate sources defined as follow:
Z:\TV Shows\French\
Z:\TV Shows\English\
O:\TV Shows\
P:\TV Shows\


In Kodi, it's different, I have defined 1 TV Shows source, with 4 media locations:
SMB://...
SMB://...
SMB://...
SMB://...
(each location goes to one of my TV show directory)
that single TV Show source is setup to use local media.

In EMM, I changed the poster of 1 TV show and tried to sync it.
it failed, with log file saying:
GetRemotePath: ""Z:\TV Shows\English\War and Peace\"" | Source not mapped!".

It seams that the different sources set up in EMM must exactly match the sources set up in Kodi. I can't have 1 source that points to multiple SMB paths.

Is that the reason?

thanks

I think the problem is that you not have mapped the source is KI Host settings:

Image
(2017-04-07, 01:56)mhendu Wrote: [ -> ]Thought I'd posted this the other day but now I can't find the post. On quite a few movies in my library when scraping (using IMDB, with the latest Alpha of Ember dated 3/7/2017) I get an error as follows:

constraint failed
UNIQUE constraint failed: actorlinkmovie.idActor, actorlinkmovie.idMovie

Any help would be appreciated.

Haven't seen any response so just wanted to make sure this was noted - I could try maybe rebuilding my database and see if that helps, but if there are any other suggestions that would be helpful.
(2017-04-08, 16:16)Bjur Wrote: [ -> ]
(2017-04-07, 17:12)DanCooper Wrote: [ -> ]
(2017-04-07, 14:12)Bjur Wrote: [ -> ]The interesting thing is also, when I check task manager the memory runs to 100% 2 gb ember or something then goes back again and up again until the program starts.

How much memory is used just after start?

I have tried to start it up. Around 2.3 gb it restart the memory in taske manager and when it is started around 120 mb of memory

Can you help getting it fixed?

I think the problem is that you not have mapped the source is KI Host settings:


[/quote]

Turned out I have some issues in Kodi and its sources. Kodi refuses to load certain sources for some reason.
but the update from EMM work for other videos.
(2017-04-08, 13:48)DanCooper Wrote: [ -> ]Yes, the uniqueid table is a part of the Kodi DB. I've tested it with a new NFO that contains multiple <uniqueid> nodes:

Call:
Code:
http://localhost:8080/jsonrpc?request={"jsonrpc":"2.0","id":"1","method":"VideoLibrary.GetTVShowDetails", "params": {"properties": ["uniqueid"], "tvshowid":10}}

Result:
Code:
{"id":"1","jsonrpc":"2.0","result":{"tvshowdetails":{"label":"Game of Thrones","tvshowid":10,"uniqueid":{"imdb":"tt0944947","unknown":"121361"}}}}

As you can see, the API tells you all saved IDs.

EDIT: tested with Kodi 17.1.

Thanks Dan! This is really a good new! I'm going to change the add-on on such basis then...

One more question: if I have all my collection using the "old" .nfo format without the <uniqueid> nodes, is EMM going to overwrite automatically them with the new format or what I would be supposed to do in order to update them? Do you have any test .exe that you can share?

Have a great start of the week!
(2017-04-10, 12:01)axlt2002 Wrote: [ -> ]
(2017-04-08, 13:48)DanCooper Wrote: [ -> ]Yes, the uniqueid table is a part of the Kodi DB. I've tested it with a new NFO that contains multiple <uniqueid> nodes:

Call:
Code:
http://localhost:8080/jsonrpc?request={"jsonrpc":"2.0","id":"1","method":"VideoLibrary.GetTVShowDetails", "params": {"properties": ["uniqueid"], "tvshowid":10}}

Result:
Code:
{"id":"1","jsonrpc":"2.0","result":{"tvshowdetails":{"label":"Game of Thrones","tvshowid":10,"uniqueid":{"imdb":"tt0944947","unknown":"121361"}}}}

As you can see, the API tells you all saved IDs.

EDIT: tested with Kodi 17.1.

Thanks Dan! This is really a good new! I'm going to change the add-on on such basis then...

One more question: if I have all my collection using the "old" .nfo format without the <uniqueid> nodes, is EMM going to overwrite automatically them with the new format or what I would be supposed to do in order to update them? Do you have any test .exe that you can share?

Have a great start of the week!

You have to rewrite the NFO with Tools => Rewrite all movie content.
EDIT: I also added the same function for tv shows and movie sets.

No sorry, atm I don't have a working version of the next Alpha.
Do you have any suggestion to what I can try?
(2017-04-10, 13:10)DanCooper Wrote: [ -> ]
(2017-04-10, 12:01)axlt2002 Wrote: [ -> ]
(2017-04-08, 13:48)DanCooper Wrote: [ -> ]Yes, the uniqueid table is a part of the Kodi DB. I've tested it with a new NFO that contains multiple <uniqueid> nodes:

Call:
Code:
http://localhost:8080/jsonrpc?request={"jsonrpc":"2.0","id":"1","method":"VideoLibrary.GetTVShowDetails", "params": {"properties": ["uniqueid"], "tvshowid":10}}

Result:
Code:
{"id":"1","jsonrpc":"2.0","result":{"tvshowdetails":{"label":"Game of Thrones","tvshowid":10,"uniqueid":{"imdb":"tt0944947","unknown":"121361"}}}}

As you can see, the API tells you all saved IDs.

EDIT: tested with Kodi 17.1.

Thanks Dan! This is really a good new! I'm going to change the add-on on such basis then...

One more question: if I have all my collection using the "old" .nfo format without the <uniqueid> nodes, is EMM going to overwrite automatically them with the new format or what I would be supposed to do in order to update them? Do you have any test .exe that you can share?

Have a great start of the week!

You have to rewrite the NFO with Tools => Rewrite all movie content.
EDIT: I also added the same function for tv shows and movie sets.

No sorry, atm I don't have a working version of the next Alpha.

Thanks a lot Dan, I will make some trials by manually editing the .nfo files. And will wait for the next alpha with a lot of passion... Wink

When you write "TV Shows" you mean also the single Episode .nfo files, right?

One question for clarification about the new format

Code:
<id>121361</id>
<uniqueid type="imdb">tt0944947</uniqueid>
<uniqueid type="tmdb">1399</uniqueid>
<uniqueid type="tvdb" default="true">121361</uniqueid>

What does the "default" property in the last line mean? Why only in the last line and what happens if it is set to "false"?

Another point is related to ratings and votes; Kodi exports them in the following format that is not recognized by EMM:

Code:
<ratings>
     <rating name="default" max="10" default="true">
           <value>9.000000</value>
           <votes>262868</votes>
     </rating>
</ratings>

Are you going to adapt it in EMM?
(2017-04-10, 14:20)axlt2002 Wrote: [ -> ]When you write "TV Shows" you mean also the single Episode .nfo files, right?
Yes, and also movie NFOs.

(2017-04-10, 14:20)axlt2002 Wrote: [ -> ]One question for clarification about the new format

Code:
<id>121361</id>
<uniqueid type="imdb">tt0944947</uniqueid>
<uniqueid type="tmdb">1399</uniqueid>
<uniqueid type="tvdb" default="true">121361</uniqueid>

What does the "default" property in the last line mean? Why only in the last line and what happens if it is set to "false"?
I've no idea. I don't know the default value for the "default" field (true or false). Maybe the XML parser in Kodi adds the "default=true" id on the first position in DB. Maybe the <id> has the hogher priority while adding the DB, maybe the <id> will be ignored if <uniqueid> is available. I think only a Kodi dev or a look into the source code can answer your question...

(2017-04-10, 14:20)axlt2002 Wrote: [ -> ]Another point is related to ratings and votes; Kodi exports them in the following format that is not recognized by EMM:

Code:
<ratings>
     <rating name="default" max="10" default="true">
           <value>9.000000</value>
           <votes>262868</votes>
     </rating>
</ratings>

Are you going to adapt it in EMM?
Yes, I will add that also to the next version. But atm I don't have run any tests with this.
(2017-04-10, 09:28)Bjur Wrote: [ -> ]
(2017-04-08, 16:16)Bjur Wrote: [ -> ]
(2017-04-07, 17:12)DanCooper Wrote: [ -> ]How much memory is used just after start?

I have tried to start it up. Around 2.3 gb it restart the memory in taske manager and when it is started around 120 mb of memory

Can you help getting it fixed?

Maybe we still have some memory leaks. Try to split your scraping processes in smaller steps (maybe 100 movies each time?).