2019-03-26, 11:14
Hi, I've installed integral scraper versions 0.0.2 and am seeing an SQL ERROR.
I attach the kodi log here
I attach the kodi log here
2019-03-26, 11:44
(2019-03-26, 11:14)eljejer Wrote: Hi, I've installed integral scraper versions 0.0.2 and am seeing an SQL ERROR.@eljejer plenty of errors in that log still, some NFS access related. The db error is about artist "Gilbert O'Sullivan", and I suspect that some of your music by him is not tagged with Musicbrainz IDs and that the naming is not consistent. Hence you possibly have two entries for him in your library, maybe one with a different unicode char between "O" and"S". One entry has mbid, the other the scraper just fetched mbid based on name match but triying to save it fails because it alreday has an entry with that mbid.
Non-ascii chars in names can be such a pain. Check all your music files and either tag with mbids or take great care that naming is consistent especially those with "`" or "-" in them.
2019-03-26, 13:20
Now I'm seeing some python errors in the log, which I attach here
And some more SQL errors as well
And some more SQL errors as well
2019-03-26, 13:23
Thanks, I will look into the non ASCII character matter on my part
2019-04-14, 10:47
@ronie I've done some album scraping with v002 and got a crop of script errors for you to check. Log is a monster 15MB so zipped here, 1435 albums out of 1542 scraped successfully taking 2h 15m.
Yet to examine the successes more carefully to validate data etc. The SQL errors I understand, down to duplicate albums in my test data one with mbid one without (so seen by Kodi as different). Scraper fetches mbid and so we get an error on trying to save it. I'll give what best to do in that case some thought, but it is a core issue not an addon one.
Edit:
Multiple Scraper runs needed
Re-scraping twice more, which just retries to scrape those that have not been successful before, took the unscraped albums down from 107 to 64. Not sure why additional attempts were needed, there were no errors or timeouts in the log for these albums and some already had mbids.
Also I think that the progress bar vanished during the initial scraping causing me to think it had finished when it had not. This could be related perhaps? Either way I have been unable to repeat the progress bar issue, and see nothing in the log. All I can say is odd and something to watch for.
Look up by name identification issues for collaboration albums
Of the remaining 64 albums some of these are expected because they are not in the Musicbrainz or Discogs databases, and some cause the db error that I will fix.
What is causing look up by name identification issues is albums with multiple album artists. Many of my examples of collaborations are classical music, which I generally tag with mbids anyway, but this does effect other genres just to a lesser extent. We could just shrug and tell users to ensure collaborations have mbids, or maybe we could try to pass and use the artist names differently?
An example from my test data is "Neck and Neck" by "Chet Atkins / Mark Knopfler" 3f831051-75f4-3085-8d0f-8b94adbb3c5e
The scraper is passed the artist display text e.g. "Chet Atkins / Mark Knopfler", but requests like
do not get a result from Musicbrainz.
However
will find the album sucessfully.
Note that it as having " / " in the display name rather than " & " that mattered in this case, in others it could be use of " and " etc. Getting a matching combined names is going to be tricky. Perhaps Musicbrainz have some advise for queries using multile artist names?
An example from classical music is
"An Irish Symphony / A Comedy Overture" by "Hamilton Harty, Ulster Orchestra, Bryden Thomson" 4c2c013d-d901-4690-97d3-cc99a770b469
As the only release in the db of those works combined it can be found using any one of the three album artists, or a with a precisely punctated combination of all three
"Hamilton Harty" - yes
"Bryden Thomson" - yes
"Ulster Orchestra" - yes
"Hamilton Harty; Ulster Orchestra, Bryden Thomson" - yes
"Hamilton Harty, Ulster Orchestra, Bryden Thomson" - no
"Hamilton Harty / Ulster Orchestra / Bryden Thomson" - no
"Hamilton Harty; Bryden Thomson" - no
Not sure if we can easily pass an array of names to the scraper, or if we just pass a string with separators that the scraper can split. Artist display name itself can not be reliably split, so best let the core tell the scraper the individuals it knows.
I will add that for much classical music name lookup is useless e.g. "Symphony No. 4" "Ludvig van Beethoven" is hundreds of releases, adding conductor and orchestra is often still not enough to indentify the album with any accuracy.
Yet to examine the successes more carefully to validate data etc. The SQL errors I understand, down to duplicate albums in my test data one with mbid one without (so seen by Kodi as different). Scraper fetches mbid and so we get an error on trying to save it. I'll give what best to do in that case some thought, but it is a core issue not an addon one.
Edit:
Multiple Scraper runs needed
Re-scraping twice more, which just retries to scrape those that have not been successful before, took the unscraped albums down from 107 to 64. Not sure why additional attempts were needed, there were no errors or timeouts in the log for these albums and some already had mbids.
Also I think that the progress bar vanished during the initial scraping causing me to think it had finished when it had not. This could be related perhaps? Either way I have been unable to repeat the progress bar issue, and see nothing in the log. All I can say is odd and something to watch for.
Look up by name identification issues for collaboration albums
Of the remaining 64 albums some of these are expected because they are not in the Musicbrainz or Discogs databases, and some cause the db error that I will fix.
What is causing look up by name identification issues is albums with multiple album artists. Many of my examples of collaborations are classical music, which I generally tag with mbids anyway, but this does effect other genres just to a lesser extent. We could just shrug and tell users to ensure collaborations have mbids, or maybe we could try to pass and use the artist names differently?
An example from my test data is "Neck and Neck" by "Chet Atkins / Mark Knopfler" 3f831051-75f4-3085-8d0f-8b94adbb3c5e
The scraper is passed the artist display text e.g. "Chet Atkins / Mark Knopfler", but requests like
Code:
http://musicbrainz.org/ws/2/release/?fmt=json&query=release:"Neck and neck" AND (artistname:"Chet Atkins / Mark Knopfler" OR artist:"Chet Atkins / Mark Knopfler")However
Code:
http://musicbrainz.org/ws/2/release/?fmt=json&query=release:"Neck and neck" AND (artistname:"Chet Atkins" OR artist:"Chet Atkins")
or
http://musicbrainz.org/ws/2/release/?fmt=json&query=release:"Neck and neck" AND (artistname:"Mark Knopfler" OR artist:"Mark Knopfler")
or
http://musicbrainz.org/ws/2/release/?fmt=json&query=release:"Neck and neck" AND (artistname:"Chet Atkins & Mark Knopfler" OR artist:"Chet Atkins & Mark Knopfler")
or
http://musicbrainz.org/ws/2/release/?fmt=json&query=release:"Neck and neck" AND (artistname:"Mark Knopfler" OR artist:"Mark Knopfler" OR artistname:"Chet Atkins" OR artist:"Chet Atkins")Note that it as having " / " in the display name rather than " & " that mattered in this case, in others it could be use of " and " etc. Getting a matching combined names is going to be tricky. Perhaps Musicbrainz have some advise for queries using multile artist names?
An example from classical music is
"An Irish Symphony / A Comedy Overture" by "Hamilton Harty, Ulster Orchestra, Bryden Thomson" 4c2c013d-d901-4690-97d3-cc99a770b469
As the only release in the db of those works combined it can be found using any one of the three album artists, or a with a precisely punctated combination of all three
"Hamilton Harty" - yes
"Bryden Thomson" - yes
"Ulster Orchestra" - yes
"Hamilton Harty; Ulster Orchestra, Bryden Thomson" - yes
"Hamilton Harty, Ulster Orchestra, Bryden Thomson" - no
"Hamilton Harty / Ulster Orchestra / Bryden Thomson" - no
"Hamilton Harty; Bryden Thomson" - no
Not sure if we can easily pass an array of names to the scraper, or if we just pass a string with separators that the scraper can split. Artist display name itself can not be reliably split, so best let the core tell the scraper the individuals it knows.
I will add that for much classical music name lookup is useless e.g. "Symphony No. 4" "Ludvig van Beethoven" is hundreds of releases, adding conductor and orchestra is often still not enough to indentify the album with any accuracy.
2019-04-22, 09:21
No Chinese support, please recommend music plug-in for Chinese support
2019-04-26, 11:31
Hi, good job.
Can you add a description of the album?
I do not know why, but it does not refresh my album ranking after re-scraping the album:
Changed "Prefer Rating from" Musicbrainz (Rating = 5) to Theaudiodb (Rating = 10) - re-scraping - KODI continually displays rating = 5
In the database table album field fRating = 5, the lastScraped field updates correctly.
Can you add a description of the album?
I do not know why, but it does not refresh my album ranking after re-scraping the album:
Changed "Prefer Rating from" Musicbrainz (Rating = 5) to Theaudiodb (Rating = 10) - re-scraping - KODI continually displays rating = 5
In the database table album field fRating = 5, the lastScraped field updates correctly.
2019-06-10, 13:26
Tested and worked well on a very simple music library with MBID tags all done in picard, just a few comments:
- Can the name of name label of this scraper be changed to "Integral Artist Scraper Python" and "Integral Album Scraper Python" so its obvious which one is being used. The same naming format is used by the new test TheMovieDB python scraper.
- Can you change the default Album "rating" to be TheAudioDB? I think our ratings are better and more accurate than MusicBrainz.
- I would say the same thing for Artist Moods, Genre, Styles personally as TADB uses a simple string that can be used with Nodes. Some may disagree with this but I deliberately kept those fields simple for Kodi.
- TADB has a new artwork type 3dThumb, it would be great to integrate that as well. Last one I promise
And finally a question, does the scraper download the Album review(strReview)? This field is available on TADB but I don't see any sign on the Estuary Skin.
- Can the name of name label of this scraper be changed to "Integral Artist Scraper Python" and "Integral Album Scraper Python" so its obvious which one is being used. The same naming format is used by the new test TheMovieDB python scraper.
- Can you change the default Album "rating" to be TheAudioDB? I think our ratings are better and more accurate than MusicBrainz.
- I would say the same thing for Artist Moods, Genre, Styles personally as TADB uses a simple string that can be used with Nodes. Some may disagree with this but I deliberately kept those fields simple for Kodi.
- TADB has a new artwork type 3dThumb, it would be great to integrate that as well. Last one I promise
And finally a question, does the scraper download the Album review(strReview)? This field is available on TADB but I don't see any sign on the Estuary Skin.
2019-06-18, 06:41
I’m so glad I found this thread, is the scraper working properly now? Is it safe to use? I run KODI on a RPi with LibreElec and my music library is somewhat on the large side. Being forced to re-tag the music would take a day easily, so I’d rather ask you people here for the “go ahead” before I take the plunge.
Missing an artist or two is nothing, I just don’t want to risk corrupting the library or tags by accident.
Missing an artist or two is nothing, I just don’t want to risk corrupting the library or tags by accident.
2019-06-18, 07:38
This is still an experimental scraper. Use the default scrapers already installed.
(2019-06-18, 06:41)markusA Wrote: Being forced to re-tag the music would take a day easilyMaybe I have misunderstood your comment, but you need to accurately tag your music files first. The scraper will not fix bad tagging, in fact it will make it worse. I strongly urge you to spend that day on re-tagging. See the How To music guide in My Signature below.
2019-11-24, 19:59
Hey Guys,
Is this still being worked on?
Like, has anyone been able to pick this up elsewhere (Github Forks, Forums etc.)?
Any Updates or Knowledge?
Is this still being worked on?
Like, has anyone been able to pick this up elsewhere (Github Forks, Forums etc.)?
Any Updates or Knowledge?
2019-11-30, 15:34
although i haven't been actively working on it for the last months, the aim still is to get this included in kodi v19.
(https://github.com/xbmc/xbmc/pull/15665)
the latest versions of these python scrapers for kodi v19 are available in my repo:
https://gitlab.com/ronie/metadata.generic.albums
https://gitlab.com/ronie/metadata.generic.artists
there will not be a version for kodi v18 though.
(https://github.com/xbmc/xbmc/pull/15665)
the latest versions of these python scrapers for kodi v19 are available in my repo:
https://gitlab.com/ronie/metadata.generic.albums
https://gitlab.com/ronie/metadata.generic.artists
there will not be a version for kodi v18 though.
2020-04-19, 10:31
Hello @ronie. Thanks for all your work on music scraping.
I just installed Kodi 19 and tested these scrapers on my music library (~400 albums). Here is my log: https://paste.ubuntu.com/p/DPz58gcgWx/
I see that all my music is not scraped, and I have to manually update the albums. It generally works by updating them so I'm wondering why they haven't been scraped during the first library scan. They are not mentioned in the logs so no errors are reported on them. And in the settings, I chose to prioritize online information over local information so I guess it is supposed to scrap all albums.
I just installed Kodi 19 and tested these scrapers on my music library (~400 albums). Here is my log: https://paste.ubuntu.com/p/DPz58gcgWx/
I see that all my music is not scraped, and I have to manually update the albums. It generally works by updating them so I'm wondering why they haven't been scraped during the first library scan. They are not mentioned in the logs so no errors are reported on them. And in the settings, I chose to prioritize online information over local information so I guess it is supposed to scrap all albums.
