Beta [Please Test] New Music Scrapers
#16
Hi, I've installed integral scraper versions 0.0.2 and am seeing an SQL ERROR.
I attach the kodi log here
Reply
#17
(2019-03-26, 11:14)eljejer Wrote: Hi, I've installed integral scraper versions 0.0.2 and am seeing an SQL ERROR.
@eljejer plenty of errors in that log still, some NFS access related. The db error is about artist "Gilbert O'Sullivan", and I suspect that some of your music by him is not tagged with Musicbrainz IDs and that the naming is not consistent. Hence you possibly have two entries for him in your library, maybe one with a different unicode char between "O" and"S". One entry has mbid, the other the scraper just fetched mbid based on name match but triying to save it fails because it alreday has an entry with that mbid.

Non-ascii chars in names can be such a pain. Check all your music files and either tag with mbids or take great care that naming is consistent especially those with "`" or "-" in them.
Reply
#18
Now I'm seeing some python errors in the log, which I attach here
And some more SQL errors as well
Reply
#19
Thanks, I will look into that
Reply
#20
Thanks, I will look into the non ASCII character matter on my part
Reply
#21
@ronie I've done some album scraping with v002 and got a crop of script errors for you to check. Log is a monster 15MB so zipped here, 1435 albums out of 1542 scraped successfully taking 2h 15m.

Yet to examine the successes more carefully to validate data etc. The SQL errors I understand, down to duplicate albums in my test data one with mbid one without (so seen by Kodi as different). Scraper fetches mbid and so we get an error on trying to save it. I'll give what best to do in that case some thought, but it is a core issue not an addon one.

Edit:
Multiple Scraper runs needed
Re-scraping twice more, which just retries to scrape those that have not been successful before, took the unscraped albums down from 107 to 64. Not sure why additional attempts were needed, there were no errors or timeouts in the log for these albums and some already had mbids.

Also I think that the progress bar vanished during the initial scraping causing me to think it had finished when it had not. This could be related perhaps? Either way I have been unable to repeat the progress bar issue, and see nothing in the log. All I can say is odd and something to watch for.

Look up by name identification issues for collaboration albums
Of the remaining 64 albums some of these are expected because they are not in the Musicbrainz or Discogs databases, and some cause the db error that I will fix.

What is causing look up by name identification issues is albums with multiple album artists. Many of my examples of collaborations are classical music, which I generally tag with mbids anyway, but this does effect other genres just to a lesser extent. We could just shrug and tell users to ensure collaborations have mbids, or maybe we could try to pass and use the artist names differently?

An example from my test data is "Neck and Neck" by "Chet Atkins / Mark Knopfler" 3f831051-75f4-3085-8d0f-8b94adbb3c5e

The scraper is passed the artist display text e.g. "Chet Atkins / Mark Knopfler", but requests like
Code:
http://musicbrainz.org/ws/2/release/?fmt=json&query=release:"Neck and neck" AND (artistname:"Chet Atkins / Mark Knopfler" OR artist:"Chet Atkins / Mark Knopfler")
do not get a result from Musicbrainz.
However
Code:
http://musicbrainz.org/ws/2/release/?fmt=json&query=release:"Neck and neck" AND (artistname:"Chet Atkins" OR artist:"Chet Atkins")
or
http://musicbrainz.org/ws/2/release/?fmt=json&query=release:"Neck and neck" AND (artistname:"Mark Knopfler" OR artist:"Mark Knopfler")
or
http://musicbrainz.org/ws/2/release/?fmt=json&query=release:"Neck and neck" AND (artistname:"Chet Atkins & Mark Knopfler" OR artist:"Chet Atkins & Mark Knopfler")
or
http://musicbrainz.org/ws/2/release/?fmt=json&query=release:"Neck and neck" AND (artistname:"Mark Knopfler" OR artist:"Mark Knopfler" OR artistname:"Chet Atkins" OR artist:"Chet Atkins")
will find the album sucessfully.

Note that it as having " / " in the display name rather than " & " that mattered in this case, in others it could be use of " and " etc. Getting a matching combined names is going to be tricky. Perhaps Musicbrainz have some advise for queries using multile artist names?

An example from classical music is
"An Irish Symphony / A Comedy Overture" by "Hamilton Harty, Ulster Orchestra, Bryden Thomson" 4c2c013d-d901-4690-97d3-cc99a770b469
As the only release in the db of those works combined it can be found using any one of the three album artists, or a with a precisely punctated combination of all three
"Hamilton Harty" - yes
"Bryden Thomson" - yes
"Ulster Orchestra" - yes
"Hamilton Harty; Ulster Orchestra, Bryden Thomson" - yes
"Hamilton Harty, Ulster Orchestra, Bryden Thomson" - no
"Hamilton Harty / Ulster Orchestra / Bryden Thomson" - no
"Hamilton Harty;  Bryden Thomson" - no

Not sure if we can easily pass an array of names to the scraper, or if we just pass a string with separators that the scraper can split. Artist display name itself can not be reliably split, so best let the core tell the scraper the individuals it knows.

I will add that for much classical music name lookup is useless e.g. "Symphony No. 4" "Ludvig van Beethoven" is hundreds of releases, adding conductor and orchestra is often still not enough to indentify the album with any accuracy.
Reply
#22
No Chinese support, please recommend music plug-in for Chinese support
Reply
#23
Hi, good job.
Can you add a description of the album?

I do not know why, but it does not refresh my album ranking after re-scraping the album:
Changed "Prefer Rating from" Musicbrainz (Rating = 5) to Theaudiodb (Rating = 10) - re-scraping - KODI continually displays rating = 5
In the database table album field fRating = 5, the lastScraped field updates correctly.
Reply
#24
Tested and worked well on a very simple music library with MBID tags all done in picard, just a few comments:

- Can the name of name label of this scraper be changed to "Integral Artist Scraper Python" and "Integral Album Scraper Python" so its obvious which one is being used. The same naming format is used by the new test TheMovieDB python scraper.
- Can you change the default Album "rating" to be TheAudioDB? I think our ratings are better and more accurate than MusicBrainz.
- I would say the same thing for Artist Moods, Genre, Styles personally as TADB uses a simple string that can be used with Nodes. Some may disagree with this but I deliberately kept those fields simple for Kodi.
- TADB has a new artwork type 3dThumb, it would be great to integrate that as well. Last one I promise Smile

And finally a question, does the scraper download the Album review(strReview)? This field is available on TADB but I don't see any sign on the Estuary Skin.
Image Image
Reply
#25
Thanks for great post its very helpful for me.
Reply
#26
I’m so glad I found this thread, is the scraper working properly now? Is it safe to use? I run KODI on a RPi with LibreElec and my music library is somewhat on the large side. Being forced to re-tag the music would take a day easily, so I’d rather ask you people here for the “go ahead” before I take the plunge. Smile
Missing an artist or two is nothing, I just don’t want to risk corrupting the library or tags by accident.
Reply
#27
This is still an experimental scraper. Use the default scrapers already installed.

(Yesterday, 06:41)markusA Wrote: Being forced to re-tag the music would take a day easily
Maybe I have misunderstood your comment, but you need to accurately tag your music files first. The scraper will not fix bad tagging, in fact it will make it worse. I strongly urge you to spend that day on re-tagging. See the How To music guide in My Signature below.
My Signature
Links to : Forum rules (wiki) | Banned_Add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create_Music_Library (wiki) | HOW-TO:Create_Video_Library (wiki) || Artwork (wiki) | Basic_controls (wiki) | Import-export_library (wiki) | Movie_sets (wiki) | NFO_files (wiki) | Quick start guide (wiki)
Reply
 
Thread Rating:
  • 0 Vote(s) - 0 Average



Logout Mark Read Team Forum Stats Members Help
[Please Test] New Music Scrapers00