Beta [Please Test] New Music Scrapers
#1
About
One of the new features in Kodi 18 Leia is the option to use python based scrapers.
The intent is to replace our current XML based scrapers in time with these new python scrapers.

I have created a python Artist and Album scraper, that provide similar functionality as the current Universal scrapers.


Dowloads
If you are interested in helping out with testing the new scrapers, you can download them here:

Setup
Installation:
Settings > Addons > Install from zip file

Configuration (optional):
Settings > Addons > My Addons > Information providers > Artist information > Integral artist scraper > (context menu) > Settings
Settings > Addons > My Addons > Information providers > Album information > Integral album scraper > (context menu) > Settings

Activation:
Settings > Media > Music > Default provider for album information
Settings > Media > Music > Default provider for artist information


Testing
Please scan a small subset of your music collection with the new scrapers.
When finished scraping, check if all metadata/artwork is properly scraped in your music library.

In case you spot something is not as it should be, or if the scraper crashing during the scan:
please post a Debug Log.
Do not PM or e-mail Team-Kodi members directly asking for support.
Always read the Forum rules, Kodi online-manual, FAQ, Help and Search the forum before posting.
Reply
#2
Hi,

I did a little test using the python artist and album scraper.

Here's what I did:

- rename the existing metadata/artwork from Artist Information Folder for an existing artist: Annbjørg Lien>Annbjørg Lien_save
- in Music>Files view, on the album Aliens Alive, Scan item to library

Now I play the album Aliens Alive from this artist Annbjørg Lien and expect the metadata/artwork to be written to the Artist Information folder, but it doesn't. Should it?

Here's the kodi log
Reply
#3
(2019-03-15, 09:30)eljejer Wrote: Hi,

I did a little test using the python artist and album scraper.

Here's what I did:

- rename the existing metadata/artwork from Artist Information Folder for an existing artist: Annbjørg Lien>Annbjørg Lien_save
- in Music>Files view, on the album Aliens Alive, Scan item to library

Now I play the album Aliens Alive from this artist Annbjørg Lien and expect the metadata/artwork to be written to the Artist Information folder, but it doesn't. Should it?

Here's the kodi log
nope, scrapers don't store artwork locally.
they only provide the url to kodi, which stores the url in the music database.
 kodi will ofc download the artwork, but afaik it's cached in userdata/thumbnails.
Do not PM or e-mail Team-Kodi members directly asking for support.
Always read the Forum rules, Kodi online-manual, FAQ, Help and Search the forum before posting.
Reply
#4
I made a quick test run and I am seeing a lot of "warning Attempt to use invalid handle nnn" when artist scraper starts up.
Also had an exception in album scraper when finding an album using album MBID.  It looked like in scraper.py when it does "getdetails" it isn't getting a result and
Code:
artist = details['musicbrainz']['artist_description'].encode('utf-8')
doesn't have the 'musicbrainz' key (I'm assuming details is empty?)

My log was kind of huge I will try to repeat with just a couple albums to scan.

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Krypton
Leia
Reply
#5
(2019-03-16, 02:40)scott967 Wrote: I made a quick test run and I am seeing a lot of "warning Attempt to use invalid handle nnn" when artist scraper starts up.

yup, those been bugging me since i started writing the scrapers.
most likely i'm not handling plugin url's correctly...
need to check the python docs again :-)
(2019-03-16, 02:40)scott967 Wrote: Also had an exception in album scraper when finding an album using album MBID.  It looked like in scraper.py when it does "getdetails" it isn't getting a result and
Code:
artist = details['musicbrainz']['artist_description'].encode('utf-8')
doesn't have the 'musicbrainz' key (I'm assuming details is empty?)  
thx! i'm indeed not checking if musicbrainz returns anything, will fix.
Do not PM or e-mail Team-Kodi members directly asking for support.
Always read the Forum rules, Kodi online-manual, FAQ, Help and Search the forum before posting.
Reply
#6
I notice some SQL errors in the kodi log when scraping
I attach kodi log here
Reply
#7
@eljejer do you have two copies of "Cornelis Vreeswijk sjunger Taube" scanned into the music library, or perhaps a similarly named album that the scaper could have mis-identified?  The error is related to the scraped album Musicbrainz ID (for an album not tagged with mbid tags) being the same as another album aready in the library. But hard to know the details without a log with debug turned on. Could you do that, go to the album information dialog for that album and click on refresh, then repost the log file.
Reply
#8
besides the db error, there's a few scraper related errors in your log as well.

i'll will look into those, but i'd also welcome a Debug Log,
it would make it easier for me to reproduce those errors if i could see which artist / album is triggering those errors.
Do not PM or e-mail Team-Kodi members directly asking for support.
Always read the Forum rules, Kodi online-manual, FAQ, Help and Search the forum before posting.
Reply
#9
Maybe the album ¨Cornelis Vreeswijk sjunger Taube¨ is duplicated as a result of the following (while creating a subset of my music collection to be handled by the new python scraper):
- I moved 5 artist folders (among which Cornelis Vreeswijk) on the filesystem from the existing music directory to a newly created directory
- I cleaned the music library in Kodi; Settings / Media > Library > Music Library > Clean Library
- I re-scanned the music library in Kodi ; Settings / Media > Library > Manage Sources > Music... > Context menu existing music library > Scan item to library
- I added the subset to the music library in Kodi; Settings / Media > Library > Manage Sources > Music... > Add music... (path on filsystem to subset) and let Kodi scan the new source to the music library
Huh

In any case is the album sjunger Taube presented twice in the UI (I attach screenshot).

I turned debug on and ¨refreshed¨ in the album information dialogue for both occurences of the album in question

Here's the log.
Image
Thanks
Reply
#10
Thanks for looking into this.

I attach the same log as in my previous post, although I'm not sure if this is what you require, because I'm not sure if it's regarding the same action I did before...?
Reply
#11
I'm not sure if I had processed this album (completely) before in MusicBrainz Picard, but now I have and re-scanned the artist to the library.
Right now, the duplicate entry for the album Cornelis sjunger Taube is gone anyway.
I attach the debug log once more.
Reply
#12
Seeing a bunch of errors in kodi log again.
This time a changed back from smb to nfs on my file server.
Then I deleted all music (& videos) from kodi and cleaned the library.
I re-added the music using nfs source.
The default provider for music is set to Integral Albums Scraper.
I attach the log here.
Reply
#13
I tested integral artist scraper 0.0.2 on the artist "The Dell Vikings"

I got artist genre from All Music

The scraper is returning 2 genres:
R&B;
Pop/Rock

I repeated the test with UAS scraper and this one returned
R&B
Pop/Rock

That is, the "R&B" genre was returned/added to the db without the trailing semi-colon.
The was repeated with some other artists having allmusic genre of R&B

I don't know the allmusic API, but from browsing the web site it displays the genre as expected (no trailing semi-colon).

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Krypton
Leia
Reply
#14
(2019-03-22, 00:18)eljejer Wrote: Seeing a bunch of errors in kodi log again.
 the errors caused by the scraper itself should be fixed in v0.0.2
(download available in the first post)
Do not PM or e-mail Team-Kodi members directly asking for support.
Always read the Forum rules, Kodi online-manual, FAQ, Help and Search the forum before posting.
Reply
#15
(2019-03-22, 02:04)scott967 Wrote: I tested integral artist scraper 0.0.2 on the artist "The Dell Vikings"

I got artist genre from All Music

The scraper is returning 2 genres:
R&B;
Pop/Rock

I repeated the test with UAS scraper and this one returned
R&B
Pop/Rock

That is, the "R&B" genre was returned/added to the db without the trailing semi-colon.
The was repeated with some other artists having allmusic genre of R&B

I don't know the allmusic API, but from browsing the web site it displays the genre as expected (no trailing semi-colon).

scott s.
.
 allmusic doesn't have an api, we scrape the website instead.

the root cause of this issue is they're using invalid html there, the & character has to be escaped (&), but they don't.
next up, the beatifulsoup library we use the parse the html, tries to correct invalid html, but that makes it worse in this case (&B - > &B;)
(https://bugs.launchpad.net/beautifulsoup/+bug/1685044)

i will have to work around it in the addon.
Do not PM or e-mail Team-Kodi members directly asking for support.
Always read the Forum rules, Kodi online-manual, FAQ, Help and Search the forum before posting.
Reply
 
Thread Rating:
  • 0 Vote(s) - 0 Average



Logout Mark Read Team Forum Stats Members Help
[Please Test] New Music Scrapers00