[meowmoo]

EDIT: Split from Music Library Development Roadmap - One Man's Plan in tidy up

scanning right now, at first look, I dont get any additional info.

I let it finish and report back

[DaveBlake]

scanning right now, at first look, I dont get any additional info.

The combined scan (tag) + scrape (additional info) sequence has changed.
To populate the lib fastest, so you can play stuff, all the scanning happens first. Then after that is complete it scrapes info and art for the added artists and albums. This is the slower part - 1s per album (or 2s if lookup by name as no mbid tags), 1s per artist (or 2s if no mbid), but it is a background task.

The key thing is to check for 503 errors in the log (or that you have info for all the artists and albums that you expect, no gaps)

I let it finish and report back

Great. Thanks for testing this, always good to have real world feedback.

[meowmoo]

All good at the End, no 503 errors I the log, and all information is there

[JanneT]

Dave,

did you introduce delays in the scraping process?

If so, is there a way to turn them off?

The reason for asking is that I have put together a simple web application with corresponding scrapers for artists and albums. Now, this process is an order of magnitude slower than before. It used to pretty fast compared to MusicBrainz etc. Ok, I have to manually update the database. But for some reasons, with the music I normally listen to, I have to do some manual work anyway. And MusicBrainz is not that user friendly, At least I have a Web GUI to handle the content with Cut-and-Paste etc. :-)

- Janne

[Martijn]

Dave,

did you introduce delays in the scraping process?

yes

If so, is there a way to turn them off?

nope.
It's all or nothing and we decided to throttle online info to reduce load on Musicbrainz to get it working

[JanneT]

Dave,

did you introduce delays in the scraping process?

yes

If so, is there a way to turn them off?

nope.
It's all or nothing and we decided to throttle online info to reduce load on Musicbrainz to get it working

OK, I have to accept that. I could scrape 600-700 albums in less than 1 minute from my local server. :-(

[Martijn]

There's a difference between scraping and scanning.
Also it's a one time job so it doesn't matter that it takes longer the first time.

[JanneT]

There's a difference between scraping and scanning.
Also it's a one time job so it doesn't matter that it takes longer the first time.

In this case, it's the scraping I'm talking about. From my local Web server.

[Karellen]

As Martijn said, the initial library scrape is a one time process. How many times are you scraping your library and why??

[steve1977]

Any thoughts on below? Cross-posting here as it is very music related.

https://forum.kodi.tv/showthread.php?tid=317925

[DaveBlake]

did you introduce delays in the scraping process?

If so, is there a way to turn them off?

The reason for asking is that I have put together a simple web application with corresponding scrapers for artists and albums. Now, this process is an order of magnitude slower than before. It used to pretty fast compared to MusicBrainz etc. Ok, I have to manually update the database. But for some reasons, with the music I normally listen to, I have to do some manual work anyway. And MusicBrainz is not that user friendly, At least I have a Web GUI to handle the content with Cut-and-Paste etc. :-)

OK, I have to accept that. I could scrape 600-700 albums in less than 1 minute from my local server. :-(

@JanneT I don't understand what you are doing.

The Universal Artist Scraper and the Universal Album Scraper make requests to Musicbrainz. Unless those requests are throttled to less than 1 per second the Musicbrainz server returns 503 errors and blocks requests from your IP address until they arrive at a slow enough rate. Previous attempts at throttling were flawed, and users got lots of scraper failures (experienced as gaps in artwork for well known music). I fixed this - so scraping is slower (correctly throttled) but gets results!

No idea what you have been doing that worked on 700 albums a minute, your own xml scraper?? You need to explain more what you are trying to achieve if you want help.

[JanneT]

did you introduce delays in the scraping process?

If so, is there a way to turn them off?

The reason for asking is that I have put together a simple web application with corresponding scrapers for artists and albums. Now, this process is an order of magnitude slower than before. It used to pretty fast compared to MusicBrainz etc. Ok, I have to manually update the database. But for some reasons, with the music I normally listen to, I have to do some manual work anyway. And MusicBrainz is not that user friendly, At least I have a Web GUI to handle the content with Cut-and-Paste etc. :-)

OK, I have to accept that. I could scrape 600-700 albums in less than 1 minute from my local server. :-(

@JanneT I don't understand what you are doing.

The Universal Artist Scraper and the Universal Album Scraper make requests to Musicbrainz. Unless those requests are throttled to less than 1 per second the Musicbrainz server returns 503 errors and blocks requests from your IP address until they arrive at a slow enough rate. Previous attempts at throttling were flawed, and users got lots of scraper failures (experienced as gaps in artwork for well known music). I fixed this - so scraping is slower (correctly throttled) but gets results!

No idea what you have been doing that worked on 700 albums a minute, your own xml scraper?? You need to explain more what you are trying to achieve if you want help.

Yes, that was with my own scraper that uses an application on a webserver at home. The idea was to become independent of MusicBrainz as a lot of the music I listen to requires some manual work with MB, TADB etc. anyway. So I thought I would be better off with a private server to maintain the metadata. It works quite well for my needs. Even for Composers.

But as others have pointed out, it's no big deal except for now when I test my webapp. Otherwise I think that throttling the requests is a good idea.

[DaveBlake]

Yes, that was with my own scraper that uses an application on a webserver at home. The idea was to become independent of MusicBrainz as a lot of the music I listen to requires some manual work with MB, TADB etc. anyway. So I thought I would be better off with a private server to maintain the metadata. It works quite well for my needs. Even for Composers.

But as others have pointed out, it's no big deal except for now when I test my webapp. Otherwise I think that throttling the requests is a good idea.

Creating your own private xml scraper is unusual. Perhaps get the webapp to create NFO files instead, Kodi will scarpe those without delay.

[JanneT]

Yes, that was with my own scraper that uses an application on a webserver at home. The idea was to become independent of MusicBrainz as a lot of the music I listen to requires some manual work with MB, TADB etc. anyway. So I thought I would be better off with a private server to maintain the metadata. It works quite well for my needs. Even for Composers.

But as others have pointed out, it's no big deal except for now when I test my webapp. Otherwise I think that throttling the requests is a good idea.

Creating your own private xml scraper is unusual. Perhaps get the webapp to create NFO files instead, Kodi will scarpe those without delay.

Yes, that is an idea I've bee thinking about. But what of albums with multiple Artists?

[DaveBlake]

Collaboration albums (multiple album artists) are a problem for scraping NFOs and artwork, something I hope to address.

Meanwhile creating a single import file and importing it could work - play with export to a single file and see what you get. Remember for music export/import only contains the scraped album and artist data, not the things derrived from tags.