[garbear]

I see . I had in mind to move them to a separate module when I would start to implement other scrapers. But I can understand if you don't want them to be a part of heimdall at all. Maybe I will re-add them in a RCB specific version later (as you might have noticed I am not an elegant coder in the first place). From a pragmatic point of view downloading artwork is the most time consuming task in the whole scraping process and doing it multithreaded saves some valuable seconds per game. So before I start writing up some crappy threading thing in RCB I prefer to reuse what Tobias has already implemented.

In my last post I mentioned one of the theories that a pragmatic point of view might overlook. And your pragmatism didn't go unnoticed Artwork downloaders might not belong in a metadata extractor, but the dataflow paradigm in heimdall is perfect for executing a large number of heavy tasks in parallel as their data becomes available. I think that when tobias makes heimdall an xbmc library, you can import the package and supply your own artwork downloader module. That way the download locations and configuration, etc. can be hidden from heimdall.

Not yet convinced that this may work but I am ready to be surprised (and to test). If it helps: I have created kind of a testset some years ago to test some of the more problematic games with RCB. Maybe I am able to dig it out or to recreate it.

Cool, I'd like to see the test. As with any sort of natural-language search, domain knowledge and machine learning will probably be the key to optimality. The last filter I wrote somehow got dumber as it went on... not a good sign

Not sure if I understand that right. Will the client get one result and decides if it is the correct one or will it be a two step approach where the client gets a list of results and starts a second run with the correct item?

Probably a callback function that takes a list of results and returns the correct/chosen one, or an error code to abort the current task.

As mobygames is one of the most complete available sources atm I think I would continue with this one. If you want to start this yourself just let me know and I will sit still. Otherwise I could do the groundwork again and you could chime in where I sucked. If I would give it a try I would start with BeautifulSoup. Any better ideas?

I'm of the opinion that the dude with the 10,000 line python program knows what he's doing And I'm pretty wrapped up in my filter right now. If you push to a development branch, I'll probably check in after a few days to make sure Heimdall is being worshiped properly

Another question about implementing scrapers: As heimdall might be more official part of XBMC, how do you handle scraping permissions for the scraped sites? For example, I got official permission to scrape data from thegamesdb, archive.vg and giantbomb. Mobygames did not respond to my request but they do not explicitly forbid scraping in their terms of use, so I decided it will be ok to implement it. No idea if XBMC can go the same route or if we have to ask again. GameFAQ did also not respond but they explicitly forbid automated scraping so I decided not to implement it. Maybe they will respond if XBMC asks officially for scraping permission.

For now, Heimdall and XBMC are separate, so TOS negotiations are handled separately.

I also want to get as much information as possible from the scrapers. E.g. thegamesdb also provides information and artwork for consoles and publishers/developers. Do you plan to use this in heimdall/RetroPlayer too?

The more the merrier I think you would target consoles or publishers as subjects in Heimdall, like container.audio.Artist and container.audio.Album classes.