Games Library (Game & Emulator database + launcher) development project, help wanted! - Printable Version
+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (/forumdisplay.php?fid=32)
+--- Forum: Kodi Application (/forumdisplay.php?fid=93)
+---- Forum: RetroPlayer (/forumdisplay.php?fid=194)
+---- Thread: Games Library (Game & Emulator database + launcher) development project, help wanted! (/showthread.php?tid=40715)
- VictorV - 2009-11-20 20:42
Nicezia Wrote:GiantBomb Game Scraper
Still got that scraper? I would love to have a scraper to test with
- tmpdir - 2009-11-20 21:37
thanks greatant1337 Missed that thread completly in my search (will read it this weekend). Without reading it I'm convinced the requirements are of the kind 'moving targets'.
- tmpdir - 2009-11-20 21:41
rickx Wrote:I'm not exactly an svn guru. If I check out the games-branch, what version (revision) will it be? Is it possible to merge it with the current rev.? If so, could somebody drop the code for doing it? (Apart from probably having to solve some issues/conflicts manually...)
svn merge is quite easy to use, just use google. Expect some minor merge issues, some of the depencendies have been updated recently.
- rickx - 2009-11-20 22:50
tmpdir Wrote:svn merge is quite easy to use, just use google. Expect some minor merge issues, some of the depencendies have been updated recently.
Yes, I've heard google before...I also had tried already, but if someone could kindly comfirm it is right, I could spare some precious time...
I checked out the branch in one folder, the latest in another.
Then(?) I went inside the latest folder and used
But maybe it should be the other way round. Or adding parameters...
- tmpdir - 2009-11-21 12:51
Next time you might want to mention what you've already tried... to many people ask before getting their hands dirty.
It's been a while, but its something like:
svn merge \
The dot is your current directorie where the merge should arrive. After checking the results etc, you can commit . to your working folder.
- the 'equal_directorie_name' folders must be the same.
- works the same with local folders
- the . points to you current dir.
- Mityama - 2009-12-01 23:09
I have been thinking about creating a game database as an alternative to something like allgame. But in contrast to those, it would be XBMC specific and contain exactly the info and media in exactly the form needed.
Furthermore.... I am not a scraper expert, but how does the following sound as a way to grab game information with a very high accuracy rate:
- XBMC calculates the MD5 hash of a game ROM, then
- sends it to the site via a HTTP request like "http://[site]/?md5=5443508gfbfg98zh",
- where a small script looks up the needed data in the database, and
- passes it back to XBMC in some form (XML?)
I would need to include all known hashes in the database, but I think this can be automated with some Good Sets and a little PHP script. I could also implement a search for the game title if a ROM is unknown.
But before I start getting more into this, I would like you guys' opinions about this first. Is it worth the effort? Would it help the game library project?
- VictorV - 2009-12-02 15:10
Mityama Wrote:Hello everybody,It's a shitload of work, but it would be awesome if you did.
Mityama Wrote:Furthermore.... I am not a scraper expert, but how does the following sound as a way to grab game information with a very high accuracy rate:I would recommend crc since that's what is commonly used in various communities
Mityama Wrote:- sends it to the site via a HTTP request like "http://[site]/?md5=5443508gfbfg98zh",you might also want to add a fuzzy search based on file name and platform in case there's no known crc.
Mityama Wrote:- where a small script looks up the needed data in the database, andIt's a lot of dat-files around containing crc¨'s for rom collections
Mityama Wrote:I could also implement a search for the game title if a ROM is unknown.I also think that high quality data is the key.
I've been playing around with the game library and have made some progress there. You probably want to start early designing a descent database, and you could see what attributes you might want (Should probably be discussed before settled). There's also a need for a good structure handling differences in platforms and games on cross of platforms (cover art and screen shots etc. will vary, but not the plot)
- rcoops - 2009-12-02 15:51
I would be more then happy to help out with this effort of setting up a good as complete as possible games database.
I think that hashing is definitely an option but I fear that there are a lot of versions of roms for many games. Looking at the thousands roms I happen to have sitting on a disk somewhere there are often 2 to 5 different versions of the same game sometimes the language changes sometimes the way the rom has been plucked from its original storage etc... a look up based on platform and game title would definitely be needed, but also a series of names for the same rom, different countries often have different names for the same rom.
I think the first thing would have to be to agree on a basic set of data required for each rom, expanding on this later on is a lot easier to do simply add more tags in the XML and mark them as optional for people using the XML. That way you do not have to add the information for all roms but only for those where it is known.
@Mityama if you are interested in help with this massive undertaking please don't hesitate to drop me a PM. Despite being short on time most of the time I would gladly spend some of my free time on a project like this.
- JustinAiken - 2009-12-02 17:20
All I'd really like to see for games scraping is
Title (and orig title)
- rcoops - 2009-12-02 19:47
I would add to that:
- mngrl1 - 2009-12-03 00:25
This all sounds Awesome guys am very happy to see people still interested in it. Just one thing scraping manuals and cheats canary and all that sounnds really good but can't we just start simple. Think there would actually be more likely for this project to see the light of day rather then putting way to much in and never finishing it
- mngrl1 - 2009-12-03 00:27
Lol still getting used to typing on this Iphone canary lol was ment to be fanart
- VictorV - 2009-12-03 09:01
This is attributes I'm working with now:
The artworks use some general naming since it could be box art for console games, but cabinet for arcade games.
Both action and intro shots are commonly used, so I figured to stick to that.
Genre would be action, adventure etc, while perspective could be side-scroller
I think manuals are more of an in-game feature, but it could be considered on a later stage.
- rcoops - 2009-12-03 11:23
Sounds good to me.
As for the manual, walk trough and cheats I would just add them as optional extra's only there where available and rely on visitors for the most part to populate these.
The only thing that I worry about a little is that certain game studio's might have an issue with box/card/arcade art being made available. Nintendo has a team of lawyers looking into the telephones that are able to run emulators in an attempt to charge for it or ban them completely as it considers these games it's intellectual property and any use even 10 or more years after they stopped selling them should be paid for. They might react in the same way to box/card/arcade art (their design after all).
It might be a good idea to contact them and try and get a ok from them. Usually big companies like that will allow these types of thing when asked, they only crackdown on it when they think there is real money to be made from it. Once you have a written approval any legal action they might feel they need to take later on will be severely hampered by such correspondence so if they do approve don't loose that email.
- rausch101 - 2009-12-03 15:26
I always enjoy seeing interest/work in this thread. Nice job guys. And remember, since there's currently nothing implemented, don't get bogged down by extras before you get the core stuff done
rcoops Wrote:Sounds good to me.
If Games posters are at all analogous to TV Shows, Movies or Music posters, then this isn't a problem at all. I imagine the websites from which he will be scraping will have dealt with all of that legal stuff anyway, if there was any.