+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Program Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=151)
+---- Thread: [RELEASE] Emulatopia - Addon for handling, browsing, and launching emulator game ROMs (/showthread.php?tid=87758)
[RELEASE] Emulatopia - Addon for handling, browsing, and launching emulator game ROMs - quakes - 2010-12-16
This is yet another addon for handling, browsing, and launching emulated games.
There are a few similar addons, such as RCB and ALA (though I didn't realize ALA existed until after I had started building this), but none worked quite the way I wanted them to. So I wrote my own! As such, this was not a humongous and ambitious project, but the goal was rather simply to make something that fit my needs. I figured that others might make use of this as well, and I am thus releasing it here.
Since I have not been involved at all in XBMC before (as a side note, XBMC is the single worst documented platform I've ever coded for), and do not have an excessive knowledge of Python, I have made some beginner mistakes that I would rather have avoided. But the end result works for my uses at least.
Since I am a sucker for consistency, and use Confluence myself, the addon tries to mimic the look and behavior of Confluence as closely as possible.
Anyway, I worked a solid two-three weeks or so to make this add-on (mostly wrestling with XBMC to make it do what I want), and I hope you guys find some use in it as well.
Download link
Version 0.2.4 (latest)
Latest changes
0.2.4
- Fixed bug regarding emulator paths that had spaces. I had forgotten to properly escape these.
- Fixed bug where XBMC would not wait until the emulator had terminated until maximizing again.
- Fixed bug where context menu entries were invisible when using skins other than confluence.
- Turns out some emulators (looking at you, project 64) do not like quoted ROM paths. Thus, I added a %UNESCAPED_ROM% command line argument that does not add the surrounding quotes.
Feature overview
- All configuration is done via the built-in addon settings function, making it relatively simple to use
- Supports automatic scraping of title, image, short description, release date, and genres from GiantBomb
- Look-and-feel made to match Confluence as closely as possible, including the shift+letter "jump to" function. That is, if you hold shift and type "LEG", it will jump to the first game starting with "LEG". It supports letters from A-Z and space.
- Supports GoodMerge-style .7z files containing several ROMs. At launch, if the addon detects that the file is a 7z file, a select dialog pops up letting you choose which of the ROMs inside it to launch.
- Supports marking games as "played" (in the same style as XBMC lets you mark videos as "watched"), either automatically at launch, or manually through the context menu
- Though not entirely intended behavior, Emulatopia can handle regular standalone games as well, both to scrape and launch, with some slight trickery (more on this later).
How to use
First of all, the add-on has been made exclusively for Windows! I do not have an xbox, nor a front-end mac/linux machine to run XBMC on. Therefore, though platform independent alternatives have been chosen where applicable, the add-on is likely not to work very well on non-Windows platforms. It could likely be modified to work on other platforms relatively easily, but I am not currently in a position to do this.
So, assuming you run Windows, the second requirement is that you run XBMC Dharma, preferably RC1 or later. Most of the testing has been done on Dharma RC2.
The first time you run the add-on, a dialog will pop up notifying you that you have not defined any collections. The settings dialog will then be opened. Once here, navigate to the Collections category. You should define one collection per platform (that is, one NES collection, one SNES collection, etc).
Defining your collections
To define a collection, first enable one. Some additional settings should then pop up, namely Console, ROM folder, ROM file extension, Emulator executable and Emulator arguments.
The console setting defines what platform the ROMs belong to. This roughly corresponds to the list of platforms on GiantBomb.com for simplicity.
The ROM folder should be set to the folder containing your ROMs.
The ROM file extension is a comma separated list of your ROM file extensions. If all your ROMs have the same file extension, simply enter that. My NES collection consists entirely of .7z files, so I enter "7z" as the ROM file extension. My PSX collection, however, contains both .bin files and .img files, so I enter "bin,img" as the ROM file extension.
The emulator executable should point to the .exe file of your emulator of choice.
The emulator arguments are the command line arguments that are passed to the emulator at run-time. %ROM% is replaced by the ROM path, within quotes (to properly handle spaces in paths).
Congratulations! You have now defined a collection. Now repeat this for all your platforms. Because XBMC currently only supports 100 settings per category, there is currently a hard coded limit of 15 collections.
Importing your collections
Now that you have defined your collections, you can launch the add-on again. This time, it will notify you that there are currently no games in the database, and ask you if you want to import them. Importing the games simply means reading the defined ROM folders, and saving them in the database.
This can also be done manually via the add-on settings, by selecting "Import collections" in the General settings category.
Optional: Scraping your collections
Once you have imported your collections, the add-on will ask you if you want to automatically scrape these for data. This means fetching stuff like images, genres, description, etc from the GiantBomb.com wiki. Note that this will take a VERY long time if you have many games! In general, it takes somewhere between 2 and 10 seconds per game. That amounts to quite a bit if you have many games.
Done!
You should now, barring any unforeseen catastrophe, be able to browse, launch, and play your games. Enjoy!
What do the settings do?
The general settings window currently contains 6 entries.
- Automatically mark as launched - Enabling this will mark games as played as soon as you launch them.
- Use scraped titles instead of filenames - Enabling this will make the browser use scraped titles for games where applicable. If disabled, filename (without file extension) will be used instead.
- Import collections - This manually imports new collections. This is what is run when the add-on asks you to import games if the database is empty.
- Clean database - This will go through all paths in the DB and remove non-existent games. For example, if you have deleted a game, the game will remain in the list until you have run this command, whereupon it will be deleted.
- Empty database - This will remove ALL games and scraped data from the database, including images.
- Scrape games - This will start scraping all currently unscraped games. This is what is run when the add-on asks you if you want to scrape your games after importing.
Thanks to
Props to malte for his Rom Collection Browser. Because of XBMCs aforementioned lack of documentation, I have more than once consulted his source code for working examples of things.
Also thanks to henks for letting me use his awesome photograph as a background (sorry for massacring it to fit).
Known bugs
- I don't recommend putting very large files in 7z archives. It will most likely crash with an OutOfMemory error when trying to extract it, and even if it doesn't, it will likely be very slow.
Possible future features
- Ability to filter and sort the list by platform/title/date/genre/etc. Also perhaps a proper "search" for game pad users.
- I want to implement some way to manually influence the scraping. That is, perhaps an entry via the context menu that would allow you to enter search terms manually, choose a game from possible matches, choose image yourself, etc.
- Related to above; I want to implement a way to import locally stored data about games (images, description, maybe release date and genres as well)
- I also want to implement some region matching. Currently when scraping, it will check for the first release image that matches the same platform, without caring for region. This means you'll sometimes end up with US/JAP covers even when you might want the European ones, for example.
- On the same note, I want to implement some support for sorting the ROMs in a GoodMerge 7z file by region and/or "confirmed working" when choosing which ROM to launch
Screenshots
An image showing the browse view
Another image of the browse view, this time showing a game that has been marked as "played".
An example of how the collections settings might be entered. Here you see an example of the "trickery" required to get PC games to work properly - the "ROMS" are simple windows shortcuts (.lnk files) to the game executables. When the emulator executable field is left blank, the shortcut will be launched, which in turn will launch the game.
- msderganc - 2010-12-16
Looks good! I'm going to try it tonight.
- nate1579 - 2010-12-16
Hey can you add different views for the box art
also when i set up it wont let me use box art i already have
- quakes - 2010-12-16
nate1579 Wrote:Hey can you add different views for the box art
I assume you mean different media views, such as poster wrap, media info, etc? I might add these in the future. It shouldn't be too hard.
nate1579 Wrote:also when i set up it wont let me use box art i already have
There is currently no (easy) way to use your own cover art. Sorry. Easiest fix is to upload it to the GiantBomb wiki - that way everyone can benefit!
- rausch101 - 2010-12-17
Can you give an idea of how accurate your GiantBomb scraper is? I have the full rom packs for each system so i'm wondering how many games you think will be misread (this seems to be an insurmountable problem for any of the rom scrapers)
- quakes - 2010-12-17
rausch101 Wrote:Can you give an idea of how accurate your GiantBomb scraper is? I have the full rom packs for each system so i'm wondering how many games you think will be misread (this seems to be an insurmountable problem for any of the rom scrapers)
I have spent a good deal of time making the scraper as accurate as possible, without getting false positives.
A basic overview of how the matching works:
- Make a game search on GiantBomb for the filename (sans extension)
- For each search result, try to match name according to following rules, and apply an "uncertainty" rating to each match (all rules are applied both to GiantBomb search results and the assumed title (filename) for purposes of matching):
- Match exact name (uncertainty 0)
- Match exact name or alias (GiantBomb specifies aliases for games as well as names) (uncertainty 1)
- Convert roman numerals to integers and match against name/alias (uncertainty 2)
- Make everything lowercase and match against name/alias (uncertainty 2)
- Turn names liike "legend of zelda, the" into "the legend of zelda" and match against name/alias (uncertainty 2)
- Turn names like "nightmare on elm street, a" into "a nightmare on elm street" and match against name/alias (uncertainty 2)
- Remove things in parenthesis and brackets, then trims (turns "Ice Climber (EUR) [!]" into "Ice Climber") and match against name/alias (uncertainty 2)
- Try to standardize subtitles by replacing " - " with ": " and match against name/alias (uncertainty 2)
- Try (temporarily) removing some special characters (!:-.) and match against name/alias (not permanently, as we wouldn't be able to detect subtitles later otherwise) (uncertainty 5)
- Remove all occurrences of 'The' and match against name/alias (uncertainty 6)
- Replace " & " with " and ", and match against name/alias (uncertainty 7)
- Remove subtitles entirely (everything after the first colon character) and match against name/alias (uncertainty 10)
- Remove all special characters again, permanently (!:-.) (uncertainty 15)
- If all else fails, simply assume that the fact that it showed up on a search at all has some chance of being a positive match (uncertainty 50)
- Match exact name (uncertainty 0)
- Filter out matches with an uncertainty that is above set threshold (currently set to 20 in scraper.py, which means that only matches that were matched by the last rule with an uncertainty of 50 will get removed)
- Sort matches by ascending uncertainty
- Proceeds with other parts of scraper, such as matching the correct platform, etc
This still fails for some games, though. If it fails, it is mostly because the game does not exist on GiantBomb, is named differently (this is what the matching rules try to alleviate), or is not marked to be for the particular platform.
If you or any one else find games that don't get any matches, but you think should (i.e. the game exists on GiantBomb), then please provide the (full) file name of the ROM, a link to the game on GiantBomb, and what platform the ROM is supposed to be for. The scraper can undoubtedly be improved further, but at this point it mostly requires examples of failed scrapes to do that.
- malte - 2010-12-17
This looks quite similar to what I do but with a different approach.
I hope it is ok for you that I post this here. I just thought it would be interesting to hold these two approaches against each other and give both of us the option to cherry-pick the missing parts.
Here is the logic that I wrote down for my scraper (in current RCB release it is implemented a bit different and leads to some mismatches, so this is the already improved version):
Code:
- create friendly gamename (remove extension, remove disk indicator (e.g. _Disk1), remove (...) and [...], replace "game, The" with "The game")
- start search with friendly gamename (returns 0-n results)
- compare friendly gamename with all found results (returns a ratio that indicates how good the result is matching)
- if ratio == 1.0 (perfect match)
--> return result
- else:
- check if game belongs to a sequel (friendly gamename MUST end with a digit or rome) --> (possible errors: "NFL '98" or "Project X")
- check if digit > 10 (to prevent "NFL '98" error) --> stupid but works in most cases
- replace rome with digit or vice versa and check again all search results
- if ratio == 1.0
--> return result
- else:
- check if sequel number is " 1" or " I"
- remove sequel number and check again all search results (e.g. "Breath of Fire I" -> "Breath of Fire")
- if ratio == 1.0
--> return result
- check if sequel numbers are equal (to prevent mismatches like "Breath of Fire I" == "Breath of Fire IV")
-> skip result if not equal
- check if ratio < fuzzyFactor
- check if game has alternate title (whole title must be found in result and result must contain "-" or ":")
-> if no alternate title is found check again for alternate titles with replaced digits or romes
-> check again if sequel numbers are equal (because "Alien Breed I" is contained in "Alien Breed II")
-> return result
- else:
-> skip result
- else:
-> return resultLooks like we are checking nearly the same things with the difference that I don't replace characters like "&" etc. because they will not lead to a ratio below 0.7 (or whatever is configured). The con on my side is that there might be slightly different names of games that will lead to a ratio above the configured one and with that to mismatches.
Things that you might miss in your logic (maybe just not written down):
- remove sequels numbers "1" or "I" to match "Breath of Fire I" -> "Breath of Fire"
- check if sequel numbers are equal (but if I read this correct you always check equality of the complete name without ratio/fuzzy)
- replace digits or romes when searching for alternate titles
Things that I could steal from your approach:
- Removing and replacing some characters to check if there is a perfect match before I start guessing names with the fuzzy factor
BTW: I hope there will be a nearly complete game db one day that can easily searched by crc values. This will ease up a lot (at least for the old systems).
- quakes - 2010-12-17
malte Wrote:This looks quite similar to what I do but with a different approach.Of course it's okay! I promote collaboration to the fullest extent.
I hope it is ok for you that I post this here. I just thought it would be interesting to hold these two approaches against each other and give both of us the option to cherry-pick the missing parts.
malte Wrote:Things that you might miss in your logic (maybe just not written down):This is true, I had not encountered this situation amongst my own collection, but it should be easily fixable.
- remove sequels numbers "1" or "I" to match "Breath of Fire I" -> "Breath of Fire"
malte Wrote:- check if sequel numbers are equal (but if I read this correct you always check equality of the complete name without ratio/fuzzy)As you said, I always check equality. The only "fuzzy" aspect to my implementation are the transforming rules.
malte Wrote:- replace digits or romes when searching for alternate titlesNot sure what you mean by this. For clarification, what you refer to as "alternate titles", I refer to as "subtitles" (though I'm not entirely pleased with this choice of words; is there an "official" word?).
malte Wrote:BTW: I hope there will be a nearly complete game db one day that can easily searched by crc values. This will ease up a lot (at least for the old systems).While this is a cool idea, one should remember that there is not a one-to-one mapping between ROMs and games; it is rather a many-to-one mapping (for example, I have 46 unique Contra ROMs for the NES, but they're still all Contra). In the case of packed ROMs, one would also have to unpack them before being able to calculate the CRC. Any bit rot or imperfect ROM rips would also lead to checksum mismatches.
Not that I'm saying it would be impossible.. just saying that there are a lot of considerations that must be made and impracticalities that must be resolved before it could be made reality.
Is anyone working on such a database at the moment?
- malte - 2010-12-17
Test
quakes Wrote:I think yours (subtitles) is better, but I don't think there is an official wording.malte Wrote:- replace digits or romes when searching for alternate titles
Not sure what you mean by this. For clarification, what you refer to as "alternate titles", I refer to as "subtitles" (though I'm not entirely pleased with this choice of words; is there an "official" word?).
What I mean is this:
- Lets say you have a rom file "Alien Breed 2"
- On giantbomb this one is called "Alien Breed II: The Horror Continues"
--> You dont't find this game with your step 3 because it has an additional subtitle and you won't find it with step 12 because without subtitle it is "Alien Breed 2" and not "Alien Breed II". I run into this with RCB atm thats why it came to my mind
quakes Wrote:Yes, I know, but tbh I thought it would not be more than a hand full of different rom files. In RCB I already look into the zip file and check the crc value of the first entry, so this works fine.malte Wrote:BTW: I hope there will be a nearly complete game db one day that can easily searched by crc values. This will ease up a lot (at least for the old systems).While this is a cool idea, one should remember that there is not a one-to-one mapping between ROMs and games; it is rather a many-to-one mapping (for example, I have 46 unique Contra ROMs for the NES, but they're still all Contra). In the case of packed ROMs, one would also have to unpack them before being able to calculate the CRC. Any bit rot or imperfect ROM rips would also lead to checksum mismatches.
But of course there are a lot of situations where all this won't work. All "newer" systems with large files for example (like PS1 etc.) would take too long to compute the crc values.
quakes Wrote:Not that I'm saying it would be impossible.. just saying that there are a lot of considerations that must be made and impracticalities that must be resolved before it could be made reality.Jas provides crc values for thevideogamedb, but there does not seem to be much progress in this project (he killed his PC). I am not sure if ghostelement is planning to add this to thegamesdb.net. There is also an interesting discussion at macscene.net. They also plan to provide an API with crc values and it looks quite promising.
- quakes - 2010-12-17
malte Wrote:What I mean is this:Ah, I see what you mean. I fix this by performing the transforming rules on both the filename and the search result.
- Lets say you have a rom file "Alien Breed 2"
- On giantbomb this one is called "Alien Breed II: The Horror Continues"
--> You dont't find this game with your step 3 because it has an additional subtitle and you won't find it with step 12 because without subtitle it is "Alien Breed 2" and not "Alien Breed II". I run into this with RCB atm thats why it came to my mind
Let us say my file is named "Alien Breed 2.7z". I will then first perform a search on giantbomb for "Alien Breed 2". Assume I get two results - "Alien Breed II: The Horror Continues" and "Alien Breed 3D II: The Killing Grounds". First I will attempt to match "Alien Breed 2" to "Alien Breed II: The Horror Continues", which will fail. I will then try to match against the alias, which will probably also fail.
Once I get to the third rule, I will convert roman numerals to regular integers in both the filename and the results. Thus, my new file name will be "Alien Breed 2" (unchanged, since it had no roman numerals), and my two search results will be "Alien Breed 2: The Horror Continues", and "Alien Breed 3D 2: The Killing Grounds". You see my point? This will continue until it reaches step 12, where it will remove the subtitle, thus making both the file name and the search result "Alien Breed 2", which will generate a match.
However, searching for "Alien Breed 2" on giantbomb only seems to give one result, "Alien Breed 2: Assault", which is an altogether different game. So it still wouldn't have scraped that properly, sadly.
One weakness is that it currently does not do any transforming of the file name before searching on giantbomb. Fixing this is on my to-do list...malte Wrote:Yes, I know, but tbh I thought it would not be more than a hand full of different rom files. In RCB I already look into the zip file and check the crc value of the first entry, so this works fine.Interesting. I will definitely keep an eye on it. The currently available and scrapable databases kind of suck, either interface-wise, structure-wise, or content-wise.
But of course there are a lot of situations where all this won't work. All "newer" systems with large files for example (like PS1 etc.) would take too long to compute the crc values.
Jas provides crc values for thevideogamedb, but there does not seem to be much progress in this project (he killed his PC). I am not sure if ghostelement is planning to add this to thegamesdb.net. There is also an interesting discussion at macscene.net. They also plan to provide an API with crc values and it looks quite promising.
EDIT: Took a closer look at the macscene thread, and that looks awesome. Will definitely be following that closely.
- msderganc - 2010-12-17
quakes -
Started using this last night and I love it! Much more intuitive than the other addons I've tried.
A couple of questions:
1. For homebrew games, where would I add custom artwork / writeup?
2. Games like Final Fantasy II for SNES aren't scraping properly. I had to change the name to Final Fantasy IV (the Japanese release) to get it to work. Would it be possible to have it pop up close matches if nothing perfect is found?
Thanks for a great addon!
- paendorz - 2010-12-17
Very nice addon.
Suggestion: The script should minimize xbmc when launching ROMs and maximize it afterwards. Otherwise some emulators will crash.
Another thing: Obviously the addon is based on Confluence and won't work properly on other skins. Any chance to fix that?
- Eldorado - 2010-12-17
Hmm.. so 3 different emu launcher apps, and all 3 using their own scrapers
Is it time to hopefully merge scapers to become one standalone addon that all can use?
- quakes - 2010-12-18
msderganc Wrote:Started using this last night and I love it! Much more intuitive than the other addons I've tried.Glad you liked it!
msderganc Wrote:A couple of questions:Currently, there is no easy way to use art or info besides scraping. Sorry about that. If you know SQL you can enter it manually into the sqlite database, but I realize this is incredibly cumbersome and not something one can expect an end-user to do. Do you have a preferred way or format to manually enter this information? Creating .xml or plaintext files manually, entering it from within XBMC using the context menu, something else...?
1. For homebrew games, where would I add custom artwork / writeup?
msderganc Wrote:2. Games like Final Fantasy II for SNES aren't scraping properly. I had to change the name to Final Fantasy IV (the Japanese release) to get it to work. Would it be possible to have it pop up close matches if nothing perfect is found?
Yes, I had this issue as well. The problem lies with the incredibly silly naming scheme that Final Fantasy has. I simply renamed my final fantasy games to avoid ambiguity (I prefer this over having two Final Fantasy II anyway). The one saving grace is that they're on different platforms, preventing accidental mismatches at least. Other than attempting to hard code "Final Fantasy II for SNES == Final Fantasy IV" (something I am loathe to do, but will consider...), there is little I can do. The problem could also possibly be fixed if you managed to add "Final Fantasy II" as an alias to the "Final Fantasy IV" entry on giantbomb. However, aliases can apparently only be changed by moderators, so you'd have to take it up with one of them.
I have thought about the pop-up thing as well, which is basically part of one of my listed "possible future features" - the ability to manually influence the scraping process. I don't think that including the pop-up during an automatic scrape of all games is a good idea, since you'd likely get hundreds of them if you had many games. However, a context menu entry to "manually scrape game" could be made to work this way.
paendorz Wrote:Suggestion: The script should minimize xbmc when launching ROMs and maximize it afterwards. Otherwise some emulators will crash.I saw some code to do this in maltes RCB, but didn't include it since none of my emulators exhibited this behavior. Do you have an example of an emulator (or several) that does this so I can check it out?
paendorz Wrote:Another thing: Obviously the addon is based on Confluence and won't work properly on other skins. Any chance to fix that?I haven't actually attempted to use the add-on together with another skin. Does it look horrible?
I don't actually use any resources from the installed skin; I have merely copy-pasted relevant images and (edited) XML files from confluence to the add-on, so technically I see no reason why it couldn't be used without confluence. I haven't tried though, so anything could happen...Eldorado Wrote:Hmm.. so 3 different emu launcher apps, and all 3 using their own scrapers
Is it time to hopefully merge scapers to become one standalone addon that all can use?
Perhaps so. It is an interesting proposition, at least. I think it will be somewhat hard, however. I assume not all of our add-ons have the same internal data structures, or are even interested in exactly the same data. As such, the standalone scraper would have to
- be able to scrape whatever kind of data our three different scrapers can separately scrape (one add-on may want videos, another may want cover art, a third may want fan art, etc...)
- make it available to other add-ons with a good API
- and thirdly, all our three add-ons would have to be rewritten to use this new standalone scraper rather than our own
This amounts to a lot of work in the end, on all our parts. Does the benefit outweigh the time spent? What if the standalone scraper actually performs worse than ones individual scraper?
- wimpy - 2010-12-18
3. <- this is a "major concern" if it's not being dealt with. this is by no means meant as critisism as i have huge admiration for all 3 addons being created. they are being created due to the love of emu/retro games so i'm all for it.
right now there are 3 addons using their own structure and scraper/database solutions and there are in addition 2 sites (thegamesdb.com and thevideogamesdb.com) which is trying to start a scraper service.
these scripts rise up due to the fact that xbmc doesn't have a games database yet. i havn't checked the trac about this so i might be wrong as it might be in the pipeline already(?). even so what i think needs to be done is to first work together creating a scraper site covering all aspects of a games database. (read: a thetvdb.com kinda site, but for games).
once that has been established the xbmc emulator "scene" have a firm platform and base to build from. i'm afraid until the first step of gathering and creating a scraper has been done we won't see a systematic games setup.
so i guess this is my "two cents"-speech about the "situation" (from my point of view) + a big "thank you" for the work being done in all aspects of the emulator-related stuff going on!