Kodi Community Forum
[RELEASE] Rom Collection Browser - Browse and launch emulator game ROMs - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Game Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=291)
+---- Thread: [RELEASE] Rom Collection Browser - Browse and launch emulator game ROMs (/showthread.php?tid=70115)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395

- Flomaster - 2010-09-24

I had n64 working earlier, but now its not launching my games Sad

here is my paste bin
http://flomaster.pastebin.com/tbCumqHy

also with mypen64plus I don't have sound in fullscreen only in small windowed screen


-=Jason=-

- hikaricore - 2010-09-24

I think I see part of your problem this launch line looks a bit suspect:
\/home\/xbmc\/\.xbmc\/addons\/script\.games\.rom\.collection\.browser/applaunch.sh /usr/games/mupen64plus --fullscreen --nogui --noask "/mnt/1tb/Games/Consoles/N64/Roms/MarioKart 64.z64"

Check your launch command and try again. Sound is most likely unrelated to either xbmc or this addon.
You should probably get your emulators working as needed before trying to configure them as I have mentioned before.

- malte - 2010-09-24

This may be a problem with escapeCommand = True.

You could try this instead:
Code:
<escapeCommand>False</escapeCommand>


- Flomaster - 2010-09-24

EDIT: Malte that escapeCommand fixed my problem now to find a working arg to get sound working for it

hikaricore Wrote:I think I see part of your problem this launch line looks a bit suspect:
\/home\/xbmc\/\.xbmc\/addons\/script\.games\.rom\.collection\.browser/applaunch.sh /usr/games/mupen64plus --fullscreen --nogui --noask "/mnt/1tb/Games/Consoles/N64/Roms/MarioKart 64.z64"

Check your launch command and try again. Sound is most likely unrelated to either xbmc or this addon.
You should probably get your emulators working as needed before trying to configure them as I have mentioned before.

this was a working config..... just out of no where while trying to get covers to be displayed in the 3D View it started not to launch the roms.

Code:
<RomCollection>
                        <name>N64</name>
                        <consoleName>Nintendo64</consoleName>
                        <romPath>/mnt/1tb/Games/Consoles/N64/Roms/*.zip</romPath>
                        <romPath>/mnt/1tb/Games/Consoles/N64/Roms/*.z64</romPath>
                        <romPath>/mnt/1tb/Games/Consoles/N64/Roms/*.v64</romPath>
                        <romPath>/mnt/1tb/Games/Consoles/N64/Roms/*.V64</romPath>
                        <romPath>/mnt/1tb/Games/Consoles/N64/Roms/*.rom</romPath>
                        <emulatorCmd>/usr/games/mupen64plus --fullscreen --nogui --noask "%ROM%"</emulatorCmd>
                        <mediaPath type="cover">/mnt/1tb/Games/Consoles/N64/Artwork/Cover/%GAME%.png</mediaPath>
                        <mediaPath type="action">/mnt/1tb/Games/Consoles/N64/Artwork/Action/%GAME%.png</mediaPath>
                        <mediaPath type="title">/mnt/1tb/Games/Consoles/N64/Artwork/Title/%GAME%.png</mediaPath>
                        <mediaPath type="cartridge">/mnt/1tb/Games/Consoles/N64/Artwork/Cart/%GAME%.png</mediaPath>
                        <mediaPath type="console">/mnt/1tb/Games/Consoles/N64/N64.png</mediaPath>
                        <mediaPath type="background">/mnt/1tb/Games/Consoles/N64/Artwork/Background/%GAME%.png</mediaPath>
                        <fileTypeForGameList>cover</fileTypeForGameList>
                        <fileTypeForGameListSelected>cover</fileTypeForGameListSelected>
                        <fileTypeForMainViewGameInfoBig>action</fileTypeForMainViewGameInfoBig>
                        <fileTypeForGameInfoViewBackground>background</fileTypeForGameInfoViewBackground>
                        <fileTypeForGameInfoViewGamelist>cover</fileTypeForGameInfoViewGamelist>
                        <fileTypeForGameInfoView2>action</fileTypeForGameInfoView2>
                        <fileTypeForGameInfoView3>cover</fileTypeForGameInfoView3>
                </RomCollection>

im going to add <escapeCommand>False</escapeCommand> and see if it makes a difference, however just today when first trying out the emulator via RCB it was able to launch games, but I had no covers so i went into the cofig and made some changes to the viewtypes only.

-=Jason=-

- Hudson_Hawk04 - 2010-09-24

I know this may be a little off topic but I was wondering if someone could rate the difficulty level of getting this to work correctly, or would it be wise to hold off for a bit longer?

- Flomaster - 2010-09-24

Hudson_Hawk04 Wrote:I know this may be a little off topic but I was wondering if someone could rate the difficulty level of getting this to work correctly, or would it be wise to hold off for a bit longer?

its dependent on your experience level.

at first glance I saw how it had to be configured and said FUCK THIS...but then was getting peeved I couldn't play the roms. so i said ok I'll dedicate some time and attempt to figure it out.

end result I have Nintendo, Super Nintendo, Nintendo64 all working and with audio.

next on my list is Sega, Playstation,


from a noobs perspective it's pretty difficult if you aren't willing to take time and read and try new things.

I personally love tinkering around and gettings things to work, some people can't do that.

bottom line if you want to learn a thing or two and don't mind trial and error give this a shot.

if you are short tempered with zero patients do not attempt this.


-=Jason=-

- hikaricore - 2010-09-24

If you stop and take time to read the reading the wiki and have a little patience it's super easy.
However if you're lazy and can't be bothered to do anything that involves minimal effort you should wait.

- Flomaster - 2010-09-24

hikaricore, I have read wiki, but some people learn from doing, and if you don't really understand some thing it makes it hard to grasp how it needs to be done. I am more a visualization type of person I also need to do it myself to really learn some thing.

BTW I have sound working by adding --audio to my args but it made it run in window mode and I had no way to make it fullsize. keyboard still not working for control keys, but I went into the mupen64plus video config and changed my window res to 1900x1080 and all is well.

-=Jason=-

- Hudson_Hawk04 - 2010-09-24

Hey guys thanks for your input I think I am going to try and get it to work but my list is pretty long, still tinkering with my media portal dvr and a few other things around the house along with my studies. Guess I just need to find the time or figure out a way to slow the rotation of the earth enough to add another day to the weekend. Thanks again guys.

synopsis files for c64, amiga and mame arcade - spyvsspy - 2010-09-26

Hi,

here are the promised synopsis files for some old home systems and the arcade machine emulator mame, for those who are interested in. I converted them from some well known Game Databases. I noticed that in the header.

mame_rom_synopsis_for_xbmc_rcb_addon.zip - 2.4 Mb

c64_rom_synopsis_for_xbmc_rcb_addon.zip - 2.1 Mb

amiga_rom_synopsis_for_xbmc_rcb_addon.zip - 7.9 Mb

amiga_rom_synopsis_big_for_xbmc_rcb_addon.zip - 19.5 Mb

The associated parserConfig.xml for the synopsis files is included.
The large Amiga synopsis contains all extra data, like instructions, walkthroughs and other stuff, but it's quite big for the parser, so I have generate a smaller one, which contains a hint to a local uri to the extra content.

I tested the import of the files in RCB and all these consoles working well on my ubuntu system. If you interested in my config.xml for RCB here it is:
config.xml.zip - 3.1 Kb

In future I will try the new xml format. I'm curious about it and also of the new scraper functionality.

Have fun!

p.s. one question:
I have problem with Non-ASCII characters, e.g. with the game name, programmer or developer properties, with RCB. Some of C64 or Amiga contains French or german special chars. Reading descriptions with Non-ASCII chars are working well. Do you think you will support Non-ASCII strings, like Charsets ISO-8859-15, for the properties too?

Here are the error message I get, if the rcb parser tries to read a developer string like Brøderbund:
Code:
RCB_WARNING: Parser complains about: 'ascii' codec can't encode character u'\xf8' in position 106: ordinal not in range(128)
RCB_ERROR: unhandled Error in onAction: 'ascii' codec can't encode character u'\xf8' in position 20: ordinal not in range(128)


- malte - 2010-09-26

Nice! I just downloaded and tested the Mame file and it works fine. May I add your links to the wiki?

The problem with the ascii-chars is on my list but I forgot in the last release. Will move it on top again.

About scraping:
Scraping functionality is nearly done (in a first experimental version). Right now I have a working config for thevideogamedb.com. But jas is still building up the database and there is not much content available atm. There will also be some changes to the API in the next weeks so this is just a very early test scenario.

I also played around with a config for mobygames.com and had some first positive results. But there are still some technical issues and anyway I am still waiting for a response to my request for scraping permission.

A beta version will be available in the next days if anyone is interested to help with testing.

- malte - 2010-09-27

There is a new version available with first experimental scraping functionality (not tested on all systems, not documented in the Wiki). It can download game descriptions and images from online sources. Parsing game descriptions can be done with the known flatfile or xml parserConfig. You can add multiple (online and offline) scrapers to one RomCollection to collect data from different sources. RCB will mix the content of all sources (first come, first go). Note that multiple scrapers are only supported in scenarios that use 1 description file per game.

So, this was the good news, now the bad one: This all has to be configured. I implemented this functionality with the old configuration scheme (config.xml + parserConfig) and plan to rework configuration with one of the next releases. If this all works correct in the future it will ease up collecting data but with this first shot it will again add some extras to RCBs configuration files.

This new release requires configuration changes in config.xml. Additionally the upcoming beta versions are not compatible with the old database structure. So you have to delete your current DB if you want to test the new version (even without scraping). (There will be an automatic DB update with the next stable versions, but it would be too much work to implement this for every test release)


Ok, how it works:

To keep your old config working, replace all:
Code:
<descFilePath>PathToData\Amiga\synopsis\%GAME%\synopsis.txt</descFilePath>
<descriptionParserFile>PathToData\Amiga\parserConfig.xml</descriptionParserFile>

with this:
Code:
<scraper parseInstruction="PathToData\Amiga\parserConfig.xml" source="PathToData\Amiga\synopsis\%GAME%\synopsis.txt"/>


To use http://thevideogamedb.com as online source, you should use this scraper configuration in config.xml (change the path to your parseInstruction):
Code:
<scraper parseInstruction="F:\Emulatoren\data\Testdata Current\Collection V9\VideoGamesDB - parserConfig.xml" source="http://thevideogamedb.com/API/GameDetail.aspx?apikey=%VGDBAPIKey%&amp;crc=%CRC%"/>
There are two placeholders in the URL: %VGDBAPIKey% is an internal thing and will be replaced with RCBs API key that I had to request at vgdb.com. Just leave it like this. %CRC% will be replaced with the crc value of the current rom file. This should be the most robust way to match your rom files with the available game description. Just to be complete: There are two more placeholders available: %FILENAME% and %FOLDERNAME%. They may be of use in other scenarios.

The parserConfig.xml should look like this (this file is also shipped with RCB, you find it in resources/database):
Code:
<parserConfig>
    <GameGrammar type="xml" root="Results/Game">        
        <Game>Name</Game>
        <Platform>Platform/Name</Platform>
        <Region>Region/Name</Region>        
        <Genre>Genre/Name</Genre>
        <ReleaseYear>ReleaseDate</ReleaseYear>
        <Developer>Developer/Name</Developer>
        <Publisher>Publisher/Name</Publisher>        
        <Filetypecover>Boxart/FullURL</Filetypecover>
    </GameGrammar>
</parserConfig>
This config tells RCB how to pick the data from the response message. There is only one "special" thing in it: "Filetypecover" grabs the url of an image. This image will be downloaded and used as filetype "cover". RCB will store this image at the path that you define as mediaPath type="cover" in config.xml. All other media types that are available can be handled the same way (Filetypetitle, Filetypeaction, ...).

(I am sorry to somehow reinvent the wheel with this scraping mechanism, but the XBMC scraper-xml seems not to be available from python and a complete implementation of this mechanism would have been a bit too much for this little project. And with the old flatfile parser I already went down the road half the way.)

Note that jas has announced to change the structure of thevideogamedb's xml response. So it may happen that the above parserConfig will not work anymore in some days. And please: if he has finished his API changes and built up the database, help him to add content to this site!

The complete RomCollection entry looks like this (ignore the emulatorCmd, it is just a placeholder):
Code:
<RomCollection>
    <name>Collection V9</name>
    <consoleName>Sega 32</consoleName>            
    <searchGameByCRC>True</searchGameByCRC>            
    <descFilePerGame>True</descFilePerGame>    
    <scraper parseInstruction="F:\Emulatoren\data\Testdata Current\Collection V9\VideoGamesDB - parserConfig.xml" source="http://thevideogamedb.com/API/GameDetail.aspx?apikey=%VGDBAPIKey%&amp;crc=%CRC%"/>    
    <emulatorCmd>%ROM%</emulatorCmd>
    <romPath>F:\Emulatoren\data\Testdata Current\Collection V9\Roms\*.zip</romPath>            
    <mediaPath type="cover">F:\Emulatoren\data\Testdata Current\Collection V9\Artwork\cover\%GAME%.jpg</mediaPath>    
    <fileTypeForGameList>cover</fileTypeForGameList>
    <fileTypeForGameListSelected>cover</fileTypeForGameListSelected>
    <fileTypeForMainViewGameInfoBig>cover</fileTypeForMainViewGameInfoBig>            
    <fileTypeForGameInfoViewBackground>cover</fileTypeForGameInfoViewBackground>
    <fileTypeForGameInfoViewGamelist>cover</fileTypeForGameInfoViewGamelist>
</RomCollection>

Please let me know what you think about this solution. I know, it adds some more configuration mess but I hope that I am quite flexible with adding new scrapers or reflect changes of existing ones. If this all is a bit more settled, I will add more documentation about how to add new scrapers and maybe we get some more online sources available in the near future. If you know more sites that allow scraping their content or even better provide an API to do this, please tell me.

With the next version I will also try to add a Google or Bing Search to get better results when searching by game name. Otherwise you have to adjust your filenames for every new online source with different naming.

EDIT:
I forgot to mention: Atm thevideogamedb.com has only content for Sega 32 and Virtualboy. So please don't fire your complete SNES-collection against the server. Just have an eye on jas' progress to see if he finishes more systems in the future.



Regards,
malte

- spyvsspy - 2010-09-28

Hello malte,
good work! Excellent!

malte wrote: <May I add your links to the wiki?> Of course!

I will try your new release as soon as possible, and give you feedback.

mobygames.com is a good idea for a scraper source.
Also http://www.arcade-history.com/ could be a good database source for a mame scraper, but I can't find out if they provide an api for their database.

In my opinion you approach is flexible enough to add new scrapers in future and add new properties.
Your new config syntax is easy to understand. Very nice!

I think the crc is the best unique value to identify a game.
I always use the crc for the gamename->romname mapping in all collections.

A question about the crc value. E.g. for mame they use the crc value of the complete zip archive for a game, not only the first entry.
The background maybe (?) that some databases support more than one kind of rom management, which allowed you to organize the roms in split, merge or no split/merge sets.
So it is possible that the same first file entry in the zip archive could be appear in other zip files of the mame rom collection, and the crc is not unique anymore.
Or I'm wrong? In the mame databases (also the c64 and amige gamebase) they only save the crc value of the whole zip archive.
(I changed it to the crc value of the first entry in my synopsis files)
This could be (not mandatory) a problem for scraping in a online database. In most instances, I think, the first value of the crc is enough.
Did you want to add support for your comma seperated rcb list of crc values for the scraper?

I take a look at jas api, maybe I could help him, to get some content. But the most work for a good game database, is to check some values, like
the correct developer/publisher, a unique schema of genre values and so on...
In my converted synopsis files I found some errors, and unfortunately I have not enough time to correct them all!

Have a nice week and keep it up!

p.s.
Here is a small new synopsis for the ColecoVision fans!
colecovision_rom_synopsis_for_xbmc_rcb_addon.zip - 92.1 Kb

- Flomaster - 2010-09-28

I need to work on getting the small details in my config like synopsis etc etc

a little off topic, anyone have a working copy of Mupen64plus 1.99.3 on a xbmc-live install?
if so check out this thread
http://forum.xbmc.org/showthread.php?tid=82022


-=Jason=-

- malte - 2010-09-29

@spyvsspy:
Added your links to the Wiki

Quote:mobygames.com is a good idea for a scraper source.
Also http://www.arcade-history.com/ could be a good database source for a mame scraper, but I can't find out if they provide an api for their database.
I already had a look at arcade-history.com but I think this site will be hard to scrape because they don`t use game names or other search strings in the url (afaics).

MobyGames did not respond to my request for permission until now and I am afraid they won't do anymore. Will wait and see what happens...

Quote:In the mame databases (also the c64 and amige gamebase) they only save the crc value of the whole zip archive.
(I changed it to the crc value of the first entry in my synopsis files)
This could be (not mandatory) a problem for scraping in a online database. In most instances, I think, the first value of the crc is enough.
I could add an option what you want to search for: first entry or complete archive. Anyway, I have to take a look at mame soon. I think there are a lot of special things that RCB does not cover atm.

Quote:Did you want to add support for your comma seperated rcb list of crc values for the scraper?

Right now, it is just a simple replacement in the url: crc=%CRC%. If there is a site with an API like vgdb.com they have to search in their db for this crc and return the correct result. Nothing more to do for me.

It may be an idea to search for crc values on scrapable sites to get better results. Atm I am planning to add a 2-step search for each scraper: 1. Do a search to get some links to scrape. This search should also be configurable, so that you can search via bing or other search engines or (maybe better) with the search tool that the scraped site provides. 2. Take these links and try to scrape them.

Quote:I take a look at jas api, maybe I could help him, to get some content.
Would be great. Did you see his thread here in the forum?

If it helps: The RCB parser is implemented as separate modules, so it would be easy to separate it from RCB. It just takes the descFile and the parserConfig as an argument and returns all results as dictionary. All you have to do is to write some python code to get these results into the database. If you are interested I could make a little zip file and a short usage example.


@Flo: sorry, no.