Hal_Emmerich Wrote:C) Bug: Interface Lockup when rapidly switching consoles
This is one I've noticed when switching between consoles with large databases quickly. If I flick from my NES to my SNES collection before the NES has finished loading, I -surmise- the SNES encounters a read error since the file is still in use. No log entries occur in the log, but the interface hangs until I restart the computer or sometimes wrestle control back with Control + Alt + Delete.
Never noticed this. But I will test in detail. Maybe my collections just aren't big enough.
Hal_Emmerich Wrote:D) Export/Import entire Database
This stems from the length of time it takes to import a database. I have about 8 rom collections of the magnitude mentioned above, and I -dread- having to reimport them all. It is literally over a hundred hours of importing thus far between NES, SNES, PSX, and the 166GB mame goodsets I still have -yet- to import makes me cringe. Please make some way to back up our files so that, in the event of a crash, we can rebuild our database without having to do all of this again.
There already are several ways to achieve this. Easiest way is just to make a file copy of the files MyGames.db and config.xml. You will find them in "XBMC/userdata/addon_data/script.games.rom.collection.browser". As long as you don't change the location of your emulators, roms and artwork, these files can be reused any time. (I have in mind to change RCBs import logic to use relative pathes so that you can even reuse your db in different setups. But this has to wait some time...)
2nd, during the first import RCB downloads all artwork that it can get and writes all info into .nfo files. RCB does not need to download artwork again, so even a full reimport of your collections will require a minimum of the time of the first import. And you can also choose to import the .nfo files instead of rescraping data again. This will reduce the time of an import dramatically. See related
section in wiki (ok, it is not much written there...).
3rd, you can already export your complete database to .nfo files (in case you did not write the files during first import or whatever). It is available via
context menu.
Hal_Emmerich Wrote:Possible Suggestions for Speeding up import:
- Index only the file names at first. Have the service run in the background and scrape behind the scenes. This gives the user control of their system back much faster and -surely- we don't need to watch as every file in a massive collection is scraped. Gradually update the database with each name as it runs.
You can already let RCB doing the whole import in background. See
wiki for more details. I have to admit this feature is still a bit experimental but last time I tested it everything worked for me.
Also I think the new missing info filters and scraping features that I mentioned in my previous post will help a lot in big collection scenarios.
Hal_Emmerich Wrote:I have noticed in reading this thread you don't seem to have access to large collections of roms. I would be happy to beta test solutions for large rom files, should you need someone with a huge list to test on.
Compared to most users I guess my collection is quite small. I also created a set of dummy files to simulate scraping and everything but all in all I guess I don't have more than up to 10.000 files to test (I already thought this was much). But as I see some users have single collections of this size.
I am always happy to get valuable input especially from users with different setups than my own one. When I work on new features I always try to release beta versions as soon as possible. Always announced here in the thread and available via RCBs project page. A link to the latest release post is always shown in first post of this thread.
Hal_Emmerich Wrote:I do however have to make a bug report for RCB however, as it seems my Playstation Rom lists are generating errors. Please find a snippet of the log below
Triggering event: Trying to pull up the Playstation Goodsets, a list of several thousand games. Sorting by category seemed to work, but the error occurs on the largest two categories, 'Action' and 'All'
So, this happens every time you try to show your Playstation roms? From the log it looks like there is a problem with one of the publishers. Maybe one with special characters in name? Can't really see whats causing the problem. But I am afraid a full log won't help me.
How big is your database file (MyGames.db)? Maybe you can zip it and send to my email? It would help me to debug this issue.
Hal_Emmerich Wrote:Actually I -think- the art thing would be as simple as not waiting for that 'ok' signal from the user to continue reading. Whether that was timing out the message after 10 seconds, or putting it on a separate thread, or just something like:
For Each Valid Extension File in Rom Path
- Find a game
- Import Game Details
- Was there error? Set Error to 1 if so
If Error
- {give error details here}
End if
If Error and 'User hit ok between last loop and now'
- Dismiss Error
End if
End Loop
It is already implemented kind of this way. When there is just an error with one game (game not found, no artwork found, ...) there is not even an error message and no interruption of the scraping process. You will just see these errors in xbmc.log and you can check the "missing artwork.txt" in RCBs userdata folder.
There are only two situations where RCB shows an error message and stops the import process: 1. The first 10 games... error message discussed before. 2. RCB is not able to save artwork to file. This is valid for ALL games with ALL artwork. So if I would continue scraping this means the user has to wait x hours to let RCB finish scraping and then sees that there is no artwork available for all games. So I think it MUST stop the import at this point.
Hal_Emmerich Wrote:Although what I would propose to speed up import immediately would be something like:
For Each Valid Extension File in Rom Path
Lookup Rom from Scraper
Record Rom rough name, Scraped=0
Loop
Launch RCB Scrape Service for background import
RCB service then has it cycle through the file, looking for the roms in the list that aren't scraped and processes them in the background. As it runs, it '1s' the Scraped variable in the file. Any file that has already been scraped is skipped unless 'Rescrape all roms' is checked. Getting the next unscraped rom would be as easy as returning the first entry on an SQL search where something like..
SELECT * FROM games.db WHERE {System = Recently Imported System & Scraped = 0}
Hmmm, I don't see that much benefit in splitting it up between client and service. I already thought about using a "scraped" marker to check if I should rescrape a game. I guess I will see how the new (re)scraping process will work and check if it makes sense to implement this. Thanks again for your suggestions!