[PhilippRaab]

Hello, I'm Philipp Raab and I am currently studying Software Engineering.
I am using XBMC/Kodi for several years now and I would like to contribute to this great project and consolidate my knowledge in C++/SQL.

I would like to implement the game library for RetroPlayer.

Kodi Game library
Name: PhilippRaab
Summary: Extend Kodi with a game library to store meta-data and save-games and add a game scraper add-on interface.
How will I achieve this: Find a suitable object relation mapper to store the game meta-data in the database. Port the existing scraping-add-on interface from the movies/tv-shows library to the new game library and provide a basic scraper add-on.
What will the project focus on: A clean storage solution for the game library and a add-on interface to get meta-data from scraping-add-ons.
Benefits: A native user experience to display meta-data and launch games for the new RetroPlayer.
Goals:

1. Find a suitable object relation mapper to use in Kodi
   
2. Extend the database system to enable the storage of game meta-data (~3 weeks)
   
3. Add a game library GUI to Kodi (~2 weeks)
   
4. Port the existing scraper system to the game library (~3 weeks)
   
5. Write a example scraper add-on (~2 weeks)
   

What does it touch in Kodi: Modifications in the database system to enable object storage. The RetroPlayer and the game API must be adapted to work with the games library.
Requirements: Knowledge of C++, SQL and ORM
Possible mentors: garbear

[garbear]

Hi Philipp,

I'll be the mentor for game-related projects. Once upon a time I created a game library (it wasn't pretty ), so I'm aware of the pitfalls you'll run into.

Primarily, the video and music libraries are in bad shape and a game library based on that code would be messy. We should base our game library on an existing ORM, and generalize the code so that someday the other libraries can be ported to our system. Fortunately, starting fresh means that the changes to the rest of the codebase will be minor.

Could you search around and evaluate several ORMs that could be used for this project? I'll weigh in on your choices and we can choose a suitable one for this project.

[PhilippRaab]

Hello garbear!

Okay now I understand better what you meant in the ideas thread.
I will edit my Proposal to include the ORMs and start searching

[topfs2]

May I suggest a NoSQL solution over traditional sql ORM.

From my small trials ejdb is a very nice document oriented database which to some extent mimics mongodb. Perhaps there exist an "ORM" which can unify them already though.

Another approach would be to simply use a key value store, and do a small indexer over that. Something based on kyoto cabinet would be awesome for this (ejdb is just that)

[garbear]

or you can do what I did. implement a nosql datastore on top of sqlite & mysql. This is not recommended but I had fun with it and learned a lot

the database abstraction for my experimental game library was optimized for reads. data was serialized using JSON or C BSON (like ejdb), and stored in a single table. the abstraction then used SQL indices and junction tables to create 1:N and N:N relationships that could be queried. read speeds rocked, but writing took ages because everything was done in serial. ACID properties were guaranteed, though.

a cool feature was, even though the data was denormalized, any fields could be normalized at runtime. you could pass the abstraction a field name, and it would modify the schema in real-time and populate new fields (or drop old ones) with data from the datastore. of course this took ages

[PhilippRaab]

I think it would be a good idea to make a object database interface to make it possible to use a different database solution like NoSQL in the future. But at the moment i think it would be best to use the existing database with a object mapper. When the other libraries are ported to the new object storage there is still the possiblity to switch to NoSQL with minimal code changes.

[RockerC]

Would it not be good idea to as a basis for this concert support m.savazzi's work on designing a new DB structure?

http://forum.kodi.tv/showthread.php?tid=200911

This could be the motivation needed for more developers to help drive that project to fruition, and not only for games.

Llooks like many good ideas specifically related to games library could probably be extracted from this old thread:

thread http://forum.kodi.tv/showthread.php?tid=40715


Sorry if off-topic but since kind of related, may also I ask happened to the Heimdall scraper API project for XBMC?

http://forum.kodi.tv/showthread.php?tid=134012

[PhilippRaab]

Would it not be good idea to as a basis for this concert support m.savazzi's work on designing a new DB structure?

http://forum.kodi.tv/showthread.php?tid=200911

This could be the motivation needed for more developers to help drive that project to fruition, and not only for games.

With an ORM there is no need for an explicit database design, the mapper would take care of the tables.

ORMs that i have found so far:

• ODB - only works with own compiler (mysql, postgresql, sqlite, oracle, mssql)
  
• Wt:bo - pure C++ code (no xml or macros) (mysql, postgresql, sqlite, oracle)
  
• LiteSQL - code generator based on xml files (sqlite, postgresql, mysql)
  
• yb-orm - code generator based on xml files, still under development (odbc)
  
• hiberlite - macro based (sqlite3)
  

Any suggestions for ORMs?

[garbear]

The way we should evaluate them:

• How much work would it be to integrate into our build system?
  
• How stable is the code?
  
• How does it scale? Can it handle large sizes and heavy IO?
  
• How intuitive is the ORM interface for unfamiliar developers tasked with maintaining the code?
  
• Does it connect to all the backends we wish to support?
  

The ORM will be abstracted so that it'll be possible to swap it out for another solution.

[topfs2]

I guess it depends a bit on what the project is meant to focus on. Is "get a game library in" or is it "Create a new stable library which expands to other media too". Doing both might actually be to much work, remember summer is short

If its the first then I think using whatever is easiest available, and that the student or the mentor is knowledgeable in. Be it a key value store or be it sqlite pure or be it an orm which is easy to compile in.

If its the latter I think investigating NoSQL solutions over ORMs should be done, yes you can use an ORM on a NoSQL but in general you don't need it as its document based (which is what ORM provides). If there is no suitable NoSQL then start looking at ORMs.

IMO the first thing is to find the focus of the project, its easy to let features and ideas get out of hand

[da-anda]

@DB structure and ORM
well, ofc ORM will handle the tables etc, but we could use the DB structure from this thread as a template for the objects. So having a generic asset interface that all aggregate roots implement etc...

[garbear]

Goals:

1. Find a suitable object relation mapper to use in Kodi
   
2. Extend the database system to enable the storage of game meta-data
   
3. Add a game library GUI to Kodi
   
4. Port the existing scraper system to the game library
   
5. Write a example scraper add-on
   

Can you attach timelines to your goals? It looks like there's three stages: the database, the gui, and the scraper system.

Speaking of scrapers, python might offer additional advantages over xml scrapers, in particular for games. I started a small library called PyRomInfo that can extract metadata from ROM headers, like the title, unique game ID, publisher name, etc. Extracting the title and publisher means that data is available offline. Unique game IDs mean 1:1 lookups at http://thegamesdb.net instead of string-matching filenames.

And if we're doing python scrapers, i know someone who might have a previous gsoc project to offer

Also, have you had experience with kodi's skinning engine? We'll probably base the gui off of videos or music so we'll have some skin code to copy from.

[PhilippRaab]

I guess it depends a bit on what the project is meant to focus on. Is "get a game library in" or is it "Create a new stable library which expands to other media too". Doing both might actually be to much work, remember summer is short

If its the first then I think using whatever is easiest available, and that the student or the mentor is knowledgeable in. Be it a key value store or be it sqlite pure or be it an orm which is easy to compile in.

If its the latter I think investigating NoSQL solutions over ORMs should be done, yes you can use an ORM on a NoSQL but in general you don't need it as its document based (which is what ORM provides). If there is no suitable NoSQL then start looking at ORMs.

IMO the first thing is to find the focus of the project, its easy to let features and ideas get out of hand

The focus of the project should be to get a game library for the Retroplayer to make it usable and provide a library view to attract users (and developers) to the project. To make a completly new and universal library system would be a project on its own.

Can you attach timelines to your goals? It looks like there's three stages: the database, the gui, and the scraper system.

Speaking of scrapers, python might offer additional advantages over xml scrapers, in particular for games. I started a small library called PyRomInfo that can extract metadata from ROM headers, like the title, unique game ID, publisher name, etc. Extracting the title and publisher means that data is available offline. Unique game IDs mean 1:1 lookups at http://thegamesdb.net instead of string-matching filenames.

And if we're doing python scrapers, i know someone who might have a previous gsoc project to offer

Also, have you had experience with kodi's skinning engine? We'll probably base the gui off of videos or music so we'll have some skin code to copy from.

Maybe we can integrate the extraction of the information from the rom headers directly into the scraper system and feed xml scrapers with that information. Do you think that would be possible?

I also think we should base the gui on the video/music gui. I only edited a few files of the skins, so my knowledge there is limited but when we reuse parts of the other libraries that would be doable i guess.

[RockerC]

Again suggest you look into http://forum.kodi.tv/showthread.php?tid=40715 as some GUI work for adding a game library XBMC was already done here:

https://github.com/xbmc/xbmc-antiquated/...amelibrary
https://github.com/xbmc/xbmc-antiquated/...melibrary3
https://github.com/xbmc/xbmc-antiquated/...melibrary2

Question is if you can still download and build spiff's old "gamelibrary" branches then why start from scratch with new GUI code if you don't need to?

Goals:

1. Find a suitable object relation mapper to use in Kodi
   
2. Extend the database system to enable the storage of game meta-data
   
3. Add a game library GUI to Kodi
   
4. Port the existing scraper system to the game library
   
5. Write a example scraper add-on
   

Can you attach timelines to your goals? It looks like there's three stages: the database, the gui, and the scraper system.

@PhilippRaab, tip; read and paraphrase garbear's example proposal outlined with timeline here http://forum.kodi.tv/showthread.php?tid=126210

[zag]

I'd concentrate on the game library only. Forget about scraping or other outlandish ideas for now

Too many GSOC projects fail because they are too wide. Concentrate on a specific goal and it should be a success.