[turok]

This thread is intended primarily for developers that want to contribute something substantial to the community. I've noticed that a lot of tools in this space try to handle many of the same tasks: downloading trailers and subtitles, generating NFOs, scraping various information from various sites, etc. That, coupled with the enormous following of users for these tools, shows that there is tons of interest in managing personal media libraries.

What I'm proposing here is a sort of 'ultimate' media downloader/manager that contains, at a high level:

• Automated downloader for any type of media (movies, tv shows, music, and perhaps apps?).
  
• Fully-functional media manager (like Ember). We're already going to have all the capabilities we need to do this, it's just a matter of providing a UI to do so.
  
• A solid codebase/framework that takes care of a lot of the common tasks for which developers are constantly re-writing code. Also, let's make it easy for other folks to come along and write classes for additional scrapers, providers, etc. Having both REST and command-line APIs would be good as well, so other applications can take advantage of what we build.
  

What do you all think? Are there any skilled developers out there that would like to help contribute to a project like this? Personally, I think this is the next logical evolution of this tooling space, so let's get it going sooner rather than later. Specific technologies, features, etc definitely warrant subsequent discussions. Just trying to assemble a team for now

-t

[connors]

What language did you have in mind?
Qt for cross-platform perhaps?

[branlr]

Wellll....

I would say this:

The existing apps are almost all open source. Therefore, it makes more sense to pull from them than to rewrite everything independently.

Sickbeard, for example, has been in alpha forever and has undergone tons of fixes to make it what it is now. Why lose all of that progress?

To me, it would mainly be an issue of getting them all to run on a single web server rather than running their own web servers, and then skinning them in a uniform way, with links that connect the interfaces.

However, these backends should remain fairly independent of one another, and the goal should be a project centered around merging these different backends into one interface, and pushing updates from their original parts to this merged interface in a systematic and periodic way...

For example,
We take snapshots of sickbeard, couch potato, etc at various stages of release, and develop a loose system for stripping away the interfaces and independent web servers and merging into this unified program....

So essentially, this project would not be responsible for any updates to these backends--just separating the backends and developing the new frontends.

[turok]

There are lots of considerations for something this big... a few of the things that come to mind are:

• Definitely cross-platform
  
• Web application for remote usage/administration
  
• APIs for easy integration with other applications
  
• Language with a solid OOP framework to make it easily extensible
  

Personally, I prefer the following approach, which takes all of the above into account:

• RESTful API -- because our implementation of a UI (the web application) is just one of many consumers of this API
  
• Dojo -- because a rich web application can be built with minimal code/time (if you know it). JQuery advocates just haven't taken the time to learn Dojo
  
• Java -- should speak for itself in terms of portability, OOP, etc.
  

I'm biased to this approach because I've used it extensively in the past, but I can say that there are no drawbacks that I ran into with this set of languages/tech. Also, I've written tons of utilities for this space in Java, so it just seems like a good language to use if we're going to be building a framework.

I think the domain model is important to nail down, so I've started something here that should give everyone an idea of my thoughts (note: you'll need to click on the image again and zoom in to see everything):
http://i.imgur.com/UtkHV.png

To explain it a little bit, we basically have the following classes:

• Entity -- a thing you download, such as a Movie, Album, or Episode
  
• Provider -- an NZB or torrent provider, such as NZBMatrix, nzb.su, thepiratebay, private torrent sites, etc
  
• Match -- a 'match' for an entity. Providers are searched for entities, and return a list of matches for an entity.
  
• Download -- a match download, to track things like date snatched, date processed, etc
  
• Notification -- a notification (EmailNotification, XbmcNotification, etc) that can be executed on specified events.
  
• Quality -- a quality definition that tells us how to identify which 'Quality' a given 'Match' is
  
• QualityProfile -- a profile that specifies things like: quality order, minimum/maximum sizes, and potentially a bunch of other settings (like provider order, notifications, etc) that can be utilized to help us manage matches/downloads
  

Then, we'll have things like a Downloader that can be subclassed as an NzbDownloader, TorrentDownloader, etc. Same for renamers, scrapers, adders (to add movies or albums automatically), data validations, the list goes on.

We have a chance to really consolidate a lot of the work that we've done (as a community) so far, and make it easier to handle changes and add enhancements in the future.

-t

[turok]

Wellll....

I would say this:

The existing apps are almost all open source. Therefore, it makes more sense to pull from them than to rewrite everything independently.

Sickbeard, for example, has been in alpha forever and has undergone tons of fixes to make it what it is now. Why lose all of that progress?

To me, it would mainly be an issue of getting them all to run on a single web server rather than running their own web servers, and then skinning them in a uniform way, with links that connect the interfaces.

However, these backends should remain fairly independent of one another, and the goal should be a project centered around merging these different backends into one interface, and pushing updates from their original parts to this merged interface in a systematic and periodic way...

For example,
We take snapshots of sickbeard, couch potato, etc at various stages of release, and develop a loose system for stripping away the interfaces and independent web servers and merging into this unified program....

So essentially, this project would not be responsible for any updates to these backends--just separating the backends and developing the new frontends.

At first, I was thinking along those same lines... MediaFrontPage (although I haven't used it) looks like an awesome UI that brings a lot of this stuff together. However, I think we're really doing ourselves a disservice by having so many independent applications out there, especially when the codebase for them is so similar.

That's the main goal of all this -- simplifying everything we currently have to be able to focus on more useful/important things in the future. I know it's a large effort initially, but it'll be so worth it in the long run. As I mentioned in my last post, I've written AutoMovies (getting close to a release) and lots of Java utilities to start this framework. Take a look at my domain proposal above, see what you think...

[thezoggy]

At first, I was thinking along those same lines... MediaFrontPage (although I haven't used it) looks like an awesome UI that brings a lot of this stuff together. However, I think we're really doing ourselves a disservice by having so many independent applications out there, especially when the codebase for them is so similar.

That's the main goal of all this -- simplifying everything we currently have to be able to focus on more useful/important things in the future. I know it's a large effort initially, but it'll be so worth it in the long run. As I mentioned in my last post, I've written AutoMovies (getting close to a release) and lots of Java utilities to start this framework. Take a look at my domain proposal above, see what you think...

the code base is not as similar as you may think.

[turok]

the code base is not as similar as you may think.

What i meant by this is that it has so much potential to work off the same base -- think about NZB search providers, notifications, etc... all these settings that are a decent part of any media downloading project. Take a look at the model (in my 2nd post) and see if you can find a scenario that it doesn't work for. Everything isn't included in there, but you should get the idea

-t

[RuudBurger]

I've been busy with rewriting CouchPotato to have a better base to work with. Using a plugin system for everything. For front-end I've developed a plugable system also based on Mootools.
Have a look at: https://github.com/CouchPotato/CouchPotato
It has plugins, api, full ajax front-end, notification system, versioned database.
I have this all working with Python 2.5 and wrapped with Titanium Desktop, making it cross platform including update functionality.
The same python code can be used on headless servers, using GIT as the main update mechanism.

I like the idea of one big ultimate media manager, but I think the ones that are out there are good, because they focus on 1 thing. Also, I have a completely different view on things then for example Midgetspy has with Sickbeard.
That doesn't mean it isn't awesome, but if I would make it, it would be completely different

The main thing I'm working on now with CP v2 is easily installable. Making it a real program and providing a setup wizard for users who don't know what to do. Also a plugin system, so Providers can be plugged, without changing any other code.

Like I said, I like the idea, but if your are gonna use Java I'm out, as I don't like it. And this is a hobby, and I'm not gonna use something I "hate" as a hobby .
I can live with Dojo (everythings better then jQuery) but I'm so in love with Mootools!

[branlr]

It has plugins, api, full ajax front-end, notification system, versioned database.
I

This is what I mean... why throw all of that away? I simply don't understand the point...

Like RuudBurger said, these programs all work so well because they focus. Usually, I'm inclined to agree with the idea of small applications that do one thing well--and it applies here too.

The goal, if you ask me (which no one did haha), should be converting all of these independent programs to backend components, and doing so in a way that allows us to incorporate all of their updates from periodic snapshots.

From the sound of it, Ruudburgers design would allow this to be done very easily... perhaps not all of the other projects would be the same--but could we not then focus development efforts on those projects to give them the functionality that would serve this purpose before undertaking this task?

As it stands, realize that you COULD actually let these programs run as-is and just make a unified interface for them. I know it'd feel kind of hacked together, but I'd rather use sickbeard than some lite, untested rewrite version of it that will take far too long too catch up in maturity...

[j1nx]

Oh, oh oh oh oh oh oh Yiihaaaa

I will see a very bright future.

[turok]

Ruud, good stuff, glad to see a continuous effort going into CP! I did notice one thing in couchpotato/core/providers/nzb/newznab/main.py -- on line 93 where it uses int(size). Just wanted to make sure you tested this on 32-bit systems, as this would (or might) have a limit of 2x10^9 (limiting the 'file size' to 2.1 GB). I guess a simple thing to make sure, would be to change it to long(size).

I'm not necessarily married to the idea of using Java, it's just that personally, I like its architecture and the fact that it's more work to 'set up' a project, but easier to maintain and change down the road.

There's plenty of room for UI development in this project as well. As far as a JS toolkit, I've never used mootools, but it looks nice!

I probably have a few weeks before things get busy for me, so I'll go see if I can get something concrete put together as a proof-of-concept. I absolutely love working on stuff like this (after all, automating manual tasks is the reason computers were invented!) and I'm hoping the community can benefit from this as well

-t

[Jaken]

I like this idea a lot. A combined effort sounds great.

I'm no programmer guru in any way. But this is the kind of project I would be willing to contribute to.

On a side note i use CP and love it. I would not mind if the software were to end up like CP on steroids. (With more management possibilities and deeper interaction with XBMC)

[RuudBurger]

I did notice one thing in couchpotato/core/providers/nzb/newznab/main.py -- on line 93 where it uses int(size). Just wanted to make sure you tested this on 32-bit systems, as this would (or might) have a limit of 2x10^9 (limiting the 'file size' to 2.1 GB). I guess a simple thing to make sure, would be to change it to long(size).
-t

Python auto-converts it to long

[turok]

I like this idea a lot. A combined effort sounds great.

I'm no programmer guru in any way. But this is the kind of project I would be willing to contribute to.

On a side note i use CP and love it. I would not mind if the software were to end up like CP on steroids. (With more management possibilities and deeper interaction with XBMC)

I've only used basic/moderate capabilities in XBMC... what kind of interaction were you thinking (anything specific)?

Python auto-converts it to long

Awesome, I figured there was some python magic taking care of it somewhere.

[Jaken]

I've only used basic/moderate capabilities in XBMC... what kind of interaction were you thinking (anything specific)?

I'm not sure the manager would NEED to interact with xbmc but i guess it could give us some nice features a little easier.

First of all would it be great if the media manager part of the program could change the xbmc database. (Through json api)? This way we would not need to scrape to nfo files. We could manage thumbnails, covers, fanart, subtitels and such very neatly. We can also edit movie titels, playlists, sets, etc.


Also there could be a lot of nice features regarding the downloader part of the program. Some ideas below.
Automagicly upgrade movies from SD to HD quality.
Watch your current TV-shows for new episodes, and download them.
From xbmc set movies to download that you find in apple trailers for example.
Find new music from your favourite artists.
Sync your library with the library of a friend to make sure you got the same movies.


There are probably quite a lot more than this that could be added if the media manager and XBMC were to interact. These were just some ideas that came to mind.