Kodi Community Forum
Fully automated BitTorrent download and sorting for XBMC integration? - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: Fully automated BitTorrent download and sorting for XBMC integration? (/showthread.php?tid=48865)

Pages: 1 2 3


- hal2100 - 2009-04-21

Hello,

I have something running like this.
It is not perfect, but a start. And most of the credits belong to other people, because I took the most parts from existing scripts.

Find them here:
http://depositfiles.com/files/90foyfwfn

Important part of rtorrent.rc:

# When the torrent finishes, it executes "mv -n <base_path> ~/Download/"
# and then sets the destination directory to "~/Download/". (0.7.7+)
#on_finished=test3,"execute=/home/gate/bin/process_torrent,$d.get_base_path=,stopped;d.set_directory=/home/gate/data/incoming/sort/"
system.method.set_key = event.download.finished,test3,"execute=/home/gate/bin/process_torrent,$d.get_base_path=,stopped;d.set_directory=/home/gate/data/incoming/sort/"

If anyone can repair the problem that my scripts create (mainly ghost files while moving the files) please post them back ;-)


- queeup - 2009-04-22

you dont need process_torrent file for finished torrents. look at my rtorrent.rc


- hal2100 - 2009-04-22

queeup Wrote:you dont need process_torrent file for finished torrents. look at my rtorrent.rc

I think you didn't read the process_torrent file completely.
The important part is the last line which is executed.


- bartdesign - 2009-04-22

A while ago I created a post at tvrename with a short howto on how to automate your downloads with SabNZBd/TVrename.

http://tvrename.com/bb/viewtopic.php?f=4&t=116&start=20#p1013


- PainToad - 2009-04-24

I don't get it, sabnzbd does a perfect job at organising TV shows by show/season out-of-the box.


- digitalhigh - 2009-04-24

I definitely need to recommend a program called TED (Torrent Episode Downloader). You tell it what shows to look for, then it grabs a schedule from tvguide and knows when to start looking for that particular show on the major sites. When found, you tell TED to save those eps to a particular folder.

You then configure Utorrent to download said files in that folder.

Third step is to get a program called therenamer (http://www.therenamer.com), and set that to run on a daily schedule. It checks folder X for new media files, and automagically renames them to a xbmc-friendly format and moves them to a specified TV directory.

Fourth...configure MIP to do it's thing on the same schedule, having it save nfo's and ep pics.

Fifth...tell xbmc to update the lib automatically.


Presto. Fully automated TV setup.


- ould - 2009-05-03

I have a pretty automated setup for TV Shows.

I use linux.

Basically I use a script called Flexget which I configure to watch RSS feeds from tvrss.net for the shows I like to watch. This gets run by cron, the torrents get downloaded to a different "watch" folder for rtorrent depending on the show name. rtorrent is setup to move the completed downloads to folders based on the watch folder.

For example:

Flexget runs and finds a new episode of "Lost"
It goes ahead and downloads the torrent file to a folder called "watch.lost"
rtorrent picks up a new torrent file in the watch folder and begins downloading
once complete it checks what to do with the file. It finds that it should be moved to folder "../../../TV/Lost"
xbmc is set to update library on startup

Obviously for each show I watch I have to configure Flexget and rtorrent but other than that it is completely automated.

Kevin


- rausch101 - 2009-05-19

Just wanted to give an update. I basically have my automation setup the same as digitalhigh explains 2 posts above this, except without TED.

I was using TED for a while, and it's great, but it didn't have every show I liked and I wanted to try to cut out the middleman. I am now using RSS feeds from tvRSS that feed into utorrent's automatic RSS Downloader (pretty much the same function as TED). From then it works the same as digital high.


- motyR - 2009-05-19

Hi,
dont know if it's goanna help anyone but i've written xchat and irssi scripts that runs on #torrentleech channel and snatch for the new torrents announcements based on user configurable filter and auto download them to some predefined watch dir.
http://www.torrentleech.org is a private members site so if u dont have an account then the scripts wont do u any good.
if anyone is interested please drop me a pm.

good day Smile

EDIT: ohhhh, it use curl to download the torrents so it runs on linux, dont know if it will work on windows.


here is a script that I run every hour - MDPauley - 2009-05-20

  • Look for any new BluRay rips and renames them to just name (year) (format).mkv
  • I download A LOT of TV shows and this step creates a new folder if needed and moves the show to that folder.
  • I query the video db and delete all TV Shows that aired more than 60 days.
  • I find shows that I have in both HD and SD and delete the SD version.
  • Lastly I update the video db.

Code:
<advancedsettings>
  <videolibrary>
    <backgroundupdate>true</backgroundupdate>
    <cleanonupdate>true</cleanonupdate>
    <hideemptyseries>true</hideemptyseries>
  </videolibrary>
</advancedsettings>

Code:
import os
import sys
import re
import shutil
from urllib import quote_plus
import datetime

print "Start Media Manager..."

def _mkdir(newdir):
    """works the way a good mkdir should :)
        - already exists, silently complete
        - regular file in the way, raise an exception
        - parent directory(ies) does not exist, make them as well
    """
    if os.path.isdir(newdir):
        pass
    elif os.path.isfile(newdir):
        raise OSError("a file with the same name as the desired " \
                      "dir, '%s', already exists." % newdir)
    else:
        head, tail = os.path.split(newdir)
        if head and not os.path.isdir(head):
            _mkdir(head)
        if tail:
            os.mkdir(newdir)

src_dir = 'F:\\My BR Rips'

for root, dirs, files in os.walk(src_dir):
    for name in files:      
        p=re.compile('(.+?).\[(.+?)-(.+?)-H.264\]-NewArtRiot.mp4$')
        match=p.findall(name)
        for moviename, year, format in match:
            if format == "HDDVDRip":
                new_format = "HDDVD"
            elif format == "BDRip":
                new_format = "bluray"
            else:
                new_format = format

            old_file = os.path.join(src_dir, moviename+' ['+year+'-'+format+'-H.264]-NewArtRiot', moviename+' ['+year+'-'+format+'-H.264]-NewArtRiot.mp4')
            new_file = os.path.join(src_dir, moviename+' ['+year+'-'+format+'-H.264]-NewArtRiot', moviename+' ('+year+') ('+new_format+').mp4')
            os.rename (old_file, new_file)

            old_dir = os.path.join(src_dir, moviename+' ['+year+'-'+format+'-H.264]-NewArtRiot')
            new_dir = os.path.join(src_dir, moviename+' ('+year+') (' +new_format+')')
            os.rename(old_dir, new_dir)

src_dir = 'F:\\TEMP'
new_dir = 'F:\\My TV Shows'

for root, dirs, files in os.walk(src_dir):
    for name in files:      
        p=re.compile('(.+?)\.S([0-9]+)E([0-9]+).+?\.mkv$')
        match=p.findall(name)
        for filename, season, episode in match:
            filename = filename.replace(".", " ")
            newdir = os.path.join(new_dir,filename)
            _mkdir(newdir)
            shutil.move(os.path.join(src_dir,name), os.path.join(newdir,name))

        p=re.compile('(.+?)\.S([0-9]+)E([0-9]+).+?\.avi$')
        match=p.findall(name)
        for filename, season, episode in match:
            filename = filename.replace(".", " ")
            newdir = os.path.join(new_dir,filename)
            _mkdir(newdir)
            shutil.move(os.path.join(src_dir,name), os.path.join(newdir,name))
           
        p=re.compile('(.+?)\.([0-9]+)x([0-9]+).+?\.avi$')
        match=p.findall(name)
        for filename, season, episode in match:
            filename = filename.replace(".", " ")
            newdir = os.path.join(new_dir,filename)
            _mkdir(newdir)
            shutil.move(os.path.join(src_dir,name), os.path.join(newdir,name))


#delete old tv shows
d = datetime.datetime.now() + datetime.timedelta(days=-60)
minusdays = d.strftime('%Y-%m-%d')

new_sql = "SELECT path.strPath AS path, files.strFileName AS filename FROM episode JOIN files ON files.idFile=episode.idFile JOIN path ON files.idPath=path.idPath WHERE episode.c05 <= '"+minusdays+"';"
records = xbmc.executehttpapi( "QueryVideoDatabase(%s)" % ( quote_plus( new_sql ), ) )

p=re.compile('<field>(.+?)</field><field>(.+?)</field>')
match=p.findall(records)
for path, filename in match:
    xbmc.executehttpapi("FileDelete(%s)" % (path + filename))
    
#let's delete duplicate shows that aren't in HD
new_sql = "select episode.c00, tvshowlinkepisode.idShow, count(*) as total from episode JOIN tvshowlinkepisode ON episode.idEpisode = tvshowlinkepisode.idEpisode group by tvshowlinkepisode.idShow, episode.c00 having count(*) > 1 order by total desc;"
records = xbmc.executehttpapi( "QueryVideoDatabase(%s)" % ( quote_plus( new_sql ), ) )

p=re.compile('<field>(.+?)</field><field>(.+?)</field><field>(.+?)</field>')
match=p.findall(records)
for c00, idShow, total in match:
    new_sql = "SELECT path.strPath AS path, files.strFileName AS filename FROM episode JOIN files ON files.idFile=episode.idFile JOIN tvshowlinkepisode ON episode.idEpisode = tvshowlinkepisode.idEpisode JOIN path ON files.idPath=path.idPath WHERE tvshowlinkepisode.idShow = '" + idShow + "' AND episode.c00 = \"" + c00 + "\" AND files.strFileName NOT LIKE '%%720P%%' LIMIT 1;"
    records = xbmc.executehttpapi( "QueryVideoDatabase(%s)" % ( quote_plus( new_sql ), ) )
    p=re.compile('<field>(.+?)</field><field>(.+?)</field>')
    match=p.findall(records)
    for path, filename in match:
        xbmc.executehttpapi("FileDelete(%s)" % (path + filename))
        print "Deleted " + path + filename
        
xbmc.executebuiltin('XBMC.updatelibrary(video)')

print "Media Manager is finished..."



- abauomy - 2009-05-26

Great work hear
ould: Best Idea
But i use xbmc as only GUI without Gnome
i want to know how to automate run rtorrent and flexget in background without interface

THX allot
Smile


- abauomy - 2009-05-26

Big GrinGreat work hear
ould: Best Idea
But i use xbmc as only GUI without Gnome
i want to know how to automate run rtorrent and flexget in background without interface

THX allot
Smile


- ould - 2009-05-26

abauomy Wrote:Big GrinGreat work hear
ould: Best Idea
But i use xbmc as only GUI without Gnome
i want to know how to automate run rtorrent and flexget in background without interface

THX allot
Smile

Hey abauomy,

Both rtorrent and flexget are command line apps and don't need a GUI at all. I do run a web interface for rtorrent called wtorrent which allows me to see what's happening remotely and it's a bit easier to work with then the text interface of rtorrent. I use screen to run rtorrent so that it runs hidden in the background and rtorrent can be brought up at any time by issuing the command "screen -r foo" foo being whatever you call your screen session. rtorrent is automatically started at boot time so it is always running. But as I said I generally only access rtorrent through the web interface I have setup as I find it more convenient.

Flexget is an python script. I called it using cron. I think I have it running hourly currently. Personally I run these apps on my main PC which I leave running all the time. My htpc is strictly for xbmc and only runs when I am using it but I don't see why you couldn't have these apps running in the same fashion on your xbmc box.

If you need more assistance let me know, you can pm me if you like. I can try to help. I am not always that good at explaining things and I don't use ubuntu so if you use that I may not be able to answer specific things.

Kevin


- xeonicxpression - 2009-07-13

Bringing back a slightly old thread. This thread is what actually got me interested in doing this. I ended up using SABnzbd+, Newzbin, and a small python program I wrote to download movies. For Tv shows I use NZB-tv and SABnzbd+.

SABnzbd+ - http://www.sabnzbd.org
NZB-tv - http://www.two-sided.com

Here's what the python program does:
  1. Goes to Filmjabber and reads their RSS feed of upcoming dvds and pulls next weeks releases.
  2. Then it looks up each one on IMDB and gets the rating.
  3. If its higher than my limit I pass the IMDB id to Newzbin with my custom search.
  4. I read that search's rss feed and see if it found anything.
  5. If it found some I pass the first results Newzbin report id to SABnzbd+.
  6. Finally check the response from SABnzbd+ to make sure everything went ok.

If anyone wants the program, put a post here and I MIGHT put it up. You'll need to get IMDBpy and Feedparser working. I really can't help you with that. I had a hell of a time getting IMDBpy working on my own, but then again I'm probably just stupid.

I would probably strip out the guts of the newzbin function and just leave the framework of it, just because it links to stuff that's against the DMCA. That part of the program is only 5 lines though. It's not like it would be hard to figure out. Hell the whole program is only around 90 lines total and I'm sure there are way more efficient ways of doing the stuff I did. I just hack away at it until I get things to work.

Keep in mind that you need a Newzbin account and they are invite only. You also need to buy credit once you have an account with them.

Edit: Oh yea, it uses os.system and I think that is windows only. I don't use Linux so I'm not sure.


- digitalhigh - 2009-07-13

xeonicxpression Wrote:Bringing back a slightly old thread. This thread is what actually got me interested in doing this. I ended up using SABnzbd+, Newzbin, and a small python program I wrote to download movies. For Tv shows I use NZB-tv and SABnzbd+.

SABnzbd+ - http://www.sabnzbd.org
NZB-tv - http://www.two-sided.com

Here's what the python program does:
  1. Goes to Filmjabber and reads their RSS feed of upcoming dvds and pulls next weeks releases.
  2. Then it looks up each one on IMDB and gets the rating.
  3. If its higher than my limit I pass the IMDB id to Newzbin with my custom search.
  4. I read that search's rss feed and see if it found anything.
  5. If it found some I pass the first results Newzbin report id to SABnzbd+.
  6. Finally check the response from SABnzbd+ to make sure everything went ok.

If anyone wants the program, put a post here and I MIGHT put it up. You'll need to get IMDBpy and Feedparser working. I really can't help you with that. I had a hell of a time getting IMDBpy working on my own, but then again I'm probably just stupid.

I would probably strip out the guts of the newzbin function and just leave the framework of it, just because it links to stuff that's against the DMCA. That part of the program is only 5 lines though. It's not like it would be hard to figure out. Hell the whole program is only around 90 lines total and I'm sure there are way more efficient ways of doing the stuff I did. I just hack away at it until I get things to work.

Keep in mind that you need a Newzbin account and they are invite only. You also need to buy credit once you have an account with them.

Edit: Oh yea, it uses os.system and I think that is windows only. I don't use Linux so I'm not sure.

SO...in order to use your setup, a person needs to get an invite to Newzbin. They then need to buy services to said site. Then, we'd need to rebuild your program that you're not even sure you want to post on here...just to make it work with the site we have to pay to use?

Not trying to be rude, but I personally don't see the point.

Maybe if it worked for a more universal OS, or worked with free sites, or hell...didn't require me to pay to get credit, it would be appealing. But I personally don't mind having to do a little legwork to get my Movies...