Kodi Community Forum

Full Version: SageTV Integration - Movies and TV Shows archiving compatible with XBMC
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
IMPORTANT: Since SageTV has been purchased by Google and its future is unknown, I have abandoned SageTV and moved to ForTheRecord. I encourage you to check out the ForTheRecord PVR integration script I have made available

I've been using Sage TV as my PVR, which is a great piece of software, however, the front end and media library are not nearly as polished and XBMC's.
It also doesn't support as many formats as XBMC, and struggles with hardware accelerated playback on my ION HTPCs.
For that reason, I want to continue to use XBMC's excellent library to manage and playback my TV Shows and Movies.

So I created a script that does the following:
  • Integrates XBMC video library and SageTV by archiving SageTV recordings in a format that XBMC understands and can scrape.
  • Can add shows to XBMC's library even if they don't have known Season/Episode numbers.
  • Optionally compresses the SageTV recordings w/ options.
  • Optionally deletes the original recording once the archived recording is created.
  • Optionally creates a symbolic link with an XBMC-friendly name that points to the original recording (allows XBMC to use its library and SageTV to both access the same video without conflict.)
  • Optionally extracts Closed Captions from SageTV recordings and saves them as .srt's for XBMC subtitle compatibility.
  • Optionally sends notifications on success/errors.
  • Optional automatic updating of XBMC Library via XBMC's JSON-RPC interface.
  • many more features (See comments in Config.xml)...

This script will archive your Movies and TV Shows like so, where DROPBOX is a folder you define:
DROPBOX\Movies\[Movie Title] (Movie Year)
DROPBOX\TV Shows\[Series]\[Season]\[SeriesName.SxxExx - Title]

XBMC likes this format, its easy to understand, and all your content will be available in XBMC's library once you add the folders to XBMC's sources and set the content type.

******************************************************
Download Here - v1.4.1
To install, read the readme.txt. Also, see the FAQ in the 2nd post below.
Post problems/feature suggestions/etc to this thread and I will try to help.
******************************************************

MORE DETAILS:
Current Features:
  • All options are configurable in the Config.xml file.
  • Filter which shows/movies you want to process by title/extension/highDefition. Also a wildcard is available for TV Shows & movies.
  • Option to keep or delete original recording on a show-by-show basis.
  • Comskip's .edl files can be copied with the video and changed to what type you want (commercial break/cut/mute etc.)
  • Support for SageTV's .properties files.
  • If .properties file does not exists, season/episode info is looked up on TheTVDB.com.
  • Support for multi-part episodes.
  • E-mail notifications on problems and successes.
  • Sync watched statuses and resume points between XBMC and SageTV
  • Optional compression via 3rd party program such as Handbrake.
  • Limit the number of simultaneous compressions allowed to ease the CPU.
  • Compression can be verified. And if it failed, you can optionally use Sage's internal compression to compress the video instead.
  • Detection of hung compression based on CPU usage and Time Encoding.
  • Closed Caption extraction.
  • Symbolic links to "redirect" XBMC to the original recording, yet still use it's library features (must be using Vista or better).
  • Option to "prefer original" recording which allows XBMC to use the original recording while it's available, and switch to the compressed recording when the original is no longer available.
  • Log file expiration after user-defined number of days.
  • Option to "manually archive" shows that don't have season/episode numbers available. They will be added to XBMC's library as "special" episodes, using meta-data from Sage.
  • Option to wait X minutes before processing a video (give other process time to finish).
Known Limitations:
  • Windows Only
  • No GUI
  • Only tested with SageTV v7 (but believed to work with v6)
To Do List:
  • Allow automatic deleting of old videos based on user defined directory size limits.
Change Log:
  • v 1.4.1 - Fixed thumbnail hash generation for hash's that start with zero. Improved TVDB lookup matching. Other minor improvements.
  • v 1.4 - Automatic cleanup of video artifacts, auto-generate thumbs when none are available on thetvdb.com, can propagate symlink deletes to the original recording, options to force using current date as 'Aired Date' for manual archivings, Sage<-->XBMC Synchronization, handle > 99 episodes per season, many bug fixes
  • v 1.3.2 - Fixed manual archiving (again). Now everything should work smoothly for manual archives.
  • v 1.3.1 - Spaces are now allowed in the folder name where this program runs. Fixed bug where manual archive (special) episodes would be incorrectly marked as multi-part episodes by XBMC's scanner.
  • v 1.3 - Added ability to archive videos that dont have season/episode numbers by using meta-data from SageTV and .nfo files. Option to extract thumbnails for manually archived videos (thumbnails will never be inside a commercial break, if you use .edl files). Added optional wait time before processing a video. Improved Config.xml to be more user friendly. Made more settings user-configurable in Config.xml. Added cleanup to keep tracking files from getting large.
  • v 1.2 - Added "redirection" via symbolic links. Added "prefer_original" option for central MySQL XBMC libraries. Added automatic updating of XBMC via JSON-RPC interface. Added subtitle extraction via CCExtractor. Set limit on time between identical TVDB queries (60 min). Improved caching. Automatic deleting of old log files. Improved multi-part episode handling. Automatically adds tvdbid to Showfilter when a match is successful. (Prevents unnecessary re-queries to get the ID every time). Added local_equiv attribute to match UNC paths to their local equivalent.
  • V 1.1 - Fixed bug where some .edls were not being read by XBMC. Fixed multi-episode naming for episodes that use external compression.
  • V 1.0 - Initial release
Upgrading:
Upgrading to 1.4.1 from 1.4: replace the dist/SageXBMC.jar with the new one. That's it.

Upgrading to 1.4.1 from an earlier version (1.3.x or earlier)
If you doing a fresh/new install, you do not need to follow these steps. Just read the readme.txt

  1. Download 1.4 and extract it next to your 1.x installation folder.
  2. If you have this running as a scheduled task, disable the task.
  3. From 1.x, copy and replace these files in 1.4
    Code:
    Essential:
    res\processedFiles.txt
    res\xbmcFileSources.txt
    res\currentEncodes.txt

    Optional:
    res\cachedMediaTypes.txt
    res\sentNotifications.txt
    res\tvdbQueries.txt
  4. Open Config.xml in both folders using Notepad++ (or your favorite XML editor).
  5. Copy any settings you want to keep from 1.x to 1.4. Notice that the layout of Config.xml has changed a bit, so watch what you copy/paste.
  6. Adjust your scheduled task so it runs in the new 1.4 folder (or change your 1.4 folder to the name of your old 1.x folder) and re-enable the scheduled task.
Post any problems/questions to this thread.

FAQ:
I'm using the redirect/symlink feature, but the vidoes are not playable in XBMC. What's up?
If you are trying to playback symlinks that are on a different computer, you have to execute this command on the machine hosting the symlink as well as any computers that access the symlink:
Code:
fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1
This allows the symlink(s) to be accessed by any computer.

I am getting errors in the log about invalid XML from the Sagex API.
#1 Make sure the sagex api is installed. You can check for it at http://YourSageServer:8080/sagex/api It can be installed under the plugins menu in SageTV v7. For v6, get it here http://forums.sagetv.com/forums/showthread.php?t=36044
#2 Your <SageRecordingDirs> in config.xml must exactly match the recording directories set up in Sage. I recommed using UNC (\\server\share) paths.
This looks excellent, way more advanced than the hacked up plugin I put together, I will have to try it out later tonight
Great work on this script! It's nice to be able to access my SageTV recordings in the XBMC native library. XBMC definitely needed a way to organize recorded shows.

Right now, I'm not compressing my HD files, so I'm wondering if you could work in a way to just make a symbolic link in the drop folder to the original .mpg and .edl files with filenames that play nice with XBMC. That way, if a user wanted to just watch the original files, there would be no need to rewrite the file in the drop folder.

I like aaronb's plugin and use it also, but like how your script deals w/ .edl files and how it puts the shows in the native library.
jessescott Wrote:Great work on this script! It's nice to be able to access my SageTV recordings in the XBMC native library. XBMC definitely needed a way to organize recorded shows.

Right now, I'm not compressing my HD files, so I'm wondering if you could work in a way to just make a symbolic link in the drop folder to the original .mpg and .edl files with filenames that play nice with XBMC. That way, if a user wanted to just watch the original files, there would be no need to rewrite the file in the drop folder.

I like aaronb's plugin and use it also, but like how your script deals w/ .edl files and how it puts the shows in the native library.

I'm one step ahead of you. I've done quite a bit of development since my last post and what you suggested is working. I'm going to test over the holidays and post an updated version here if all is well.
bradvido88 Wrote:I'm one step ahead of you. I've done quite a bit of development since my last post and what you suggested is working. I'm going to test over the holidays and post an updated version here if all is well.

Nice, I'll keep an eye out for the updated version.
Been busy. 2nd son was just born a couple days ago, so I'm not really keeping up with this stuff. But I still plan to get an updated version posted for those interested.

I just have to modify the code so the settings are read from the config file instead of hard-coded.
bradvido88 Wrote:Been busy. 2nd son was just born a couple days ago, so I'm not really keeping up with this stuff.

Congrats on the new kiddo. Whenever you get around to posting the updated version, it will be helpful to anyone who wants to organize SageTV recordings on XBMC.

No rush needed here, though. I upgraded my HTPC to Windows 7 over Christmas and I'm toying with the idea of switching to Media Center for recording.
Ok, so far not much luck getting this to work for me.

I've running SageTV v7 (latest release 23) on a Windows Home Server (2003). I'm running XBMC on a Vista Machine and have successfully added Movies and Music manually (through the network to the WHS.

I'm trying to get this to run and it ends successfully, but skips all recordings because they are not videos? Here is the output from the cmd line run:

C:\XBMC\dist>"C:\Program Files (x86)\Java\jre6\bin\java.exe" -jar "SageXBMC.jar"
01/09/2011 01:57:55 PM INFO Getting config from XML file: C:\XBMC\config.xml
01/09/2011 01:57:55 PM INFO Checking for external encodings that might have finished
01/09/2011 01:57:55 PM INFO 0 current encodings found, skipping
01/09/2011 01:57:58 PM INFO
Done processing 1 Sage Recording Directories: [\\WHEELHOUSE\Videos\SageTV\]
1776 files scanned
0 files successfully processed
0 files can't be processed right now
1424 files skipped because they are not Videos
0 files skipped because the dont match any filters
0 files skipped because they've previously been processed
01/09/2011 01:57:58 PM INFO Done...


I have confirmed that their are mpgs in the \\Wheelhouse\Videos\SageTV directory, so I'm not sure what to try next.

My Config file contains two shows:
<add extension="mpg">GoodEats</add>
<add extension="mkv">GoodEats</add>

and I have both mpgs and mkvs of this program in the folder.
skacey Wrote:Ok, so far not much luck getting this to work for me.
...
I'm trying to get this to run and it ends successfully, but skips all recordings because they are not videos?

It currently only recognizes .mpg .ts .mkv .avi as video extensions.
Change your logging level to 4 and run it again and then copy the log to a pastebin for me. That'll tell us what's really happening.
Ok, not sure how to do the pastebin, but here is the first portion including the first two found mpg files. It seems that the problem is Sage not recognizing the files.

I double checked to make sure Sage API is working and it is (version 7.0.23.3) - I also pasted the first mpg's URL into my browser to see what it returns and I got the following:

Quote:body { position:relative; top:0px; margin-top: 33px; }

Here is the first part of the verbose logging:
Quote:C:\XBMC\dist>"C:\Program Files (x86)\Java\jre6\bin\java.exe" -jar "SageXBMC.jar"
01/10/2011 06:41:13 PM INFO Getting config from XML file: C:\XBMC\config.xml
01/10/2011 06:41:13 PM DEBUG Logging level set to: 4
01/10/2011 06:41:13 PM DEBUG E-mail notification level set to: 2
01/10/2011 06:41:13 PM DEBUG NotifyOnSuccess set to: false
01/10/2011 06:41:13 PM DEBUG Notification expiration hours set to: 12.0
01/10/2011 06:41:13 PM DEBUG Found sage recording directory: \\WHEELHOUSE\Videos\SageTV\
01/10/2011 06:41:13 PM DEBUG Found Dropbox: \\WHEELHOUSE\Videos\Recorded TV\
01/10/2011 06:41:13 PM DEBUG Found TheTvAB Api Key: 29C444BCE06CC385
01/10/2011 06:41:13 PM DEBUG Sage Server URL found: http://wheelhouse:8080/
01/10/2011 06:41:13 PM DEBUG Wait for Comskip's .edl file = true
01/10/2011 06:41:13 PM DEBUG EDL Type set to 3
01/10/2011 06:41:13 PM DEBUG Found show filter: GoodEats for .mpg files. KeepOriginal=true, ExternalCompression=None, HDOnly=false
01/10/2011 06:41:13 PM DEBUG Found show filter: GoodEats for .mkv files. KeepOriginal=true, ExternalCompression=None, HDOnly=false
01/10/2011 06:41:13 PM DEBUG Found show filter: ALL_MOVIES for .mpg files. KeepOriginal=true, ExternalCompression=None, HDOnly=true
01/10/2011 06:41:13 PM INFO Checking for external encodings that might have finished
01/10/2011 06:41:13 PM INFO 0 current encodings found, skipping
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\30Days-ProChoiceProLife-3974296-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\30Rock-TheBubble-1426156-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AccessHollywood-1197429-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AlfredHitchcockHour-1676779-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AllMyChildren-1710961-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AllMyChildren-1710962-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".edl"): \\WHEELHOUSE\Videos\SageTV\AllMyChildren-4692570-0.edl
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".log"): \\WHEELHOUSE\Videos\SageTV\AllMyChildren-4692570-0.log
01/10/2011 06:41:14 PM DEBUG Checking for cached media type at: C:\XBMC\res\cachedMediaTypes.txt
01/10/2011 06:41:14 PM DEBUG Looking up media type by retrieving XML from Sage at: http://wheelhouse:8080/sagex/api?c=GetMe...2570-0.mpg
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(null): \\WHEELHOUSE\Videos\SageTV\AllMyChildren-4692570-0.mpg
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".txt"): \\WHEELHOUSE\Videos\SageTV\AllMyChildren-4692570-0.txt
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AmericasGotTalent-1406884-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AmericasGotTalent-1637027-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AmericasGotTalent-1637028-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AmericasGotTalent-1637029-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AmericasGotTalent-1637030-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AmericasGotTalent-1645101-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AmericasGotTalent-1651210-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AmericasGotTalent-1671346-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\AmericasGotTalent-Recap-1688453-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\Babar-KingsoftheCastle-4672299-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\BeachParty-4567765-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".properties"): \\WHEELHOUSE\Videos\SageTV\BeyondBeliefFactorFiction-BatteredDollPokerJustice-3061093-0.mpg.properties
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".edl"): \\WHEELHOUSE\Videos\SageTV\Bones-TheBlondeintheGame-4674783-0.edl
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(Skipping, not a valid video extension (mpg|ts|avi|mkv) -- ".log"): \\WHEELHOUSE\Videos\SageTV\Bones-TheBlondeintheGame-4674783-0.log
01/10/2011 06:41:14 PM DEBUG Checking for cached media type at: C:\XBMC\res\cachedMediaTypes.txt
01/10/2011 06:41:14 PM DEBUG Looking up media type by retrieving XML from Sage at: http://wheelhouse:8080/sagex/api?c=GetMe...4783-0.mpg
01/10/2011 06:41:14 PM DEBUG Skipping. This file is not recognized as a SageTV file(null): \\WHEELHOUSE\Videos\SageTV\Bones-TheBlondeintheGame-4674783-0.mpg
Follow up - The Sage API does seem to work, I tried another command and got the following:

Command:
http://wheelhouse:8080/sagex/api?c=GetFi...2570-0.mpg

Returns:
mpgbody { position:relative; top:0px; margin-top: 33px; }

so I'm assuming the mpg is the return and the rest is formatting?
skacey Wrote:Command:
http://wheelhouse:8080/sagex/api?c=GetFileExtensionFromPath&1=\\WHEELHOUSE\Videos\SageTV\AllMyChildren-4692570-0.mpg

Returns:
mpgbody { position:relative; top:0px; margin-top: 33px; }
That is not expected output from the sagex API. It appears your output is getting mashed with some style information.
Are you just browsing to that URL in Internet Explorer? What does opening this URL give you?
Code:
http://wheelhouse:8080/sagex/api?c=GetMediaFileForFilePath&1=%5C%5CWHEELHOUSE%5CVideos%5CSageTV%5CAllMyChildren-4692570-0.mpg
Make sure that your directories in the Config file are the same as the recording directories in Sage. For example, dont' use a local path "C:\Sage\Recordings" in SageTV and a UNC path "\\server\sage\recordings" in the config file. The paths have to be equal in order for the Sagex API to handle them.
Navigate to your Install Directory/res/temp.xml and open that file in notepad. Then past the contents here.
That will tell us exactly what the program is seeing.

You should get something like this:
Code:
<MediaFile>
  <MediaFileFormatDescription>
  <![CDATA[ MPEG2-PS[MPEG2-Video 16:9 720p@60fps, Dolby Digital/192Kbps@48kHz Stereo eng]
  ]]>
  </MediaFileFormatDescription>
  <IsLocalFile>true</IsLocalFile>
  <IsLibraryFile>false</IsLibraryFile>
  <IsCompleteRecording>true</IsCompleteRecording>
  <IsDVD>false</IsDVD>
  <IsBluRay>false</IsBluRay>
  <IsDVDDrive>false</IsDVDDrive>
  <IsMusicFile>false</IsMusicFile>
  <IsVideoFile>true</IsVideoFile>
  <IsPictureFile>false</IsPictureFile>
  <IsTVFile>true</IsTVFile>
  <SegmentFiles size="1">
  <File>
  <![CDATA[ \\onyx\DVR-1\SageTV Recordings\2020-128003-0.mpg
  ]]>
  </File>
.....
FYI - I expect to release 1.2 tomorrow.
I'm getting "This program is already running in another instance. Stopping..."

Rebooted a couple times, cleaned out some temp folders, cleared IE temp, cookies,etc. and it's still happening. Hints?
TugboatBill Wrote:I'm getting "This program is already running in another instance. Stopping..."

Rebooted a couple times, cleaned out some temp folders, cleared IE temp, cookies,etc. and it's still happening. Hints?
In the installation directory/res/singleInstance.txt file, change the text from "running" to "stopped".
The program tries hard to only run as a single instance, but can get messed up if it ends prematurely.

This problem will be fixed in the next release.
Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15