HOWTO: Kodi + tvheadend + AUTOMATIC Metadata/Library integration for ALL YOUR TV
#1
Lightbulb 
If you like the way Windows Media Center handled Live TV recordings, in terms of grouping them and letting you watch and manage them from a single "Recorded TV" interface, you may have discovered that while Kodi has much better library and presentation features, the way PVR backends integrate don't always allow you to view your recordings in that library or in one place. In some instances, you will have a recordings interface that is different from the TV Shows interface and your files are split. In other cases, you don't have metadata for Live TV recordings. For me, this was not ideal, so I looked to find a solution to bring it all together in an integrated fashion for the ultimate TV viewing experience.

After following this guide, you will be able to:
  • Record live TV and view the results in the Kodi TV library, automatically.
  • Add TVDB information to these recordings, whether or not the backend recorded in a container that supports metadata
  • Delete or modify them
  • View them alongside TV episodes you may have downloaded, not recorded, all in one integrated UI

This may sound simple, but the implementation involves several key steps that you have to plan ahead for.

This guide relies on the hard work of a lot of people other than me, whose work I reference here. All I did was put various things together and do some problem-solving to create a recipe for success, because I did not see one place that took things all the way from setting up the PVR to viewing Live, downloaded, AND Recorded TV in a single place.

What this guide does NOT cover:
  • Picking computing or server hardware
  • Picking TV tuner hardware
  • Picking Kodi hardware (this is THE thread you need for this: http://forum.kodi.tv/showthread.php?tid=252916)
  • Whether or not you run a distributed system vs. PVR backend and Kodi together
  • Detailed installation of Kodi or tvheadend
There are lots of fantastic posts and materials already existing for those things. So I will cover them very sparsely and link to references for these. I will get into details in configuration and setup only where changes are needed to ensure the above requirements are met.

Step 1: Install tvheadend
You have lots of choices and ways. One great way is written here: http://forum.kodi.tv/showthread.php?tid=...pid2315547
Very briefly, here is what I did:
  • Installed Ubuntu 16.04 LTS
  • Updated all packages and core system modules to the latest
  • Added the tvheadend repo: https://tvheadend.org/projects/tvheadend...Repository is the place to look for instructions. When you follow them, you will see this in /etc/apt:
    Code:
    deb http://dl.bintray.com/dreamcat4/ubuntu stable main
  • Executed:
    Code:
    sudo apt-get install tvheadend
    Somewhere along the way, you will pick a username and password for tvheadend. Remember these. When installation finishes,
    Code:
    sudo systemctl enable tvheadend
    sudo systemctl start tvheadend
Guess what? tvheadend is installed!

Step 2: Install any needed TV tuner hardware
This varies from device to device. I highly recommend the HD Homerun products by Silicon Dust, as they are now supported out of the box with tvheadend. I literally added or installed nothing on top of the above instructions to make the HD Homerun work.

Make sure to restart the tvheadend service if you add / modify / remove /etc any tuner drivers.

Step 3: Configure tvheadend up through TV channels.
Here's another area where other people have better guides than me. Follow most of this guide, except in areas where I indicate there are differences: https://tvheadend.org/boards/4/topics/20429 - it does not matter that this person's example is on a RPi. The steps you want, copied from his guide and generalized somewhat, are:
Quote:Next Configure tvheadend 4.0.7
- Determine your tvheadend's server IP
- Launch tvheadend web based config: http://IP:9981/
- log in with your tvheadend user and password
- Click Configuration
- Click DVB inputs
- Click Networks. Add a new network. Choose Type: ATSC. Give it any name.
- Click TV adapters - you should see the HDHomerun tuners (or other tuners you may have) displayed.
- Open the system log by clicking the double ^ symbol in the lower right corner, so you can see what's going on.
- Click TV tuners subfolder icon, if there is one. Change the network type to ATSC. ATSC is used for over the air broadcasts in the USA. Click Save. If there isn't a subfolder for your tuners, skip this step
- Click TV adapters.
- Click each tuner. Enable it. Change the network to whatever you named it above. Turn off idle scan. Save.
- Click Networks.
- Click Pre-defined Muxes:
- Choose "United States: us-ATSC-center-frequencies-8VSB". Save.
- Scanning now begins. Look for errors in the log. If you see "locked" messages, restart tvheadend, or restart your server, navigate back into here when things are back up, and then click "force scan" to start scanning again.
- Click Muxes. You should see a bunch of muxes. These are the potential base channels. You should see "Pending" in scan results until it detects channels.
- During scanning, the channels that exist will say "OK", the others will say "FAIL". Wait until scanning is done. This is when all the PENDING ones are no longer saying PENDING, and no more scrolling is occurring in the system log area.
- Click Services, you should now see all of the channels you can receive with your TV.
- Click Map All, click the "check availability" checkbox.
- Click "Channel / EPG", Channels, Uncheck the channels you don't want to see, then save.

Step 4: Install and configure xmltv-based grabbers - tv_grab_file step
At this point you have everything you need for live TV viewing, but not the EPG or ability to record. The following steps are now needed. Pay attention to this part, this is the magic that is going to allow you to get guide listings that you can then use to make sure your recordings get put nicely into your Kodi TV library.
  • Read this guide: https://freetoairamerica.wordpress.com/2...-frontend/
    Seriously, read it. It's well-written and tells the story. Assuming you've read this guide, now do the following:
  • Download tv_grab_file, and place it in /usr/bin or whereever the rest of your tvheadend tv_* grab scripts are. Make sure it is executable.
  • Edit tv_grab_file and make sure the path it's reading from is the path where you expect your .xmltv file to be (the xmltv file is covered in the next step)
  • Restart tvheadend so it picks up the tv_grab_file binary in the drop down list for grabbers - see below where you will use this.

Step 5: Install and configure xmltv-based grabbers - creating the XMLTV file
For the U.S., Zap2it is an excellent listings provider because it contains season AND episode info in the extended listings. THIS IS THE KEY! If you're doing this in the USA, DO NOT use mc2xml; use zap2xml and grab the Zap2It listings rather than TV Guide listings. I have not paid for or tried Schedules Direct and cannot say whether or not those listings contain the season and episode info. For other regions, mc2xml may be the right way to go, and there are a lot of posts and docs on downloading and installing mc2xml. If you go that route, you will modify the steps below to call the mc2xml script with its options instead of zap2xml. I will document zap2xml in detail.
  • Have I mentioned you need to read this guide? https://freetoairamerica.wordpress.com/2...-frontend/
    We will do almost everything in there, except we will make changes to how zap2xml is used and called.
  • Download zap2xml, and place it in /usr/bin or whereever binaries need to go on your tvheadend server. Make it executable.
  • Register for your free Zap2It account. Unlike what's in the guide, you DO NOT need to make favorite channels, nor do you HAVE to use the 6 hour grid option on their website. You can if you wish but it makes no difference anymore to zap2xml and Zap2It's API.
  • Read up on zap2xml. http://zap2xml.awardspace.info/
  • Modify zap2xml to write the XMLTV file where you want it, and where tv_grab_file (see above!) expects to find it.
  • We're now going to figure out its argument list and test it. MAKE SURE you use the extended guide listing options of -D and -I, especially -I (that's a capital "eye"). It will make your initial zap2xml run very long but it's worth it because you need to get the extended descriptions, episode titles, and the TVDB season and episode numbers, which are part of Zap2It's service now!
    Code:
    $ ./zap2xml.pl -u "your_username" -p "your_pass" -o /path/to/your/tv_grab_file.xmltv -d 3 -c /path/to/your/cache -i /path/to/your/icons -L -A -T -D -I
  • If it works, you'll see the XMLTV file you specified with -o and it'll have valid listing XML in it for 3 days.
  • Wire it up to cron. Use crontab -e and put this in:
    Code:
    # run zap2xml everyday at 2:12 am
    12 2 * * * /path/to/your/zap2xml.pl -u "your_username" -p "your_pass" -o /path/to/your/tv_grab_file.xmltv -d 13 -c /path/to/your/cache -i /path/to/your/icons -L -A -T -D -I -q
    Note the addition of the -q option so you don't get tons of output. I am asking for 13 days of listings here. You can ask for more or fewer, depending on your Internet speed, server speed, etc. This job has to finish running before you run tv_grab_file from tvheadend, which is covered in the next step. So pick the times you prefer.

Step 6: Finish setting up the EPG in tvheadend
Now you have your grabber set up, and it's going to be grabbing valid XML from that XML file being written by zap2xml (or mc2xml, if that is where you live). We're going to go back to quoting https://tvheadend.org/boards/4/topics/20429 - pay attention to some key changes in the quoted text below.

Quote:- Log into the tvheadend web page as before
- Go to Configuration -> Channel / EPG -> EPG Grabber
- In General Config, uncheck all.
- In Internal Grabber, choose the "XMLTV: tv_grab_file grabber.
- Uncheck all items in Over-the-air grabbers
- Uncheck all items in External Interfaces.
- Click the tiny double ^ icon in the lower right to open the system log window.
- For Cron multi-line, schedule your grab:
Code:
# run@ 4:04 am
4 4 * * *

- Click save configuration. If everything worked right you should see it download schedules and channels.
- Wait until it finishes, then reboot,--> YOU DO NOT NEED THIS STEP
- Go to Configuration -> Channel / EPG -> Channels
- Uncheck the channels you don't want to see.
- Click on each channel, choose the correct EPG source for each channel. Usually, you will want to match up the EPG guide listing data to the channel name or number. You can pick more than one source, if that's appropriate for your channel, but this is uncommon. Save.
- Reboot and close your browser --> YOU DO NOT NEED THIS STEP. Just reload the page.
- Log into the tvheadend web page
- Wait a few minutes then click refresh on the Electronic Program Guide. Schedules should show up.
What you need to verify is that the season and episode numbers are showing up. They will if you followed the above instructions correctly. Check this by heading into your guide listings, and looking at the episode column. Most episodes should now show up in the SXX.EXX format. If so, then congratulations, you're almost there.

Step 7: Modify tvheadend's recording profile
We're now going to make sure that our recorded filenames contain the episode data we saw above.
  • Go into tvheadend's config and choose Recording. On the right side make sure your default recording profile has the following:
  • A recording system path that points to a location you will set up as a source in Kodi - this is important so you can automate everything later. REMEMBER THIS PATH
  • Check the "include episode in filename" option - this is not the episode name, it's the season and episode NUMBERS, which is the #1 key
  • Make subdirectories per title (don't choose the other two)
  • To be on the safe side, in case you record something that does not have episode numbers, also choose the two options for date and time. I also select Put Episode In Filename Before Date And Time. Time is helpful if your recording gets interrupted so you can resume and not overwrite the older file for that same episode/recording.
  • Save everything! Your recordings will now have the episode AND season info in the filename, which is exactly what the TVDB scraper in Kodi loves.

Step 8: Make your tvheadend recording system path available on the network so Kodi can find it (OPTIONAL)
If your tvheadend and Kodi setup is running on the same box, then you can probably find the recorded TV files within Kodi pretty easily for Step 9. If not, you will want to make your files available to Kodi over the network. In this example, I use SMB because my NAS runs Windows Home Server (my last remaining holdover from my Windows Media Center days), but if you're using a Synology or some Unix based NAS, use NFS.
  • Make a directory to mount your files on. We'll make one up here as an example. We'll assign ownership to 'hts', which is the default tvheadend user.
    Code:
    sudo mkdir -F /home/hts/tv/recordings
    sudo chown has:hts /home/hts/tv/recordings
  • If your NAS needs a username and password, you're not going to put your username and password in /etc/fstab, since this file is world-readable, so create a file called .smbfile, put it somewhere, assign the right permissions, and fill it with goodness:
    Code:
    username=your_nas_user
    password=your_nas_share_pass
    Now change permissions so no one but hts can see it:
    Code:
    chmod 600 /path/to/.smbfile
  • edit fstab and automatically mount your NAS share to this area, MAKING SURE to assign proper ownership to 'hts' again. This way, this stuff survives a reboot of your tvheadend server. Automation is key, right? Add a line like this:
    Code:
    //NAS-SERVER-NAME/path/you/want/for/your/TV/recordings /home/hts/tv/recordings cifs credentials=/path/to/.smbfile,user,rw,uid=111,gid=118  0     0
    There's a lot of magic in this line. uid and gid have to match the user name and group name of the hts user (or whichever user you want to use). This ensures tvheadend can write the share when you start or schedule recordings.
    Pro tip: If you want to include spaces in your path name, use \040 as an escape sequence in /etc/fstab - NOT A REAL SPACE!
  • Reboot to test automation of mounting the share. If this worked, df -h will show the share mounted.

Step 9: Set up a TV Library source in Kodi that matches where tvheadend is recording
If your tvheadend and Kodi setup is running on the same box, then you can probably find the recorded TV files within Kodi pretty easily here. If not, use the NAS share you set up in Step 8.
  • Start Kodi, go into Videos > Files
  • Add a source - either pick the local drive your recordings are on, or Choose a network location and browse to the NFS or SMB share where your files are at
  • Enter any credentials needed to access the NAS (hint: in Step 8, you probably are using the same credentials, but it's possible they are different)
  • Set the type to be TV Shows - make your selections about where you want to scrape (hint: use TVDB - it's great)
  • Since your files in Step 7 have season and episode numbering in the filename, sit back and wait while Kodi scrapes your files and makes beautiful meta data entries. You now have a TV Shows library that tvheadend is recording to, from which you can watch directly!

Step 10: Kodi automation and management tweaks
This part is technically optional, but not for me. Remember, at the beginning of this article, we wanted to achieve WMC-like management in that we could automatically get recordings to show up in our "TV Shows" library. We wanted an integrated view across all our downloaded, recorded, or otherwise procured TV shows, NOT the "recordings" area of Live TV, which often lacks metadata and is in a separate place in Kodi. Also if we want to, we want to delete these TV shows from the library and from the NAS when done viewing to save space. Not all TV shows we record are worth keeping, are they?
  • Install the Library Watchdog add on from the add-on repositories in Kodi. This will update your library as soon as a new file is written into the recordings area (provided its filename has the right naming, per above). If you don't do this step, you will need to either automatically have Kodi update your library on startup (there is an option for this in settings), or you will need to trigger an update yourself. Neither of these two options is ok for me, so I used this add-on, and it works very well.
  • To allow deletion, enable this feature from Kodi's Appearance > Settings > File Lists area (Allow file renaming and deletion). See here: http://kodi.wiki/view/Settings/Appearance.
    Note: You will need to first delete the file from your library, confirm that this is ok, then you will get another prompt to confirm that you really wish to blow away the actual file.
You are now done! Every time a new recording is placed in the recordings folder, it will be scanned in, and the proper show with metadata will now show up in your TV library. This works even during the beginning of a recording, so you can start to watch even before the recording is complete. This also works for instant recordings, one-off recordings, timer-based recordings, recordings you schedule directly in tvheadend, and files you may move into this folder on your own (downloaded, etc.). If you have multiple files for the same show (you stopped and restarted recording) this works if you are putting the timestamp in the filename - you will get two instances of the episode showing up so you can watch both parts, etc.

I hope you enjoyed this guide for the ultimate PRO setup of Kodi TV viewing!
Reply
#2
I'm a little bit confused about this:
(2016-07-14, 23:48)MasterYous Wrote: Most episodes should now show up in the SXX.EXX format. If so, then congratulations, you're almost there.

When Tvheadend writes to the filename it only puts EXX with no season. The scraper only matched up one 1 of 3 shows in my directory and the one that it did match is incorrectly matching shows to episodes. I'm in Canada if that matters. I believe the only difference is that I have not put the date in the file name as you suggested.
Reply
#3
(2016-11-28, 15:56)igbee Wrote: I'm a little bit confused about this:
(2016-07-14, 23:48)MasterYous Wrote: Most episodes should now show up in the SXX.EXX format. If so, then congratulations, you're almost there.

When Tvheadend writes to the filename it only puts EXX with no season. The scraper only matched up one 1 of 3 shows in my directory and the one that it did match is incorrectly matching shows to episodes. I'm in Canada if that matters. I believe the only difference is that I have not put the date in the file name as you suggested.

Not a guide sources provide season/episode information. I know that Schedules Direct's JSON API serves proper season/episode numbers, while their older DataDirect API usually only supplies the episode/production number rather than the season as well.
Reply
#4
(2016-11-28, 16:58)rpcameron Wrote: Not a guide sources provide season/episode information. I know that Schedules Direct's JSON API serves proper season/episode numbers, while their older DataDirect API usually only supplies the episode/production number rather than the season as well.

I'm using this great plugin http://forum.kodi.tv/showthread.php?tid=267022 to grab the epg data from zap2it. I can see there are season and episodes on zap2it.com for 2 of the shows I'm recording. I'll have the check the xmltv file to see if that info is getting in there.
Reply
#5
It was a good guide, the Guy from freetoairamerica wrote, he might consider setting his clock forward to 2016, i lmao where he wrote it was fairly new watching IPTV, then i had a jurika moment, he was ofc an American
it took me a while though to figure out, the guy was for real, but very good guide easy to follow.
English is not my native language; please excuse typing errors.
Grampa Simpson: Unfortunately, like all true stories, this one has a crappy ending. You have a story with an ending.
Grampa, are you sure you're okay to drive at night? Grampa: It's night?
Banned add-ons
Reply
#6
I believe my problem is that in the xmltv file the only episode info I have is "dd_progid" and I don't think you can properly gather Season and Episode from this ID, which looks like this: EP00018693.0612 for an episode of the Simpsons, for example.
Reply
#7
(2016-11-29, 21:26)igbee Wrote: I believe my problem is that in the xmltv file the only episode info I have is "dd_progid" and I don't think you can properly gather Season and Episode from this ID, which looks like this: EP00018693.0612 for an episode of the Simpsons, for example.

Correct. That is your grabbers internal name for that specific program in its database.
Reply
#8
(Newbie here)
Is it possible to do this using kodi/hdhomerun on Windows or this would only work with tvheadend (linux only)?
Reply

Logout Mark Read Team Forum Stats Members Help
HOWTO: Kodi + tvheadend + AUTOMATIC Metadata/Library integration for ALL YOUR TV0