Kodi Community Forum

Full Version: mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Quote:If mythicalLibrarian determines that there is suitable information, but it cannot pull the episode or season from TheTvDb, it will mark it as what I call an "epi-show". Epi-shows get put into the --doover que. You can have mythicalLibrarian rerun the doover que by typing mythicalLibrarian --doover.

So, the idea behind the doover is that if TheTVdb gets better information about the episode, mythicalLibrarian will rename the epi-show accordingly, reclassify it as an episode, and move it to the 'Episode' folder? If so, that is very cool. Would it be possible to have the doover run every time mythicalLibrarian runs? It would seem that it wouldn't add too much overhead, and the benefits would be better named files and a better library!

Quote:Please post the name and path of the episode. I can try to recreate the problem on my end.

Late Night with Jimmy Fallon/Season 2/Late Night with Jimmy Fallon S02E316 (Amy Poehler).mpg
and
Late Night with Jimmy Fallon/Season 2/Late Night with Jimmy Fallon S02E317 (John Legend).mpg

Quote:You need to have the source set up on each XBMC installation. If XBMC is receiving communications messages, but there are no library updates, then there is a problem with the source. mythicalLibrarian sends two messages, one says "display a message" the other says "update your sources".

Sure, I understand this. Here is what I did. I had a single XBMC IP Address in mythicalLibrarian and the "display a message" and "update sources" worked fine on the XBMC installation. I then added additional XBMC IP Addresses to mythicalLibrarian, and the result is that the original XBMC Installation gets the "display a message", but does not get its library updated. If I update the Library manually, it works just fine. Maybe I have the syntax wrong for multiple IP Addresses. I followed what looked to be the correct format:

XBMCIPs=( '1stIPaddressTongueort' '2ndIPaddressTongueort' '3rdIPaddressTongueort' )

Is this correct?

Quote:The ~/.mythicalLibrarian folder contains mythicalLibrarian's working files. It is the storage space for mythicalLibrarian's lookup files, the --doover que, --undo que, created folder tracking, and the comskip/NFO file tracking. It is also home to the DailyReport(a log of all files generated) and the output.log file(a detailed account of what happened during processing of each file). You can see what shows were processed each day by looking at the Daily Report folder. If you want to find out more about the episode, you can look at the output.log file. I wouldn't recommend changing anything in the folder, but it is good information.

So based on this information, since I am using a MythTV user job, should I only run mythicalLibrarian from the command line as the mythtv user? In other words, if I want to run mythicalLibrariabn --doover or mythicalLibrarian --update, should I only do so as the mythtv user?
ubuntite Wrote:So, the idea behind the doover is that if TheTVdb gets better information about the episode, mythicalLibrarian will rename the epi-show accordingly, reclassify it as an episode, and move it to the 'Episode' folder? If so, that is very cool.
Yep. It's very cool!

Code:
Would it be possible to have the doover run every time mythicalLibrarian runs?
The overhead and absence of the file have stopped me from doing this. mythicalLibrarian basically reverts the file to it's original form and then runs it as new. Some people have a NAS setup where the file gets recorded locally and then gets transferred across the NAS. This can take up to 10 minutes each way. If every 1/2 hour the file is rerun, it takes 10 minutes one way and 10 minutes the other way. Then, on top of that, each file will be redone, so it might be that every 1/2 hour, 10 hours of work are added to the que.


Quote:Late Night with Jimmy Fallon/Season 2/Late Night with Jimmy Fallon S02E316 (Amy Poehler).mpg
and
Late Night with Jimmy Fallon/Season 2/Late Night with Jimmy Fallon S02E317 (John Legend).mpg
It's not working here either. Maybe you should ask in a new thread in the general help area about this. Possibly XBMC cannot handle "Episode 317"... I don't understand this error.


Quote:XBMCIPs=( '1stIPaddressTongueort' '2ndIPaddressTongueort' '3rdIPaddressTongueort' )

Is this correct?
Yes. That's correct. Try a manual update. When running scan mode, mythicalLibrarian sends out sequential library update requests. For some reason the first one starts the update and the 2nd one stops it. It is a XBMC issue, but it should be fine after it gets onto a normal schedule of 1 recording at a time.



Quote:So based on this information, since I am using a MythTV user job, should I only run mythicalLibrarian from the command line as the mythtv user? In other words, if I want to run mythicalLibrariabn --doover or mythicalLibrarian --update, should I only do so as the mythtv user?
Yes. Always run as mythtv. This also ensures permissions are correct. If you run as root, then any files created can only be accessed by root. I think i've put a few preventive measures in place to ensure that user mythtv is used.
Quote:
Code:
Would it be possible to have the doover run every time mythicalLibrarian runs?
The overhead and absence of the file have stopped me from doing this. mythicalLibrarian basically reverts the file to it's original form and then runs it as new. Some people have a NAS setup where the file gets recorded locally and then gets transferred across the NAS. This can take up to 10 minutes each way. If every 1/2 hour the file is rerun, it takes 10 minutes one way and 10 minutes the other way. Then, on top of that, each file will be redone, so it might be that every 1/2 hour, 10 hours of work are added to the que.

I, too, copy my recordings to a NAS drive. Would it be possible to rethink the --doover process? Could you process the doover files in place (in my case on the NAS), and thus making it feasible to run --doover every time mythicalLibrarian runs?

Quote:Yes. That's correct. Try a manual update. When running scan mode, mythicalLibrarian sends out sequential library update requests. For some reason the first one starts the update and the 2nd one stops it. It is a XBMC issue, but it should be fine after it gets onto a normal schedule of 1 recording at a time.

A manual update always works, but I would like it to be automatic. I don't quite follow what you are saying above. Are you saying that if two files are being processed at the same time, (i.e. two MythTV recordings finish at the same time), that the automatic update library will fail?
^^ For the doover process I made a change last night in the SVN for you. --unmanned_doover provides a way to set up an "at" or "cron" job to perform the doover at a regular interval.

It should go something like this:
Code:
sudo -u mythtv sudo mythicalLibrarian --unmanned_doover
I think this should allow flexability in the --doover process without requiring mythicalLibrarian to keep track of the state of the jobs which ran previously.

Another option could be to include an option where if it has been 24 hours (according to TheTvDb). I'm still working out the kinks on this because it requires sudo access. I may be able to come up with something more perminant within the week. I'd say 12-24 hours would be an appropriate timeframe to run the --doover que. It can put a large strain on a server.


Also, yes, if 2 update commands are run at the same time then one will start the update process and the seccond will stop it. Take a moment to post in this thread that you are experienceing this issue http://forum.xbmc.org/showthread.php?tid=82133
Please do not use the SVN right now. It is bork beyond belief. Several variable names are in transition. the following user-job variable names will be changed

originalext = OriginalExt
exx = Exx
sxx = Sxx
absoluteepisodenumber=AbsoluteEpisodeNumber
seriesid=SeriesID
stars=Stars

Also, currently being changed is --doover and --undo will now be interruptable. They will simply be restored the next time mythicalLibrarian is run and will be able to be rerun as if nothing happened.

Again, all of these changes have made the SVN VERY unstable. Add that to the upcomming python bindings, and there's a world of problems to deal with. Please use stable.
the SVN is in operation. Python Bindings are up and running. Quite a bit of restructuring was done. Everything seems to be working properly now.

If you are running mythtv 0.24 (development repository) then you can take advantage of the new Python Bindings. You need to run the following:

Code:
curl http://mythicallibrarian.googlecode.com/svn/trunk/pythonBindings/mythicalPythonBindings.py >/usr/local/bin/mythicalPythonBindings.py
chmod +x /usr/local/bin/mythicalPythonBindings.py
echo "DatabaseType=MythTVPythonBindings" |sudo tee /etc/mythicalLibrarian/globalSettings

Please check out mythicalPythonBindings.py. It's a great python script written by barney_1 and myself cooperatively. It will try until it extinguishes all methods of connecting to the mythtv database, then extract all usable information from the database about a specific file and deliver it in a neat "var = value" format.
As a note... The mythtv pin option is not yet supported in our python bindings. If you set the pin then do not set the global setting which was mentioned earlier.
The Python Bindings are now to the point where they support everything required. They interface mythicalLibrarian with a named mkfifo pipe which is fully POSIX compliant. The DBPin option is now supported for those who use a DBPin. There should be no problems with the python bindings at this time.

In order to reach full maturity, the Python Bindings requires a test routine much like mythicalLibrarian's --mythicalDiagnostics.

mythicalLibrarian itself is fully supporting the python bindings in their current state. some changes will be required for full fledged diagnostics.

mythicalSetup requires addition of python bindings options and dependencies checking. Until I write mythicalSetup to merge python bindings into mythicalLibrarian, the directions on the previous page are still the preferred ones for use with MythTV v0.24.

When mythicalPythonBindings.py is passed a filename, it pulls all information out of the MythTV database about that file, including commercial skip information. I feel that the title "mythicalPythonBindings.py" is a bit too long and underdescriptive. Will someone recommend a new name to me? I'm open for anything.
Also in the SVN, selectable default options. mythicalLibrarian has too many options which cause confusion. I think people feel obligated to change at least one setting. Usually it causes undesired operation. Using the defaults bypasses many problems associated with many setups. This means the options are there, but using the defaults will get things working in a logical manner the first time. The remaining fields which are not filled out by defaults may require human intervention at some point, but are automatically filled out with logical defaults.
Things are up and running well in the SVN at this time. New options and features are added. The python foundation is strong and steady. A few more options will be added soon for mythicalDiagnostics of the mythicalPython bindings.



Image

If you are running v0.24, i'd recommend updating now.

All that remains is mythicalDiagnostics for the python bindings....

Comming soon Tvheadend support
Hmmm...

I have updated a couple of times, but now my episodes and showings are not being named properly. I have this problem:

Episodes are not identifed
When episodes are not identified after an upgrade it is because the database handling has been upgraded, however, the database itself has not. Delete all database folders to correct the issue http://forum.xbmc.org/showpost.php?p=492...stcount=40

I tried the things suggested, but am still getting episodes and showings without season and episode information. Is there a way to clear out all data and start from scratch? Can I delete everything in /home/mythtv/.mythicalLibrarian ? I do not care about keeping the doover and undo stuff. Should I care?
I need information.

Can you find an example in /home/mythtv/.mythicalLibrarian/output.log?

Are you using Stable or SVN? Are you using MythTVv0.24? I will be releasing a mythicalDiagnostics for the 0.24 python bindings soon, maybe tonight. It's nearly complete and has not been committed to SVN yet.


To automate the cleaning process, type
Code:
mythicalLibrarian --doMaintenance
Then do an undo and a rescan.
Code:
mythicalLibrarian --undo
mythicalLibrarian --scan mpg /path_to/your/files

This should take care of any problems you may encounter from using an old internal database with a new version of mythicalLibrarian.


If you're using 0.24, run the following: MythDataGrabber --Diagnostic=True

I can't really tell you more without a debug log from ~/.mythicalLibrarian/output.log
I'll be committing the python bindings to stable soon. I want to make sure that there are no problems before I do so.

Please let me know if there are any problems.
I have a few comments regarding updating:

1. After updating, MoveDir always has a trailing /. I must manually edit mythicalLibrarian and remove it.

2. Even though I do not choose UseOriginalDir when updating (i choose the other option which I believe is manually select folders or something), UseOriginalDir is set to Enabled. I have to manually change it to Disabled.

3. DatabaseType is always reset to MythTVPythonBindings. This doesn't seem to work for me. I have to manually change it to MythTV for it to work every time I update.

4. PrimaryShowDir and AlternateShowDir end up with trailing /s. It doesn't say whether that is allowed or not, but I always manually delete them.
outleradam Wrote:To automate the cleaning process, type
Code:
mythicalLibrarian --doMaintenance
Then do an undo and a rescan.
Code:
mythicalLibrarian --undo
mythicalLibrarian --scan mpg /path_to/your/files

This should take care of any problems you may encounter from using an old internal database with a new version of mythicalLibrarian.
I am a bit unclear as to what the 'mythicalLibrarian --scan' will do. Are you telling me to run this command on recordings that have not previously been processed by mythicalLibrarian? I use the settings:

UseOriginalDir=Disabled
and
SYMLINK=Disabled

therefore it does not seem to make sense to run 'mythicalLibrarian --scan' on the recordings once they have been moved into a new dir and have been removed from the myth database.