Kodi Community Forum

Full Version: mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Ubuntite: please upgrade to SVN. Problem with deleting MythTV files and folders is finished. This will now work on MythTV 0.23. 0.24 is a work in progress.

For 0.24, deleting the files will be a mythicalLibrarian user job. The user job will be something along the lines of:
Code:
test "$RequiresDoover" != "1" && MythDataDelter "$InputPath"
outleradam Wrote:Ubuntite: please upgrade to SVN. Problem with deleting MythTV files and folders is finished. This will now work on MythTV 0.23. 0.24 is a work in progress.

For 0.24, deleting the files will be a mythicalLibrarian user job. The user job will be something along the lines of:
Code:
test "$RequiresDoover" != "1" && MythDataDelter "$InputPath"

Thanks, Adam! Looks to be working just fine!
Ubuntite, if you just upgraded today, please upgrade again within 18 hours or your episodes will be processed incorrectly. I did some serious revisions on the parser today. The changes from here to next stable release are purely odds and ends oriented..
I committed MythDataDeleter to the SVN.

MythDataDeleter enables a proper deletion of a recording as a mythicalLibrarian UserJob under MythTV/0.24.

MythDataDeleter is a full standalone python program. You can find the manual here http://www.mythtv.org/wiki/MythDataDeleter

To utilize under mythicalLibrarian with 0.24:
Code:
sudo curl "http://mythicallibrarian.googlecode.com/svn/trunk/pythonBindings/MythDataDeleter" > /usr/local/bin/MythDataDeleter
chmod +x /usr/local/bin/MythDataDeleter
echo "test \"$RequiresDoover\" != \"1\" && /usr/local/bin/MythDataDeleter \"$InputFile\" --writeFile --output=\"$MoveDir/$ShowFileName.info.txt\" |tee -a \"$mythicalLibrarian\output.log\""|sudo tee -a "/etc/mythicalLibrarian/JobSucessful "
mythicalLibrarian --update
The 0.23 depreciation workaround will be:
/etc/mythicalLibrarian/JobSucessful
Code:
if [ "$RequiresDoover" != "1" ]; then
    echo "REMOVING - $FileBaseName - THUMBNAILS - DATABASE ENTRIES"|tee -a "$mythicalLibrarian"/output.log
    mysql -u$MySQLuser -p$MySQLpass -e "use '$MySQLMythDb' ; delete from recorded where basename like '$FileBaseName'; "
    rm -f "$originaldirname/$FileBaseName".*    
fi
Please let me know if there are any problems.

As of now, SYMLINK=Disabled is depreciated and will be removed soon after next stable revision.
SVN Revision 904 committed to stable.

I would consider this a major revision since the last stable. The database was greatly changed so run mythicalLibrarian --doMaintenance or wait 18 hours since the last recognition for the database to redownload before running the new version. If you do not, then shows matched on Airdate will be recognized improperly.

Changes since last stable

Usage of python bindings for MythTV/0.24
Improved logic
10 point Confidence Rating of match is used to determine if the match was of high quality.
Low quality resolution of season and episode are put in the --doover que for later matching
Conflict resolution for shows with multiple releases on the same airdate
Logic path shows why decisions were made
Readable output during "delays" in processing to show what is being processed

There are alot more changes coming up so stay tuned.
outleradam Wrote:I committed MythDataDeleter to the SVN.

MythDataDeleter enables a proper deletion of a recording as a mythicalLibrarian UserJob under MythTV/0.24.

MythDataDeleter is a full standalone python program. You can find the manual here http://www.mythtv.org/wiki/MythDataDeleter

To utilize under mythicalLibrarian with 0.24:
Code:
sudo curl "http://mythicallibrarian.googlecode.com/svn/trunk/pythonBindings/MythDataDeleter" > /usr/local/bin/MythDataDeleter
chmod +x /usr/local/bin/MythDataDeleter
echo "test \"$RequiresDoover\" != \"1\" && /usr/local/bin/MythDataDeleter \"$InputFile\" --writeFile --output=\"$MoveDir/$ShowFileName.info.txt\" |tee -a \"$mythicalLibrarian\output.log\""|sudo tee -a "/etc/mythicalLibrarian/JobSucessful "
mythicalLibrarian --update
The 0.23 depreciation workaround will be:
/etc/mythicalLibrarian/JobSucessful
Code:
if [ "$RequiresDoover" != "1" ]; then
    echo "REMOVING - $FileBaseName - THUMBNAILS - DATABASE ENTRIES"|tee -a "$mythicalLibrarian"/output.log
    mysql -u$MySQLuser -p$MySQLpass -e "use '$MySQLMythDb' ; delete from recorded where basename like '$FileBaseName'; "
    rm -f "$originaldirname/$FileBaseName".*    
fi
Please let me know if there are any problems.

As of now, SYMLINK=Disabled is depreciated and will be removed soon after next stable revision.

Ok, I don't quite know what all this means. Do I need to add another user job to my mythtv setup to delete recordings, or will your script continue to delete my recordings from my mythtv database, just using a new method?
Also, I had internet connection problems for more than 24 hours, therefore I have a bunch of recordings that didn't get deleted from my mythtv db. I am still not quite sure how doover works, particularly with symlink=disabled. I believe that doover is of no value to anyone using symlink=disabled. Am I right?
^^For the last few weeks, mythicalLibrarian will not delete data if the file is put into the --doover que. This means, if it was a low-confidence match, mythicalLibrarian will not delete the database, and you can doover the file.

You can hand-edit the dover.sh to remove previous data. If you are unsure, then delete ~/.mythicalLibrarian/doover.sh.

There will be a future modification for the --doover que which will ask you to manually identify files and remove them from the --doover que. User identified files will be treated as high confidence and therefore removed from the --doover que. I'm thinking it will be a dialog menu based system for use during --doover.

But, for now, your que will build up unless a high confidence match is made. You can type the following for more information on how to improve your confidence ratings.
Code:
mythicalLibrarian --confidence

ubuntite Wrote:Ok, I don't quite know what all this means. Do I need to add another user job to my mythtv setup to delete recordings, or will your script continue to delete my recordings from my mythtv database, just using a new method?
For now, you can continue using what you've been using. The depreciation is just a warning for the future. I will be removing the functionality from the core of mythicalLibrarian eventually. The reason is I don't want to try to maintain backwards compatability with a feature I don't use. It makes it simpler to just have it as a user job.

The user job is simple...
Code:
test "$RequiresDoover" != "1" && /usr/local/bin/MythDataDeleter "$InputFile" --writeFile --output="$MoveDir/$ShowFileName.info.txt" |tee -a "$mythicalLibrarian/output.log""

which basically says: If this file is not in the --doover que, then run MythDataDeleter on the file, write the database output file to $MoveDir/ShowFileName.info.txt, and put all terminal data into the mythicalLibrarian log.

I need to pretty much rewrite the enitre wiki.
At this time there is no difference between SVN and stable. I am going to begin a series of commits which will destablize SVN version. These are strictly new features and no bug fixes. I am not aware of any problems in stable.

This destablization will be different from previous... I've formatted my media center and then run it for several days recording TV shows. The idea is to add the features and then process TV shows with mythicalLibrarian working out bugs along the way.

Features to be added:
1. Add capabilities for multiple instances of mythicalLibrarian at the same time.
2. Speed up database by checking with theTvDb if our database is out of date based on last updated time.
3. Query the user instead of using fuzzy logic in --doover mode. (this will be a rather large task)

This work will begin tomorrow.

Please let me know if there are any problems with stable so I can correct them.
forgot something...
I got some tips on speeding up BASH so I will be moving several "sed" actions from external commands to internal BASH commands.
so...
4. speed up operations

Again.. Please let me know if there are any problems with stable.
This thread has been quiet for a long time. I assume that people just don't know how mythicalLibrarian operates and therefore don't know when they're experiencing a bug.

Image

Let me know if there are any problems
No matter what I do I can not get my User Job to transcode. Here is the mythBackend log. Any help would be greatly appreciated.



PHP Code:
2010-12-27 09:39:17.483 MainServer::ANN Monitor
2010
-12-27 09:39:17.532 addingMythBE01 as a client (events2)
2010-12-27 09:39:17.895 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
2010-12-27 09:39:33.388 MainServer::ANN Monitor
2010
-12-27 09:39:33.433 addingMythBE01 as a client (events2)
2010-12-27 09:39:33.485 MainServer::ANN Monitor
2010
-12-27 09:39:33.533 addingMythBE01 as a client (events2)
2010-12-27 09:39:33.616 MainServer::ANN Monitor
2010
-12-27 09:39:33.667 addingMythBE01 as a client (events2)
2010-12-27 09:39:34.030 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
2010-12-27 09:39:34.283 ProgramInfo(): Updated pathname '':'' -> '2132_20101226231100.mpg'
2010-12-27 09:39:34.390 ProgramInfo(): Updated pathname '':'' -> '2134_20101226230400.mpg'
2010-12-27 09:39:40.482 MainServer::ANN Monitor
2010
-12-27 09:39:40.535 addingMythBE01 as a client (events2)
2010-12-27 09:39:40.593 MainServer::ANN Monitor
2010
-12-27 09:39:40.646 addingMythBE01 as a client (events2)
2010-12-27 09:39:40.693 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
2010-12-27 09:39:40.910 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
2010-12-27 09:39:43.354 MainServer::ANN Monitor
2010
-12-27 09:39:43.403 addingMythBE01 as a client (events2)
2010-12-27 09:39:43.460 MainServer::ANN Monitor
2010
-12-27 09:39:43.514 addingMythBE01 as a client (events2)
2010-12-27 09:39:43.572 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
2010-12-27 09:39:43.633 MainServer::ANN Monitor
2010
-12-27 09:39:43.703 addingMythBE01 as a client (events2)
2010-12-27 09:39:44.001 Reschedule requested for id -1.
2010
-12-27 09:39:44.380 Scheduled 0 items in 0.4 0.21 match 0.17 place
2010
-12-27 09:39:44.531 MainServer::ANN Monitor
2010
-12-27 09:39:44.583 addingMythBE01 as a client (events2)
2010-12-27 09:39:44.651 MainServer::ANN Monitor
2010
-12-27 09:39:44.705 addingMythBE01 as a client (events2)
2010-12-27 09:39:44.763 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
2010-12-27 09:39:44.979 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
2010-12-27 09:39:51.754 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
2010-12-27 09:39:51.816 JobQueueStarted mythicalLibrarian for "106_4 (WNYO)" recorded from channel 2064 at Sun Dec 26 23:15:00 2010
2010
-12-27 09:39:51.866 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
 
mythicalLibrarian --:invalid pattern

usage
mythicalLibrarian 'path/to/file' ['title'] ['subtitle']
usagemythicalLibrarian --mode [parameter] [parameter]
 
items in brackets are [context sensitve]
 
ensure you have set the user settings set properly and run
 
--mythicalDiagnostics before using mythicalLibrarian
 in 
--scan mode or as a UserJob.
 For 
more information typemythicalLibrarian --help or --special
 done
.
2010-12-27 09:39:52.520 JobQueue ErrorUser Job '/usr/local/bin/mythicalLibrarian "/home/mythtv/recorded/2064_20101226231500.mpg"' failed.
2010-12-27 09:39:52.618 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
2010-12-27 09:43:24.764 MainServer::ANN Monitor
2010
-12-27 09:43:25.210 addingMythBE01 as a client (events2)
2010-12-27 09:43:25.278 MainServer::ANN Monitor
2010
-12-27 09:43:25.321 addingMythBE01 as a client (events2)
2010-12-27 09:43:25.368 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg'
2010-12-27 09:43:25.659 ProgramInfo(): Updated pathname '':'' -> '2064_20101226231500.mpg' 
Assuming you have some sort of guide data to work with...

Run the following from the command line
Code:
sudo su mythtv
cd ~/.mythicalLibrarian/mythicalSetup
mythicalLibrarian --update
then when you've performed the update, it should pass all diagnostics, otherwise it will not be fit for use. Check for database login errors. If there are errors, then you should reconfigure mythtv-common and mythtv-database (see MythTV wiki) then run mythicalLibrarian --update again.

If you are not yet upgraded to mythtv 0.24, then you should apt-get dist-upgrade

on version 0.24 mythicalLibrarian uses python bindings to interface with the MythTV database. To test the python bindings:
Code:
MythDataGrabber [color=#dd0000][COLOR=black]2064_20101226231500.mpg
[/color][/COLOR]

The problem is that the user job cannot obtain information from the mythtv database about the recording. This is either improperly configured python bindings, or improperly configured username/password. You must also have valid guide data in the database for this to work. mythicalLibrarian must at least have a Title to work with and be run on the machine which recorded the show.

Let me know if there are any problems.
I do have guide data but I did notice the following when I run the upgrade, this is what I get.


Press any key to perform MythTV Database test...
Performing MythTV Python Bindings database test
Establishing database connection
Failed: attempting to use system default configuration
2010-12-27 11:48:18.314 Python Database Connection: Using connection settings from /home/rmikulec/.mythtv/config.xml
Connected to: MythBE01
Db Identified as: sql://[email protected]:3306/
Diagnostics passed.
Diagnostics failed.
MythTV Python Bindings are installed properly
MythTV Database Test ***PASSED**
ok, that does not seem to make sense. It got data and reported a failure. Run this:

Code:
MythDataGrabber [color=#dd0000][COLOR=black]2064_20101226231500.mpg
[/color][/COLOR]