Kodi Community Forum

Full Version: mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Thank you. I fixed the trailing = sign

Recordings which are not matched properly will not be removed unless they have proper episode matching data. Were the shows in question properly identified, or are they in the --doover que?

For the commercial markups, please give me the output of the following commands
Code:
cat ~/.mythicalLibrarian/markupstart.txt
cat ~/.mythicalLibrarian/markupstop.txt
cat ~/.mythicalLibrarian/markupframes.txt
grep CommercialMarkup /usr/local/bin/mythicalLibrarian
I cannot reproduce that here. Comskip.txt files will only be generated if the MythTV commflag job is enabled. There is no .txt file present in the recordings folders?
outleradam Wrote:Thank you. I fixed the trailing = sign

Recordings which are not matched properly will not be removed unless they have proper episode matching data. Were the shows in question properly identified, or are they in the --doover que?

For the commercial markups, please give me the output of the following commands
Code:
cat ~/.mythicalLibrarian/markupstart.txt
cat ~/.mythicalLibrarian/markupstop.txt
cat ~/.mythicalLibrarian/markupframes.txt
grep CommercialMarkup /usr/local/bin/mythicalLibrarian
I cannot reproduce that here. Comskip.txt files will only be generated if the MythTV commflag job is enabled. There is no .txt file present in the recordings folders?

--doover que is empty. The files have been deleted from the recordings directory, but they are not being removed from the MythTV database. This used to work properly.

markupstart.txt is empty.
markupstop.txt is empty.
markupframes.txt is empty (except for a header row).

grep CommercialMarkup /usr/local/bin/mythicalLibrarian:
Code:
#CommercialMarkup will generate comskip files for recordings when they are moved. Enabled|Disabled
CommercialMarkup=Enabled
#CommercialMarkupCleanup will execute a maintenance routine which will remove comskip files if they are not needed
CommercialMarkupCleanup=Enabled
            CommercialMarkup="Created"
        if [ "$CommercialMarkup" = "Enabled" ]; then    
    if [ "$CommercialMarkup" = "Enabled" ]; then
    if [ "$CommercialMarkupCleanup" = "Enabled" -a -f "$mythicalLibrarian/created.tracking" ]; then
            if [ "$CommercialMarkup" = "Created" ]; then
         if [ "$CommercialMarkup" = "Created" ]; then
Your markups and headered file says that the last time you ran it, there was no comskip data to work with, or mythtv could not access the file. Make sure the permissions are set properly
Code:
sudo chown -R mythtv:mythtv /home/mythtv/.mythicalLibrarian
sudo chmod -R 666 /home/mythtv/.mythicalLibrarian
sudo chmod -R 600 /home/mythtv/.mythicalLibrarian/mythicalSetup


ok, so lets verify you have data to work with. We are going into the raw mysql commands which grab data from mythtv 0.23 and below.

replace -umythtv -pmythtv with your -uUSERNAME and your -pPASSWORD
Code:
mysql -umythtv -pmythtv -e 'use mythconverg; select chanid, starttime from recordedmarkup where type = "4";'
will return something like this
Code:
chanid             starttime
------------------------------------
|   1956 | 2010-11-02 23:30:00 |
|   1956 | 2010-11-02 23:30:00 |
|   1956 | 2010-11-02 23:30:00 |
+--------+---------------------+
Then, pick one set of chanid/starttime

ok, so we have our startime and chanid. We need to find out what file they go to.

replace 1956 with one of your chanids and 2010-11-02 23:30:00 with one of your starttimes
Code:
mysql -umythtv -pmythtv -e 'use mythconverg; select basename from recorded where chanid like "1956" and starttime like "2010-11-02 23:30:00"'
+-------------------------+
| basename                |
+-------------------------+
| 1956_20101102233000.mpg |
+-------------------------+

This will identify a show in your database which for sure has comskip data to work with.

Code:
mythicalLibrarian /path_to_my_videos/1956_20101102233000.mpg

That file should have no problem generating a comskip.

If there is a problem, I need to see the markupstart, markupstop and the markupframes file.
OK, here you go:

Code:
mythicalLibrarian /var/lib/mythtv/recordings/1018_20101103210000.mpg
Revision 752 Wed Nov 3 18:55:38 PDT 2010
Accessing MythTV DataBase:
RECSTART: DATE:
PROGRAMID: ShowCategory:
PLOT:
JOB: mythicalLibrarian '/var/lib/mythtv/recordings/1018_20101103210000.mpg' 'Larry King Live' ''
WARNING: EPISODE INFORMATION COULD NOT BE OBTAINED
HANDLING AS A GENERAL SHOWING
Please consider helping out and adding to thetvdb.
TARGET SET:/media/recordings/Showings/Larry King Live/Larry King Live S0E0 ( Recorded  on )
FILE NAME EXISTS.  FILE WILL BE KNOWN AS: Larry King Live S0E0 ( Recorded  on )-10
/usr/local/bin/mythicalLibrarian: line 921: [: too many arguments
mv: preserving times for `/media/recordings/Showings/Larry King Live/Larry King Live S0E0 ( Recorded  on )-10.mpg': Operation not permitted
mv: preserving permissions for `/media/recordings/Showings/Larry King Live/Larry King Live S0E0 ( Recorded  on )-10.mpg': Operation not permitted
MOVED: /var/lib/mythtv/recordings/1018_20101103210000.mpg
TO: /media/recordings/Showings/Larry King Live/Larry King Live S0E0 ( Recorded  on )-10.mpg
chmod: changing permissions of `/media/recordings/Showings/Larry King Live/Larry King Live S0E0 ( Recorded  on )-10.mpg': Operation not permitted
SENDING REQUESTED COMMANDS TO:192.168.0.100:8080
/usr/local/bin/mythicalLibrarian: line 542: test: too many arguments
/usr/local/bin/mythicalLibrarian: line 543: test: too many arguments
RSS ENTRY SUCCESSFULLY CREATED: /var/www/mythical-rss/rss.xml
SUCCESSFUL COMPLETEION TYPE: MoveModeSuccessful

Some additional information... my recordings are being moved to a samba share. I don't know if that will be relevant or not.
There's some wierd stuff going on there. Are you getting any errors at all during mythicalSetup? When you run mythicalLibrarian --update, can you post the entire diagnostic?

This may be unrelated, but I'm removing all test statements with || in it starting now.


Code:
[email protected]:~$ $a=1
0=1: command not found
[email protected]:~$ echo $?
127
[email protected]:~$ a=0
[email protected]:~$ b=1
[email protected]:~$ c=1
[email protected]:~$ d=1
[email protected]:~$ test $a = 1 || test $b = 0 &&  test $c = 0  && test $d = 0 && echo "why is this true?"
[email protected]:~$ test $a=1 || test $b=0 &&  test $c=0 && test $d=0 && echo "why is this true?"
why is this true?
[email protected]:~$
I cannot explain this... But I know that there may be a problem with the test command in Maverick.
outleradam Wrote:There's some wierd stuff going on there. Are you getting any errors at all during mythicalSetup? When you run mythicalLibrarian --update, can you post the entire diagnostic?

This may be unrelated, but I'm removing all test statements with || in it starting now.


Code:
[email protected]:~$ $a=1
0=1: command not found
[email protected]:~$ echo $?
127
[email protected]:~$ a=0
[email protected]:~$ b=1
[email protected]:~$ c=1
[email protected]:~$ d=1
[email protected]:~$ test $a = 1 || test $b = 0 &&  test $c = 0  && test $d = 0 && echo "why is this true?"
[email protected]:~$ test $a=1 || test $b=0 &&  test $c=0 && test $d=0 && echo "why is this true?"
why is this true?
[email protected]:~$
I cannot explain this... But I know that there may be a problem with the test command in Maverick.

I just upgraded to 754:

Code:
Testing file system permissions
df: `/home/username/Episodes': No such file or directory
df: no file systems processed
COULD NOT READ FREE SPACE on ""
QUICK FIX: chmod -R 775 ""
FOLDER DOES NOT EXIST: /home/username/Movies
QUICK FIX: mkdir "/home/username/Movies"
FOLDER DOES NOT EXIST: /home/username/Showings
QUICK FIX: mkdir "/home/username/Showings"
USER SETTING     |FLAG|Folder name
MoveDir          |1   |/media/recordings/Episodes
AlternateMoveDir |0   |/home/username/Episodes
PrimaryMovieDir  |1   |/media/recordings/Movies
AlternateMovieDir|0   |/home/username/Movies
PrimaryShowDir   |1   |/media/recordings/Showings
AlternateShowDir |0   |/home/username/Showings
mythicalLibrarian|1   |/home/username/.mythicalLibrarian
Critical Filesystem checks ***FAILED***
OVERALL FAILURE
press any key to continue on to the final results...

  All tests complete            RESULTS
  MythTV Database Test:      ***PASSED***
  Installed Packages:        ***PASSED***
  Notifications Test:        ***PASSED***
  XBMC Communications:       ***PASSED***
  File System Checks:        ***FAILED***
  mythicalTesting:           COULD NOT COMPLETE
  Overall mythicalReadiness: ***FAILURE***
-----End of diagnostics-----
The filesystem checks failed.  Likely causes:
The user named root does not have permission to use the folder
  corrective action: sudo chown root /Name/of/My_failed_folder
The folder belongs to a different group then the user named root
  corrective action: sudo adduser mythtv root
                     sudo adduser root mythtv
The folder does not exist
  corrective action: mkdir /Name/of/My_failed_folder
mythicalLibrarian has encountered an error.  This may be due to
lack of network connection.   If the problem persists, contact
outleradam here: http://forum.xbmc.org/showthread.php?tid=65644

These issues must be corrected in order for mythicalLibrarian to
perform correctly
Please try again.  If problem persists, please post here: http://forum.xbmc.org/showthread.php?tid=65644
MythTV job not added because mythicalLibrarian already exists in slot 1
permissions were set for user: mythtv.
RSS Feed will be located at http://192.168.0.100/mythical-rss/rss.xml .
mythicalLibrarian is located in /usr/local/bin
mythicalLibrarian's log is located in ~/.mythicalLibrarian/output.log
User specified folders are to be used for placement of recordings.
Check the mythicalLibrarian checkbox when setting up recordings.
For more information, type 'mythicalLibrarian --help'
Done.

There seem to be complaints about the choice of alternate directories. I am using ~/ for my alternates, which of course change between when I install or upgrade mythicalLibrarian (I am logged in as username, but run sudo mythicalLibrarian --update), and when mythicalLibrarian is run as a user job in mythtv (run as user mythtv).

Request:

Could you possible include a "smarter" upgrade option that just uses all of the users current settings, rather that having to step through the upgrade process. That might actually prevent some bad upgrades. I find I use very few of your assumed defaults, probably because of my choice of SYMLINK=Disabled.
There is a dumb option which does what you would like. It's in the help under --special
Code:
======globalSettings=====
If you wish to override any settings in mythicalSetup, use
globalSettings.  globalSettings allows you to take any setting
from the top of the mythicalLibrarian file and make it perminant.
mythicalSetup will not change this file.  Your settings will
remain static after upgrades.

globalSettings can be applied by creating a file called:
/etc/mythicalLibrarian/globalSettings
##############################################################
#SYMLINK=Disabled                                            #
#AlternateMoveDir=/media/usbDrive                            #
#NotifyUserName=adam                                         #
#XBMCIPs=( user:[email protected]:8080 )                #
#Database=Disabled                                           #
##############################################################
Done.
You would want to type something like:
Code:
echo "MoveDir='/media/recordings/Episodes'
AlternateMoveDir='/home/ubuntite/Episodes'
PrimaryMovieDir='/media/recordings/Movies'
AlternateMovieDir='/home/ubuntite/Movies'
PrimaryShowDir='/media/recordings/Showings'
AlternateShowDir='/home/ubuntite/Showings'
SYMLINK=Disabled">/etc/mythicalLibrarian/globalSettings
globalSettings will override all other settings.

What flavor/version of POSIX are you running? I can't reproduce most of these errors you are having. "~" should always expand to the person who called it. It's echo'd directly into mythicalLibrarian, without any sort of expansion.
outleradam Wrote:What flavor/version of POSIX are you running? I can't reproduce most of these errors you are having. "~" should always expand to the person who called it. It's echo'd directly into mythicalLibrarian, without any sort of expansion.

Yes, "~" is expanding properly. The problem is that I never actually am logged in as mythtv, so when I upgrade, I am logged in as 'username', and run 'sudo mythicalLibrarian --upgrade'. It resolves '~' to /home/username. Those alternate dirs don't exist there, because mythicalLibrarian is never run as "username".

What are the alternate dirs for anyway? Maybe I should change them to not be dependent on the user running mythicalLibrarian.

I am running the latest version of mythbuntu:

Linux 2.6.35-22-generic #35-Ubuntu SMP Sat Oct 16 20:36:48 UTC 2010 i686 GNU/Linux
You can always use original and let mythbuntu distribute the files across your selected recordings dirs then just share your selected recordings dirs.

I created the alternates so that when running a NAS it would use the alternate in case the NAS became disconnected.

Are you able to grab any recordings data now? I can't understand why it would pull title and nothing else.

Mythbuntu should be running 0.24 or 0.25head. Either of those require python.
outleradam Wrote:You can always use original and let mythbuntu distribute the files across your selected recordings dirs then just share your selected recordings dirs.

Not a good solution for me. The drive on my mythbuntu machine is very small (by design) and that is why I want to record to it, then move the file to my NAS, then delete the original recording.

Quote:I created the alternates so that when running a NAS it would use the alternate in case the NAS became disconnected.

That's what I thought, and that is how I was using it.

Quote:Are you able to grab any recordings data now? I can't understand why it would pull title and nothing else.

Just reconfigured. Filesystem check now passes. Does this help:

Code:
Critical Filesystem checks ***PASSED***
press any key to test mythicalLibrarian...

Recordings Database is not used for testing.
Performing fuzzy logic lookup on database file.  Please wait.
It may take a moment to build mythical librarian database files
executing mythicalLibrarian testfile.ext 'mister rogers' 'show 1332'
testing for exit status 0
Revision 754 Thu Nov 4 21:21:14 PDT 2010
RECSTART: DATE:
PROGRAMID: ShowCategory:
PLOT:
agrep: 1 word matches within 2 errors
FUZZY LOGIC SHOW NAME: Mister Rogers' Neighborhood ID: 77750
agrep: 1 word matches within 1 error
COULD NOT MATCH AIRDATE. FUZZY LOGIC Exx NUMBER:462
TOTAL MATCHES:12 TOTAL VERIFIED FUZZY ERRORS:0
EPISODE:Show 1332 ABSOLUTE NUMBER:462 S07E07
JOB: mythicalLibrarian 'testfile.ext' 'mister rogers' 'show 1332'
chmod: changing permissions of `/media/recordings/Episodes/Mister Rogers Neighborhood': Operation not permitted
chmod: changing permissions of `/media/recordings/Episodes/Mister Rogers Neighborhood/Season 7': Operation not permitted
TARGET SET:/media/recordings/Episodes/Mister Rogers Neighborhood/Season 7/Mister Rogers Neighborhood S07E07 (Show 1332)
exited with status 0
press any key to continue on to the final results...

  All tests complete            RESULTS
  MythTV Database Test:      ***PASSED***
  Installed Packages:        ***PASSED***
  Notifications Test:        ***PASSED***
  XBMC Communications:       ***PASSED***
  File System Checks:        ***PASSED***
  mythicalTesting:           ***PASSED***
  Overall mythicalReadiness: ***PASSED***
-----End of diagnostics-----
Installation and tests completed successfully.
MythTV job not added because mythicalLibrarian already exists in slot 1
permissions were set for user: mythtv.
RSS Feed will be located at http://192.168.0.100/mythical-rss/rss.xml .
mythicalLibrarian is located in /usr/local/bin
mythicalLibrarian's log is located in ~/.mythicalLibrarian/output.log
User specified folders are to be used for placement of recordings.
Check the mythicalLibrarian checkbox when setting up recordings.
For more information, type 'mythicalLibrarian --help'
Done.


Quote:Mythbuntu should be running 0.24 or 0.25head. Either of those require python.

That's what I thought too, but Synaptic is reporting 0.23. I upgraded from mythbuntu 10.4 to 10.10 beta. Could that somehow have prevented mythtv from updating to 0.24?
The problem seems to be grabbing information from the database. It may be possible that the DB Schema was upgraded without the version being upgraded. If so, try using the Python Bindings. It was said that early in the 0.25 development cycle, the database would change to the point where it would break mythicalLibrarian. The Python Bindings will take care of that. Try
Code:
su mythtv
/usr/local/bin/MythDataGrabber --Diagnostic=True

This should connect to the database, identify itself, identify the database and then disconnect.


If that works try
Code:
usr/local/bin/MythDataGrabber NAME_OF_MYTHTV_FILE.mpg
which should dump all of the information out of the database about a file.

The other way would be

Code:
mysql -umythtv -pmythtv -e 'use mythconverg; select * from recorded where basename like "NAME_OF_MYTHTV_FILE.mpg";'

I would really like to get to the bottom of this. I planned ahead on releasing the MythDataGrabber python bindings by more then a full version of release time to ensure proper database connectivity would not lapse on mythicalLibrarian. For some reason you're not getting database information though.

If it comes down to it, I can always SSH in to get the proper information sometime this weekend so that we can get you up and avoid this issue by using mythicalSetup in the future.
outleradam Wrote:Try
Code:
su mythtv
/usr/local/bin/MythDataGrabber --Diagnostic=True

This should connect to the database, identify itself, identify the database and then disconnect.

OK, here goes:
Code:
sudo -u mythtv /usr/local/bin/MythDataGrabber --Diagnostic=True
Establishing database connection
Connected to: Revo
Db Identified as: sql://[email protected]:3306/
Diagnostics passed.

Quote:If that works try
Code:
usr/local/bin/MythDataGrabber NAME_OF_MYTHTV_FILE.mpg
which should dump all of the information out of the database about a file.

Alright:
Code:
sudo -u mythtv /usr/local/bin/MythDataGrabber /var/lib/mythtv/recordings/1009_20101105180000.mpg
Establishing database connection
Traceback (most recent call last):
  File "/usr/local/bin/MythDataGrabber", line 201, in <module>
    rec = db.searchRecorded(basename=options['filename']).next()
  File "/usr/lib/python2.6/dist-packages/MythTV/MythFunc.py", line 60, in __call__
    % (self.__name__, key))  
TypeError: searchRecorded got an unexpected keyword arguemnt 'basename'

It looks like we might be getting somewhere.
I really need to put in a schema check > version expected into the pybindings.

how about the mysql command?

Code:
mysql -umythtv -pmythtv -e 'use mythconverg; select * from recorded where basename like "1009_20101105180000.mpg ";'
outleradam Wrote:I really need to put in a schema check > version expected into the pybindings.

how about the mysql command?

Code:
mysql -umythtv -pmythtv -e 'use mythconverg; select * from recorded where basename like "1009_20101105180000.mpg ";'

Returns absolutely nothing...
ok... then lets try it this way....

Code:
x=`mysql -umythtv -pmythtv -e 'use mythconverg; select basename from recorded where starttime != "" ;'|sed -n 2p`
echo $x
mysql -umythtv -pmythtv -e "use mythconverg; select * from recorded where basename like '$x';"

and you should see a bunch of information at this point. If not, try to change that sed to sed -n 3p
Code:
[font=monospace]myfile=`locate $x|sed -n 1p[/font]`
echo $myfile[font=monospace]
MythDataGrabber $myfile
mythicalLibrarian $myfile[/font]
Please copy the output of this into a pastebin.