[outleradam]

In this particular example, that you posted, mythicalLibrarian was working properly. It did not delete anything because it was not able to obtain information from theTvDb and categorize the program properly. I modified TheTvDb to have a zap2it ID, which properly identifes Conan as Conan(2010), instead of Conan the barbarian.

Another fix for you would have been to apply a showTranslations (see mythicalLibrarian --special for an example). showTranslations allows you to tell mythicalLibrarian Conan = Conan (2010), this forces mythicalLibrarian to search for the proper name as defined in TheTvDb.com's database.


Can you pastebin your entire output.log (copy and paste it onto www.pastebin.com and post the link here). I would like to see what's happening. In your example mythicalLibrarian was working properly. I can go through your logs and see what's going on.


As for automating --doovers, you can create a file called ~/crontab.txt

Code:

0 2 * * * /usr/local/bin/mythicalLibrarian --unmannedDoover

This says At: 2AM on all days on all months on all years, run mythicalLibrarian --unmannedDoover.

then run

Code:

crontab ~/crontab.txt
crontab -l

Verify everything works and make changes as necessary (I'm not at a linux computer right now)

[outleradam]

Just making notes on a code review.

lookup by airdate appears to have an extra return in it, causing inoperation
Remove option for useshowname as dir
refactor useoriginaldir as something more descriptive like setTargetFolderAsInputPathFolder
ignorelist whitespace, maintenance routine whitespace. Pretty much alll whitespace needs to be reviewed.


I still want the output.log, ubuntite. I am just making notes on things I saw wrong because they cannot be fixed right now.

[ubuntite]

Here you go...

http://pastebin.com/TvxiF0xu

[outleradam]

Reafactored UseOriginalDir as TargetPathIsInputPath. Make a change if you have this option specified in /etc/mythicalLibrarian/globalSettings

[outleradam]

mythicalLibrarian Revision 777 fixed movie naming conventions.

Previously, movies would be named Title (airdate).ext. If no airdate was present, it would be Title ().ext. Now, no airdate will be simply Title.ext

7 is heaven

[outleradam]

Ubuntite: can you verify that this show made it into your library?

Code:

/media/recordings/Episodes/CTV News/CTV News S2010E1111 ( Recorded  [email protected]).mpg

Towards the end of your log I see

Code:

[LIST=1]
[*]@@@@@@@@@@@@@OPERATION COMPLETE Fri Nov 12 21:03:04 PST 2010 @@@@@@@@@@@@@@@@
[*]@@@@@@@@@@@@@OPERATION COMPLETE Fri Nov 12 21:07:14 PST 2010 @@@@@@@@@@@@@@@@
[/LIST]

I've never seen this in my logs. Did you splice two together? I have seen two New search initiated before, but never two operation complete.


I don't see any remarks about Updater Activated. This is usually the first thing I see in my logs.

Please update to revision 777 and see if the problem persists.

I noticed that there are some of your episodes like "Powell Family" which fuzzy did not jive with original airdate, but the recognition was perfect. This means that fuzzy logic would have failed and matched the wrong episode. eg...

Code:

Pow
Powell Family

Would have matched Pow first.

So, I will improve the fuzzy logic routine to first "grep --line-regexp $subtitle" and if that fails, then agrep the subtitle. This seems like a much better option and it could be applied to Title as well.

[omobeanz]

First off, great work on this. It will be useful if I can get it working. I have been trying to get this to work for about a week with mixed results. Currently on 783 (latest). I would appreciate any help with these issues.

1) To run manually you need to "sudo su mythtv". This leads to this error:
Do not run this command with sudo.
Logging in as mythtv or modifying the script fixes this, but it is quite annoying.

2) The globalSettings file is completely ignored when running an update and all questions are asked again.

3) The --doover command does not ever reprocess any of these unrecognized showings:

Code:

"HANDLING EPISODE AS GENERAL SHOWING AND DUMPING INTO --doover USER JOB"

Code:

$ mythicalLibrarian --doover
--You have entered doover mode.
In this mode of operation, mythicalLibrarian will rerun
mythicalLibrarain against all files which failed on the
last run of mythicalLibrarian.  It is important that this
process does not get interupted or data loss may occour.
  Press 'y' to continue or 'n' to exit..... y/(n):>ydone.

4) Sometimes, when run with exactly the same set of data, the series Scrubs is correctly identified. On other runs it is identified as Wormwood Scrubs. These are runs with exactly the same version if the script directly one after the other. The inconsistency is baffling...

5) My shows, including the test show that is run during the setup are very rarely recognized. This is not due to my internet connection, which is perfectly fine. They all usually end up with S0E0.

For example, the series Entourage has an entry to look up, and the episodes exist but nothing is returned:
http://www.thetvdb.com/api/6DF511BB2A64E...all/en.xml

Code:

@@@@@@@@@@@NEW SEARCH INITIATED AT Wed Nov 17 12:46:29 PST 2010@@@@@@@@@@@@@@
Revision 783 Wed Nov 17 11:18:18 PST 2010 on Linux
PERFORMING MAINTENANCE ROUTINE
SEARCHING: www.TheTvDb.com SHOW NAME: Entourage EPISODE: Oh, Mandy
DATE: 2005-07-24 FILE NAME: /media/shared/mythtv/recordings/1878_20101116210000.mpg
SEARCH FOUND: ID#:
MATCH FOUND BASED ON Zap2itID: ID:
SERIES ID WAS NOT FOUND. THETVDB MAY BE DOWN. TRY USING A showTranslations
DB ENTIRES- RECSTART:2010-11-16 21:00:00- MOVIE:- ORIGAIRDATE:2005-07-24- CHID:1878- CAT:Comedy-
###################DEBUG MODE
ENABLED####################
MY USER NAME:mythtv-
LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.
INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/mythtv/.mythicalLibrarian-
MOVE DIR:/media/shared/mythtv/Episodes-
FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/mythtv/FailSafe- ALTERNATE MOVE DIR:/media/shared/mythtv/Episodes-
USE ORIGINAL DIR:Disabled NOTIFICATIONS:Disabled
INPUT SHOW NAME:Entourage- LOCAL SHOW NAME TRANSLATION:- ShowName:Entourage
DATABASE UPDATED:- TVDB LAST UPDATED:- CURRENT:-
RESOLVED SERIES ID:- RESOVED SHOW NAME:-
INPUT EPISODE NAME:Oh, Mandy- ABSOLOUTE EPISODE NUMBER:- RESOLVED EPISODE NAME:Oh, Mandy-
SEASON:- EPISODE:- SYMLINK MODE:LINK- FILESIZE: 0kB-
CREATE AND DELETE FLAGS: ORIGINALDIR:1- FREE:266464032kB- WORKINGDIR:1 FREE:126996700kB-
MOVEDIRWRITABLE:1- FREE:266464032kB- ALTERNATEMOVEDIR:1- FREE:266464032kB-
PRIMARYSHOWDIRWRITABLE:1-  FREE:266464032kB-ALTERNATESHOWDIRWRITABLE:1- FREE:266464032kB-
PRIMARYMOVIEDIRWRITABLE:1- FREE:266464032kB- ALTERNATEMOVIEDIR:1- FREE:266464032kB-
DATABASE TYPE:schedulesdirect1-
RECSTART:2010-11-16 21:00:00- MOVIE YEAR:- ORIGINAL SERIES DATE:2005-07-24-
PROGRAMID:EP006654200017- CHANNEL ID:1878- CATEGORY:Comedy- GOFORDOOVER:0-
EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:Series With Episode Data- STARS:0 RATING:0
ZAP2IT SERIES ID:665420- MATCHED TVDB SERIES ID:-
PLOT: An actress exposes a secret from Vince's past; Ari gets bumped from an important list; Drama and Turtle try to enjoy their new house.
####################END OF DEBUG LOG#####################
WARNING:SHOW WAS CONSIDERED AN EPISODE BUT INFORMAITON WAS NOT OBTAINABLE
%%%%%%%%%%%%Please consider helping out and adding to thetvdb%%%%%%%%%%%%
HANDLING EPISODE AS GENERAL SHOWING AND DUMPING INTO --doover USER JOB
Symlink created /media/shared/mythtv/Showings/Entourage/Entourage S0E0 (Oh, Mandy Recorded [email protected] on 1878)-3.mpg
@@@@@@@@@@@@@OPERATION COMPLETE Wed Nov 17 12:46:30 PST 2010 @@@@@@@@@@@@@@@@

[omobeanz]

First off, great work on this. It will be useful if I can get it working. I have been trying to get this to work for about a week with mixed results. Currently on 783 (latest). I would appreciate any help with these issues.

1) To run manually you need to "sudo su mythtv". This leads to this error:
Do not run this command with sudo.
Logging in as mythtv or modifying the script fixes this, but it is quite annoying.

2) The globalSettings file is completely ignored when running an update and all questions are asked again.

3) The --doover command does not ever reprocess any of these unrecognized showings:

Code:

"HANDLING EPISODE AS GENERAL SHOWING AND DUMPING INTO --doover USER JOB"

Code:

$ mythicalLibrarian --doover
--You have entered doover mode.
In this mode of operation, mythicalLibrarian will rerun
mythicalLibrarain against all files which failed on the
last run of mythicalLibrarian.  It is important that this
process does not get interupted or data loss may occour.
  Press 'y' to continue or 'n' to exit..... y/(n):>ydone.

4) Sometimes, when run with exactly the same set of data, the series Scrubs is correctly identified. On other runs it is identified as Wormwood Scrubs. These are runs with exactly the same version if the script directly one after the other. The inconsistency is baffling...

[omobeanz]

5) My shows, including the test show that is run during the setup are very rarely recognized. This is not due to my internet connection, which is perfectly fine. They all usually end up with S0E0.

For example, the series Entourage has an entry to look up, and the episodes exist but nothing is returned:
http://www.thetvdb.com/api/6DF511BB2A64E...all/en.xml

Code:

@@@@@@@@@@@NEW SEARCH INITIATED AT Wed Nov 17 12:46:29 PST 2010@@@@@@@@@@@@@@
Revision 783 Wed Nov 17 11:18:18 PST 2010 on Linux
PERFORMING MAINTENANCE ROUTINE
SEARCHING: www.TheTvDb.com SHOW NAME: Entourage EPISODE: Oh, Mandy
DATE: 2005-07-24 FILE NAME: /media/shared/mythtv/recordings/1878_20101116210000.mpg
SEARCH FOUND: ID#:
MATCH FOUND BASED ON Zap2itID: ID:
SERIES ID WAS NOT FOUND. THETVDB MAY BE DOWN. TRY USING A showTranslations
DB ENTIRES- RECSTART:2010-11-16 21:00:00- MOVIE:- ORIGAIRDATE:2005-07-24- CHID:1878- CAT:Comedy-
###################DEBUG MODE
ENABLED####################
MY USER NAME:mythtv-
LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.
INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/mythtv/.mythicalLibrarian-
MOVE DIR:/media/shared/mythtv/Episodes-
FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/mythtv/FailSafe- ALTERNATE MOVE DIR:/media/shared/mythtv/Episodes-
USE ORIGINAL DIR:Disabled NOTIFICATIONS:Disabled
INPUT SHOW NAME:Entourage- LOCAL SHOW NAME TRANSLATION:- ShowName:Entourage
DATABASE UPDATED:- TVDB LAST UPDATED:- CURRENT:-
RESOLVED SERIES ID:- RESOVED SHOW NAME:-
INPUT EPISODE NAME:Oh, Mandy- ABSOLOUTE EPISODE NUMBER:- RESOLVED EPISODE NAME:Oh, Mandy-
SEASON:- EPISODE:- SYMLINK MODE:LINK- FILESIZE: 0kB-
CREATE AND DELETE FLAGS: ORIGINALDIR:1- FREE:266464032kB- WORKINGDIR:1 FREE:126996700kB-
MOVEDIRWRITABLE:1- FREE:266464032kB- ALTERNATEMOVEDIR:1- FREE:266464032kB-
PRIMARYSHOWDIRWRITABLE:1-  FREE:266464032kB-ALTERNATESHOWDIRWRITABLE:1- FREE:266464032kB-
PRIMARYMOVIEDIRWRITABLE:1- FREE:266464032kB- ALTERNATEMOVIEDIR:1- FREE:266464032kB-
DATABASE TYPE:schedulesdirect1-
RECSTART:2010-11-16 21:00:00- MOVIE YEAR:- ORIGINAL SERIES DATE:2005-07-24-
PROGRAMID:EP006654200017- CHANNEL ID:1878- CATEGORY:Comedy- GOFORDOOVER:0-
EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:Series With Episode Data- STARS:0 RATING:0
ZAP2IT SERIES ID:665420- MATCHED TVDB SERIES ID:-
PLOT: An actress exposes a secret from Vince's past; Ari gets bumped from an important list; Drama and Turtle try to enjoy their new house.
####################END OF DEBUG LOG#####################
WARNING:SHOW WAS CONSIDERED AN EPISODE BUT INFORMAITON WAS NOT OBTAINABLE
%%%%%%%%%%%%Please consider helping out and adding to thetvdb%%%%%%%%%%%%
HANDLING EPISODE AS GENERAL SHOWING AND DUMPING INTO --doover USER JOB
Symlink created /media/shared/mythtv/Showings/Entourage/Entourage S0E0 (Oh, Mandy Recorded [email protected] on 1878)-3.mpg
@@@@@@@@@@@@@OPERATION COMPLETE Wed Nov 17 12:46:30 PST 2010 @@@@@@@@@@@@@@@@

[outleradam]

1. You don't run sudo mythicalLibrarian filename.ext. You run just mythicalLibrarian. Only sudo when it asks you to run sudo. The sudo commands are just used when mythicalLibrarian requires the permissions to access files or create files in special areas.

2. globalSettings. There are 3 levels of configuration. 1. modification of the script, will last until the next update. 2. mythicalSetup will remember the last value and recall it for the next setup 3. globalSettings will persist thoughout updates. globalSettings is used to override any other settings produced in updates.

3. Each use maintains his/her own database. This is done to prevent permission errors. mythtv can't access root's files and xbmc can't access mythtv created files. So the work-around is for each user to have his or her own data to manage the files that they created.


4. I'm revising the logic right now. If you look at the logs you can see a new addition, the confidence rating and confidence reasoning. Confidence rating will give a correct factor. Reasoning will show the steps which were took to identify the episode. This is a work in progress. I'll post here when this is done.

SVN should be considered unstable right now... I need to get to a steady point where I can commit. Stable is about 2 months old now.

[omobeanz]

Thanks outleradam. I'm hoping these comments might help you find and fix some bugs Besides, the stable version does not work for me either because for some reason it thinks all my directories do not exist. I'll have to keep looking into that.

1. Understood, but a lot of people will not log into their system as the mythtv user, in which case you would use "sudo su mythtv" to become the mythtv user. This is mentioned in your documentation. I do not use the sudo command to run the script once I am the mythtv user, but the earlier user switch is still detected.

Code:

user1@ubuntu-htpc:~$ sudo su mythtv
[sudo] password for user1:
mythtv@ubuntu-htpc:/home/user1$ whoami
mythtv
mythtv@ubuntu-htpc:/home/user1$ mythicalLibrarian -s mpg "/media/shared/mythtv/recordings"
Do not run this command with sudo.

Side note -> the --scan option no longer works, only -s

2. So you mean that what is written in the globalSettings file is read first on every run of the script or that it is meant to set the defaults in the script during setup?

3. I'm not sure I understand. I run the script as mythtv, then nothing is recognized so they are made into doover jobs. Then I run the doover as mythtv again and nothing happens. I'm not changing users at this point.

4. Sounds good to me Odd that it has different results on runs directly one after the other though.

[omobeanz]

Another thing to mention, when running the setup there is no option that will set SYMLINK=LINK which seems like it should be the default especially for the SVN version as it has the least affect on the system. I have already messed up all my recordings so far.

Code:

test $? = 0 && echo "SYMLINK=MOVE" >> ./mythicalPrep || echo "SYMLINK=Disabled" >> ./mythicalPrep

[outleradam]

^^ I don't really support Link Mode as an option. The reason is because there is difficulties across networks with windows computers and first time Linux users. MOVE mode is the only sure-fire method

I was not able to reproduce the side note, but issue 1 and 3 is fixed

The globalSettings was created on request of a dev at knoppmyth. They were considering including it as part of their distribution. It is designed to add consistency through updates. It overrides any settings at runtime. globalSettings is loaded after User Settings when the script is run.

I will check out the doover issue. I made a revision on that and will inspect it later this week.

Are you per chance running agrep and tre-agrep on the machines?

[outleradam]

Do not update to SVN. I am revising several points right now. SVN is extremely unstable.

This is part 2 of the top down rewrite of the episode matching and I'm improving other functions as I go along.

[ubuntite]

Do not update to SVN. I am revising several points right now. SVN is extremely unstable.

This is part 2 of the top down rewrite of the episode matching and I'm improving other functions as I go along.

Any work being done on SYMLINK=Disabled deleting db entries? My db is still not being touched.