• 1
  • 123
  • 124
  • 125(current)
  • 126
  • 127
  • 197
[RELEASE] Texture Cache Maintenance utility
(2016-02-05, 02:05)Milhouse Wrote:
(2016-02-05, 01:08)McButton Wrote: And, lastly...is there a command to find duplicate episodes? Or, does duplicate cover the whole db? Seems like it just returns Movies.

No, "duplicates" is just for movies - it uses imdbnumber to determine a duplicate.

Tvshows don't have imdbnumbers, and I don't think the media library database supports multiple paths for the same tvshow/season/episode so I'm not sure how it would be possible to accurately determine that some random file path is a duplicate of an episode. If you've got any ideas let me know.

Ah. I assumed it picked name. I think all of my movies are scraped via TMDB id. TV shows have some sort of IMDB listing. The episodes are individualized too...but they're kinda unstable. I see shows often that don't have a full listing. It may be difficult by name due to listing extensions for episodes. You'd have to make the utility not only match and ignore the show as a duplicate, it would then have to find duplicated naming within that.
Like...
TVSHOWNAME - S01E02 - TITLE
TVSHOWNAME - S01E03 - TITLE (would maybe match due to same titles and titles are rarely covered with the same season/episode info. You could also do the same to match S01E02 -TITLE & S01E02 - TITLE2 = Match due to the season/episode listing. Depends on what you're looking to find as a duplicate.
I say, if nobody has had concern until me, let it ride. I'm over/under a few episodes. I'll figure it out. The Missing Movie scanner works well enough for it. If there isn't an .nfo listed, it's easy to fix....but sometimes.........you have one jerky file that just stays elusive.
Reply
(2016-02-05, 02:52)McButton Wrote: Ah. I assumed it picked name. I think all of my movies are scraped via TMDB id. TV shows have some sort of IMDB listing. The episodes are individualized too...but they're kinda unstable. I see shows often that don't have a full listing. It may be difficult by name due to listing extensions for episodes. You'd have to make the utility not only match and ignore the show as a duplicate, it would then have to find duplicated naming within that.
Like...
TVSHOWNAME - S01E02 - TITLE
TVSHOWNAME - S01E03 - TITLE (would maybe match due to same titles and titles are rarely covered with the same season/episode info. You could also do the same to match S01E02 -TITLE & S01E02 - TITLE2 = Match due to the season/episode listing. Depends on what you're looking to find as a duplicate.
I say, if nobody has had concern until me, let it ride. I'm over/under a few episodes. I'll figure it out.

Unfortunately it's not very reliable - not everyone has titles in their filenames. Matching on season/episode might be possible, but then you're having to employ the same kind of regular expressions that Kodi uses to parse season and episode from the filename and they're not trivial (so many different formats). It's a lot of work for potentially very little gain (you might be the second person to ask).

(2016-02-05, 02:52)McButton Wrote: The Missing Movie scanner works well enough for it. If there isn't an .nfo listed, it's easy to fix....but sometimes.........you have one jerky file that just stays elusive.

Have you tried the texturecache.py "missing" option?

Code:
texturecache.py missing tvshows <tv source>
eg.
Code:
texturecache.py missing tvshows "my tv shows"
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2016-02-05, 02:23)Milhouse Wrote:
(2016-02-05, 01:26)guido1138 Wrote: As an aside, I've seen the "nc" function, which shows required artwork that is not cached. I wonder if we could have a converse function like "oc" (e.g. over-cached), which shows non-required artwork that is cached. Or perhaps even a sort of modified "C" function that doesn't ignore some artwork removal?

Thanks!

What's over-cached for you is not over-cached for someone else - how to tell?

This has nothing to do with me or someone else. As I explained clearly in my post, "required" refers to the artwork types defined in the cache.artwork property. That's how you tell what's unnecessarily cached. This is the same the terminology you yourself use in the README so I assumed it would be clear:
Code:
[nc] Identify those items that require caching (and would be cached by c option)

As I also explained my post, a hypothetical "oc" command would be the inverse of "nc", showing cached items which wouldn't normally be cached using the "c" command and according to the cache.artwork propoerty.

(2016-02-05, 02:23)Milhouse Wrote: Have a look at "purge" (and "purgetest" for testing) to remove artwork en mass. Give it the partial path for the cached artwork and it will remove them, eg.
Code:
texturecache.py purgetest all %clearart.png %logo.png %banner.jpg
substitute purge for purgetest to perform the actual deletion once you're happy with what it matches.
I already looked at the purge command but since it only scans the texture DB it wont work since most online-scraped artwork doesn't match fixed patterns like your example. That's why I was looking at the "c/nc/C" command family which cross-check the media library against texture DB.

Hopefully this is all clear now.
Reply
Thanks for the clarification, I'll think about it, but ultimately if your cache is so small that you can't spare the space for clearart/logo artwork then a) why are you scraping this artwork and b) you should use Path Substitution to relocate the Thumbnails folder to network or USB/SD storage.

The "prune" option gets rid of the dross that accumulates over time in the cache, but artwork that is in your media library is, almost by definition, not dross, which is why there's not (currently) an option to bulk remove artwork by type.

One option would be to add a prune property that prevents artwork types from being matched in the media library, something like "prune.artwork.delete = clearart, logo, banner" - with that added, next time you run "P" all of your clearart, logo and banner artwork (for any media type - movies, tvshows, whatever) would be deleted from the cache.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Perhaps if the cache.artwork property is set, eg.:
Code:
cache.artwork = poster, fanart, landscape, thumb, season.poster

then logically, while pruning, any artwork type that wouldn't otherwise be cached by c/C (because it's not in the cache.artwork list) should be removed during the prune - eg. clearart, logo, banner would all be pruned as they're not being cached by c/C, but poster, fanart, landscape, thumb and season.poster would be retained while pruning as they are being cached by c/C.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2016-02-05, 11:25)Milhouse Wrote: Perhaps if the cache.artwork property is set, eg.:
Code:
cache.artwork = poster, fanart, landscape, thumb, season.poster

then logically, while pruning, any artwork type that wouldn't otherwise be cached by c/C (because it's not in the cache.artwork list) should be removed during the prune - eg. clearart, logo, banner would all be pruned as they're not being cached by c/C, but poster, fanart, landscape, thumb and season.poster would be retained while pruning as they are being cached by c/C.

That would work. It definitely makes more sense to extend the pruning functions ("p/P") instead of modifying a caching function ("nc/oc") as I suggested earlier. But rather than conflating caching and pruning by overloading the cache.artwork property, I think I prefer your previous idea of having a prune-specific property, like:
Code:
prune.artwork.delete = clearart, clearlogo, discart, banner

This has the advantage of being clearer for users by explicitly specifying what to delete instead of "inferring the inverse set" from a cache.artwork property.

But honestly, I'd be happy to have the functionality regardless of how the parameters are set. So thanks for giving this some consideration.
Reply
(2016-02-05, 02:05)Milhouse Wrote: Tvshows don't have imdbnumbers, and I don't think the media library database supports multiple paths for the same tvshow/season/episode so I'm not sure how it would be possible to accurately determine that some random file path is a duplicate of an episode. If you've got any ideas let me know.

i sorted this out in bash by concatenating 'showID-season#-ep#' to form a uniqueID

the first query makes a list of any uniqueIDs with more then 1 entry. (find the dups)
then we loop though the LIST and query again for the path/file.name of each offender

hope this gets the creative juice flowing ...

Code:
#!/bin/bash
# xbmc.tv.duplicate ep detector
mysqlCMD="mysql --skip-column-names -u<USERNAME> -p<PASSWORD> <DB> -e"

LIST="$($mysqlCMD "SELECT CONCAT( idShow,  \"-\", c12,  \"-\", c13 ) AS id,
    COUNT( * ) AS num
    FROM episode_view
    GROUP BY id
    HAVING num >1" | cut -f1 | sort -n)"

for x in $LIST ; do
    $mysqlCMD "SELECT c18 from episode_view
            where CONCAT (idShow,  \"-\", c12,  \"-\", c13 ) = \"$x\" "
done

BTW, thank you for this awesome sauce !!
in the last 8yrs i've amassed 7.8gb of thumbnail cache
... now 5.2 after a "quick" prune
rPi 2&3 | android phones | fireHD8 | linux | win10 + NFS NAS w/ mySQL + props to...
libreElecyatse, titan, AELflexGet, context.manageTags (a zosky original)
Reply
Hi Milhouse

I'm finally getting to Do Battle with mklocal.py, and it's definitely winning Sad

Nothing will satisfy the "--local" parameter. Really, three hours of nothing, but...

Code:
mklocal.py: error: local DIRECTORY /FREENAS/media/ does not exist!

...and all the naming variations I could imagine, to define the NAS as seen "locally" by OpenELEC.

I'm sure it's a laughably simple thing I'm missing (being a Me-No-Linux guy), but the laughter stopped a long time ago Confused

The Movies source looks like this:

Image

but I have no clue (heh-heh, obviously) what "--local" is in terms of the script ?

FYI, I PuTTY in to the OpenELEC box to run the scripts (from a Windows desktop), if that's of any significance.

Please, help Stop the Madness !!
::  LibreELEC 9.2.6 RELEASE - Generic x86_64  ::  Intel 847 NUC  ::  KVR1333D3S9/4G  ::  Kingston SMS200S3/30G mSATA  ::  MS 1044 MCE keyboard  ::  GP-IR02BK remote  ::  Xonfluence  ::  10.9TiB on FreeNAS v11.3-U5 (RAID-Z2)  ::
Reply
"--local" needs to be the smb://FREENAS/media path as it is mounted by the client on which you're running mklocal.py.

For a Linux/OpenELEC machine, this would be the mount point. Since this is an smb path, you'd need to mount /FREENAS/media as a CIFS filesystem in OpenELEC, perhaps with autostart.sh or as part of whatever script you're using to run mklocal.py, then use the local mount path as the --local parameter.

Something like the following should work:
Code:
mkdir -p /storage/media
mount -t cifs -o username=UNAME,password=PWORD,sec=ntlm //FREENAS/media /storage/media

change UNAME and PWORD to be a Windows username and password that has read/write access to the media share. Then use "--local /storage/media".

When you're done running mklocal.py, use "umount /storage/media" to unmount the filesystem (no, umount isn't a typo).

If you were to run the script on Windows, assuming you have /FREENAS/media mapped as your Z drive then "--local Z:\" would work. Or you could use a UNC, ie. "--local \\FREENAS\media".
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi Milhouse

Thanks for quick reply. Let me apologize for the Linux-n00b exercise in advance...

mkdir = successful, but I'm stuck on the mount command (...let me count the hours Big Grin )

The operation fails with either this:
Code:
OpenELEC:~ # mount -t cifs -o username=*****,password=*****,sec=ntlm //FREENAS/media /storage/media
mount: bad address 'FREENAS'
mount: mounting //FREENAS/media on /storage/media failed

or - in desperation - this:
Code:
OpenELEC:~ # mount -t cifs //192.168.1.175/media/ -o username=*****,password=*****,sec=ntlm /storage/media
mount: mounting //192.168.1.175/media/ on /storage/media failed: Invalid argument

In the second case (ie: using explicit IP address) the device is recognized alright, but I can't tell what "Invalid arguement" is actually complaining about.

I'm curious why it's necessary to use the UNAME and PWORD options... IF permissions on the FreeNAS box are open (yeah, I know... next project). Kodi has full control of "media", and so presumably the scripts running on the Kodi box will have the same authority. Is that view too simple ?

Thanks.
::  LibreELEC 9.2.6 RELEASE - Generic x86_64  ::  Intel 847 NUC  ::  KVR1333D3S9/4G  ::  Kingston SMS200S3/30G mSATA  ::  MS 1044 MCE keyboard  ::  GP-IR02BK remote  ::  Xonfluence  ::  10.9TiB on FreeNAS v11.3-U5 (RAID-Z2)  ::
Reply
If you don't need the username/password options then drop them - I gave an example mount command, you'll need to figure out the specifics that your setup requires.

sec=ntlm usually fixes the "invalid argument" problem, if it doesn't work in this case then please start another thread - sorting out CIFS mount issues is OT for this thread. You could use a systemd mount service but again that's OT for this thread - read the OpenELEC wiki.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi Milhouse

Sorry for the OT... and thanks for the pointer to the systemd thing - that's got me going.

I've issued this command, and it's been a long time with no action in the terminal session window...
Code:
OpenELEC:~/.kodi/userdata # ./mklocal.py --singlefolders --local /storage/media
--prefix smb://FREENAS/media --artwork poster fanart clearlogo clearart banner l
andscape --verbose --output fixup.dat
Current configuration:

  Local Path    : /storage/media/
  Alt Local     : Not specified
  Kodi Path     : smb://FREENAS/media/
  Extra Fanart  : Not specified (max: 4)
  Extra Thumbs  : Not specified (max: 4)
  Read Only     : No
  Dry Run       : No
  Single Folder : Yes

  Artwork       : poster       as poster.[png,jpg]
                  fanart       as fanart.[png,jpg]
                  clearlogo    as logo.[png,jpg]
                  clearart     as clearart.[png,jpg]
                  banner       as banner.[png,jpg]
                  landscape    as landscape.[png,jpg]

  Checking      : Not specified

HAW-HAW... doing a Ctrl^C to copy the text from the PuTTY window cancelled the run - Oh Boy is it ever a LEARNING day Rofl

Now that mklocal.py is halted, should I be able to see the "fixup.dat" file, written to the same directory (ie: userdata), or is it only created at the end of a complete run ?

Also: my movies are in single folders, BUT the local artwork is named using the movie-prefix convention. Should I use --singlefolders switch or not if I wish to be consistent and retain that naming ?

Thanks (again !)
::  LibreELEC 9.2.6 RELEASE - Generic x86_64  ::  Intel 847 NUC  ::  KVR1333D3S9/4G  ::  Kingston SMS200S3/30G mSATA  ::  MS 1044 MCE keyboard  ::  GP-IR02BK remote  ::  Xonfluence  ::  10.9TiB on FreeNAS v11.3-U5 (RAID-Z2)  ::
Reply
mklocal.py is reading from stdin, you need to pipe the output of "texturecache.py jd movies" so that mklocal.py has something to process, ie.:

Code:
./texturecache.py jd movies | ./mklocal.py --singlefolders --local /storage/media --prefix smb://FREENAS/media --artwork poster fanart clearlogo clearart banner landscape --verbose --output fixup.dat

(2016-02-19, 03:01)gjwAudio Wrote: Also: my movies are in single folders, BUT the local artwork is named using the movie-prefix convention. Should I use --singlefolders switch or not if I wish to be consistent and retain that naming ?

Don't use --singlefolders if your artwork includes the movie name prefix. Try a dry-run to be sure.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Hi Milhouse

Here's me today - Image - but, I'm determined to learn the correct (and sensible) way to use your wonderful tools !

Firstly... now I realize that what I need is the Three Amigos - working together - to get my library & texturecache into shape (ie: texturecache.py jd /followed by mklocal.py blah-blah /followed by texturecache.py set)

I ran the first two steps (with --dryrun), and now am looking through the fixup.dat file. Very interesting. One issue I see, is what's happened in the "Movies_VTS" folder. There's a mess to fix before running this in update mode.

The folder structure is: /media/Movies_VTS/Movie-Name/VIDEO_TS/VIDEO_TS.BUP, VIDEO_TS.IFO, etc.

A couple "bad things" have happened here by my hand, and running the existing fixup.dat will make things even worse (in the VTS world). As an aside, perhaps this is a funky file-naming case you have not considered yet.

Let me illustrate using Defendor (2009)...

Artwork Downloader (in the past, before I knew better) was set to save images to the media folders - resulting in NON-movie-name-prefixed files, placed in the Movie-Name folder. However, the associated folders for .actors/ extrafanart/ extrathumbs are found one level down, within the main VIDEO_TS folder. Not cool, it seems to me.

Image

Worse... I used to think it "good" to export the Library my to the local folders, and would do this frequently - maybe it was Kodi that put them down there. At least one movie has both prefixed- and NON-prefixed- artwork duplicates. Gee, I don't rememer doing that by hand Tongue

Checking in fixup.dat, I find the script is planning to rename the Defendor files with prefix alright... BUT it's using VIDEO_TS as the Movie-Name element.
Code:
{
    "items": {
      "art.banner": "smb://FREENAS/media/Movies_VTS/Defendor (2009)/VIDEO_TS/VIDEO_TS-banner.jpg",
      "art.clearart": "smb://FREENAS/media/Movies_VTS/Defendor (2009)/VIDEO_TS/VIDEO_TS-clearart.png",
      "art.clearlogo": "smb://FREENAS/media/Movies_VTS/Defendor (2009)/VIDEO_TS/VIDEO_TS-logo.png"
    },
    "libraryid": 1902,
    "title": "Defendor",
    "type": "movie"
  },
Uh-oh...

With only 12 movies in VTS format, I can go through them manually to move & rename stuff without too much hassle, but will the scripts be OK with that naming convention in the future ?

Best approach (as I see it now) is to remove the Movies_VTS folder from the Library, correct the naming and location anomalies, and then "Scan for new content". In the context of a VTS rip, does it make sense to name the artwork as if there were a single video file (ie: with Movie-Name-prefix) ?

In your opinion, are there other practices I should adopt for keeping the artwork under control (I know, this is sorta asking "What do you do").

Thank You again, for ALL your help !!
::  LibreELEC 9.2.6 RELEASE - Generic x86_64  ::  Intel 847 NUC  ::  KVR1333D3S9/4G  ::  Kingston SMS200S3/30G mSATA  ::  MS 1044 MCE keyboard  ::  GP-IR02BK remote  ::  Xonfluence  ::  10.9TiB on FreeNAS v11.3-U5 (RAID-Z2)  ::
Reply
Hi,
I prefer musicvideos in Kodi Isengard. I'm using Windows 7.
My problem is that the artist artwork from the musicvideo isn't shown automatically by default in Kodi under the artist view.. To show my artist artwork I have to add the artist artwork manually with the context menu. It's a lot of work.
Fanarts and thumbs are working well and shown from Kodi automatically.

I use self created .nfo files for my musicvideos, I doesn't use any scraper for my musicvideos. So there are no musicbrainz information stored in my library. There ist also no "id" in the .nfo file and no "id" in my library.

I run python in a portable Version. The path to my portable Python is:
C:\Users\xxx\Downloads\kodi texture cache maintenance utility\python-3.5.1-embed-win32\python.exe

The Path to the "texturecache.py" and "texturecache.cfg" is:
C:\Users\xxx\Downloads\test\

To execute python with the "texturecache.py" I run / type for example in the windows command line:
First under C:\Users\xxx> cd Downloads\kodi texture cache maintenance utility\python-3.5.1-embed-win32
Than its shown C:\Users\xxx\Downloads\kodi texture cache maintenance utility\python-3.5.1-embed-win32>
Than I type python.exe C:\Users\xxx\Downloads\test\texturecache.py c

The "texturecache.py" is working well and the "texturecache.cfg" settings are right.

The path to my Kodi database:
C:/Users/xxx/Appdata/Kodi/userdata

The path to my local media folder (scanned into my library):
C:/Users/xxx/Downloads/mp4/
AC-DC/
poster.jpg (artist artwork)
AC-DC - Hells Bells/
AC-DC - Hells Bells.mp4 (musicvideo)
AC-DC - Hells Bells.nfo (self created .nfo)
AC-DC - Hells Bells.tbn (thumbnail)
fanart.jpg

Adele/
poster.jpg (artist artwork)
Adele - Hello/
Adele - Hello.mp4 (musicvideo)
Adele - Hello.nfo (self created .nfo)
Adele - Hello.tbn (thumbnail)
fanart.jpg

and so on !!

I've read about the "texture cache maintenance utility". Is there a way to populate / add with this tool my poster.jpg (artist artwork) automatically to my library so that the artist artwork (Artist Picture) is shown automatically in Kodi and I have not to do this manually?

And if this is possible, please write the code that I have to use. I'm a noob. Please help!!
Reply
  • 1
  • 123
  • 124
  • 125(current)
  • 126
  • 127
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17