• 1
  • 152
  • 153
  • 154(current)
  • 155
  • 156
  • 197
[RELEASE] Texture Cache Maintenance utility
Hi,

I have a problem I would like some advice on how to resolve.

My entire Library sits inside my nginx webserver.

When scanning using the normal process it is fine.

I have noticed when performing the command qax I see these errors in my kodi.log

Quote:09:58:41 T:1713718304 WARNING: No information found for item 'http://10.8.0.1:40080/Movies/2017The.Wall.2017.REPACK.720p.BluRay.x264-DRONES.mkv', it won't be added to the library.
09:58:41 T:1713718304 WARNING: No information found for item 'http://10.8.0.1:40080/Movies/2017The.Zookeepers.Wife.2017.720p.BRRip.X264.AC3-EVO.mkv', it won't be added to the library.
09:58:42 T:1713718304 WARNING: No information found for item 'http://10.8.0.1:40080/Movies/2017to.the.bone.2017.720p.webrip.x264-strife.mkv', it won't be added to the library.

And they repeat for all movies.

The actual data structure inside this folders is also just a bunch of movies year:

Quote:root@mediaserver:/data/Movies# ls
1931 1940 1944 1948 1952 1956 1960 1964 1968 1972 1976 1980 1984 1988 1992 1996 2000 2004 2008 2012 2016
1933 1941 1945 1949 1953 1957 1961 1965 1969 1973 1977 1981 1985 1989 1993 1997 2001 2005 2009 2013 2017
1934 1942 1946 1950 1954 1958 1962 1966 1970 1974 1978 1982 1986 1990 1994 1998 2002 2006 2010 2014 Christmas
1937 1943 1947 1951 1955 1959 1963 1967 1971 1975 1979 1983 1987 1991 1995 1999 2003 2007 2011 2015
root@mediaserver:/data/Movies#

It appears that the slash after the year folder isn't present? im not sure why.

Shows in logs: http://10.8.0.1:40080/Movies/2017The.Wal...DRONES.mkv
Should be like: http://10.8.0.1:40080/Movies/2017/The.Wall.2017.REPACK.720p.BluRay.x264-DRONES.mkv

The source entry is fairly standard:

Quote:root@osmc:/home/texturecache.py# python texturecache.py sources
video: Movies: http://10.8.0.1:40080/Movies

So I am not so sure it is scanning or checking things quite correctly.

The reason for this is I notice it is identifying missing artwork, but subsequently doesn't seem to fix it in some examples.

Please help. Thank you.
Reply
@yhs4260 can you run your qax command with "@logfile=tc.log" then upload tc.log somewhere, and send me the link.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Sent in a pm thanks Milhouse.
Reply
Hello everyone.

Have been playing around with the program and love it. Have noticed that when running 'c' scan all items are skipped because they have already been cached but it is not the exact number of movies I have in the folder i.e. lets say I have 100 files but 97 posters and 98 fanart are skipped. Tried cleaning with vclean and qax checks. Any other ideas or should I just leave it alone? Is there a way to find out which ones are missing and 'C' fix them.

I have also noticed in tvshows output that three images wont download from imdb. I have tried to 'C' the specific show and it returns no episode images. I can extract error output if required. Cheers.
Reply
(2017-08-30, 10:34)yhs4260 Wrote: Sent in a pm thanks Milhouse.
sorry yhs4260, haven't forgotten - been rather busy. Will get onto it hopefully later today.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2017-09-05, 04:14)80njc80 Wrote: Hello everyone.

Have been playing around with the program and love it. Have noticed that when running 'c' scan tall items are skipped because they have already been cached but it is not the exact number of movies I have in the folder i.e. lets say I have 100 files but 97 posters and 98 fanart are skipped. Tried cleaning with vclean and qax checks. Any other ideas or should I just leave it alone? Is there a way to find out which ones are missing and 'C' fix them.

If "nc movies" says there's nothing to cache, then chances are there's nothing to cache. If you have two movies that share the same poster then the poster will only be cached once which could explain the difference between the number of movies and number of cached posters. Otherwise enable logging during "c movies" by adding "@logfile=tc.log" then look in tc.log to see if there's any explanation why a particular poster is skipped (it may not be obvious in which case upload your tc.log file and post me the link).

(2017-09-05, 04:14)80njc80 Wrote: I have also noticed in tvshows output that three images wont download from imdb. I have tried to 'C' the specific show and it returns no episode images. I can extract error output if required. Cheers.

Post the urls that are failing to download. I assume you've checked and the urls are still valid (ie. you're able to view the images in your browser). If they're no longer valid (returning 404 or 403 errors etc.) then you'll need to find new urls by refreshing the corresponding tvshows or episodes so that new image urls are scraped.

Note that this script is not a metadata scraper, it only works with whatever information the scrapers find - if that information becomes stale and invalid (a common problem when using remote artwork) then there's no alternative but to rescrape the stale items (ie. refresh an episode in the Kodi GUI).
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Thank you Milhouse. That information allows me to move forward. Cheers!


I know you busy but on a separate note can I ask you something about the following example you give:

./texturecache.py jd movies | ./mklocal.py --local /freenas/media --prefix nfs://192.168.0.3/mnt/share/media --artwork fanart poster clearlogo clearart --output fixup.dat


I am running Libreelec on my pi and accessing my NAS via NFS. Is --local where the cache will be stored on my pi i.e. /freenas/media? Is this a fixed folder I have to create for it to work? Will try below tonight and see if it works. I have my media in a parent folder Videos... Not sure I have to specify or leave as parent?


movies:
./texturecache.py jd movies | ./mklocal.py --local /freenas/media --prefix nfs://IP>ADDRESS/Videos/Movies --artwork fanart poster clearlogo clearart --output fixup.dat

tvshows:
./texturecache.py jd tvshows | ./mklocal.py --local /freenas/media --prefix nfs://IP>ADDRESS/Videos/TV Shows --artwork fanart poster clearlogo clearart --output fixup.dat
Reply
(2017-09-05, 05:11)80njc80 Wrote: ./texturecache.py jd movies | ./mklocal.py --local /freenas/media --prefix nfs://192.168.0.3/mnt/share/media --artwork fanart poster clearlogo clearart --output fixup.dat


I am running Libreelec on my pi and accessing my NAS via NFS. Is --local where the cache will be stored on my pi i.e. /freenas/media? Is this a fixed folder I have to create for it to work? Will try below tonight and see if it works. I have my media in a parent folder Videos... Not sure I have to specify or leave as parent?


movies:
./texturecache.py jd movies | ./mklocal.py --local /freenas/media --prefix nfs://IP>ADDRESS/Videos/Movies --artwork fanart poster clearlogo clearart --output fixup.dat

tvshows:
./texturecache.py jd tvshows | ./mklocal.py --local /freenas/media --prefix nfs://IP>ADDRESS/Videos/TV Shows --artwork fanart poster clearlogo clearart --output fixup.dat

No, --local is the OS equivalent mount point for nfs://192.168.0.3/mnt/share/media - in my example, "/freenas/media" is a mount point (for the OS) that gives access to the same content as nfs://192.168.0.3/mnt/share/media (which is a Kodi "mount point"). The script can only access OS mounts, not nfs://whatever, so it has to map and translate between the two, but can access only the content specified by --local.

You can call the --local folder whatever you like, it's just a mount point. The only requirement is that it gives access to the same content as your --prefix. That way the script can translate the paths from Kodi to OS when it needs to, accessing the content (reading, writing, etc.) via the --local path, and then translating the OS paths back to the Kodi nfs:// format for subsequent updating.

Given your examples, assuming your NFS export is actually <IPADDRESS>/Videos (ie. a common top level root used by both your Movies and TV Shows), then the following should work so long as the /freenas/media mount point gives access to the <IPADDRESS>/Videos top-level directory:
Code:
./texturecache.py jd movies | ./mklocal.py --local /freenas/media --prefix nfs://<IPADDRESS>/Videos --artwork fanart poster clearlogo clearart --output fixup.dat
./texturecache.py jd tvshows | ./mklocal.py --local /freenas/media --prefix nfs://<IPADDRESS>/Videos --artwork fanart poster clearlogo clearart --output fixup.dat

The script will then use your prefix, ie. nfs://<IPADDRESS>/Videos, to construct a path suitable for the OS, eg. nfs://<IPADDRESS>/Videos/Movies/Avatar-poster.jpg will be mapped/tranlsated to /freenas/media/Movies/Avatar-poster.jpg.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
OK cheers. You are certainly wise and I will need to digest that a little. Maybe i'm not fully understanding what this program is doing.

So if I don't make my artwork 'local' does that mean that my database is fetching information 'remotely', albeit in a cached format that would slow down scrolling through my library (as an example)?

To take it back a step and I apologise for the layman undertones but I just hate not understanding things.. I have my pi running Kodi on 192.168.1.4 that is accessing my NAS via NFS on 192.168.1.7. Via Putty I can run your programs in W10 (or alternatively linux based systems) without issues and can edit files via Filezilla or the like. So, when I run the below script for movies as an example is it fetching the movie data again and linking it between pathways? Or is it providing a link between the cached movie database already on the pi to the NAS for local storage? The latter doesn't seem to be right...

./texturecache.py jd movies | ./mklocal.py --local /freenas/media --prefix nfs://192.168.1.7/share/Videos --artwork fanart poster clearlogo clearart --output fixup.dat

Thanks in advance.
Reply
@yhs4260 I've taken a look at your log, thanks.

The problem seems to be with Kodi, as it's returning the file paths without the slash when responding to a JSON-RPC Files.GetDirectory query.

The script queries Kodi for the contents of the Movies/2017 directory:
Code:
2017-08-29 09:53:51.966314:MainThread: libDirectory.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "params": {"directory": "http://10.8.0.1:40080/Movies/2017", "media": "files", "properties": ["file", "lastmodified"]}, "method": "Files.GetDirectory", "id": "libDirectory"}]
and Kodi replies with the following (edited to show a subset of movie data):
Code:
2017-08-29 09:53:52.503402:MainThread: libDirectory.PARSING JSON DATA: {"id":"libDirectory","jsonrpc":"2.0","result":{"files":[{"file":"http://10.8.0.1:40080/Movies/201768.Kill.2017.720p.WEB-DL.H264.AC3-EVO.mkv","filetype":"file","label":"68.Kill.2017.720p.WEB-DL.H264.AC3-EVO.mkv","lastmodified":"2017-08-15 18:06:00","type":"unknown"},{"file":"http://10.8.0.1:40080/Movies/2017A.Cure.for.Wellness.2017.720p.BRRip.X264.AC3-EVO.mkv","filetype":"file","label":"A.Cure.for.Wellness.2017.720p.BRRip.X264.AC3-EVO.mkv","lastmodified":"2017-05-28 20:10:00","type":"unknown"},
...
{"file":"http://10.8.0.1:40080/Movies/2017xXx.Return.of.Xander.Cage.2017.BRRip.XviD.AC3-EVO.avi","filetype":"file","label":"xXx.Return.of.Xander.Cage.2017.BRRip.XviD.AC3-EVO.avi","lastmodified":"2017-05-09 23:57:00","type":"unknown"}],"limits":{"end":138,"start":0,"total":138}}}

As you can see, the slashes are missing in the reply from Kodi, so this would appear to be an error in the Kodi JSON-RPC processing, and this then leads to the subsequent errors you saw in your Kodi log when the script attempts to refresh these non-existent files.

Since this problem doesn't occur with nfs:// or smb:// sources it would appear to be specific to this type of http source that you are using.

You should open a ticket on trac.kodi.tv for the attention of @Montellese.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2017-09-05, 07:21)80njc80 Wrote: So if I don't make my artwork 'local' does that mean that my database is fetching information 'remotely', albeit in a cached format that would slow down scrolling through my library (as an example)?

Almost certainly, yes. You can view your movie metadata with "texturecache.py jd movies [moviename]" and if the artwork (poster, fanart, etc.) urls are http-based then your artwork is remote.

Once the artwork is cached there's not going to be any performance difference between remote and local artwork.

The main advantage of local artwork is that if you re-scan your library in future, you get the same artwork you had before - when using remote artwork it's a lottery what artwork the scraper will find. Also, if you decide to recache a movie, the remote artwork might have moved or been re-named since it was scraped, and it won't be possible to re-cache the artwork. This is a particular issue while using a shared MySQL library as new clients cannot cache and display the artwork that was scraped months or even years ago.

(2017-09-05, 07:21)80njc80 Wrote: To take it back a step and I apologise for the layman undertones but I just hate not understanding things.. I have my pi running Kodi on 192.168.1.4 that is accessing my NAS via NFS on 192.168.1.7. Via Putty I can run your programs in W10 (or alternatively linux based systems) without issues and can edit files via Filezilla or the like. So, when I run the below script for movies as an example is it fetching the movie data again and linking it between pathways? Or is it providing a link between the cached movie database already on the pi to the NAS for local storage? The latter doesn't seem to be right...

./texturecache.py jd movies | ./mklocal.py --local /freenas/media --prefix nfs://192.168.1.7/share/Videos --artwork fanart poster clearlogo clearart --output fixup.dat

Thanks in advance.

The script is taking the artwork urls defined in the library for each movie, and converting the artwork paths from the nfs:// format used by Kodi into a format that can be accessed by the script running on the current OS - in the case of a W10 machine your --local path would have to be something like D:/Videos (or whatever drive mapping you use to access the NFS Videos share), so that "nfs://192.168.1.7/share/Videos/Movies/Avatar-poster.jpg" is mapped to "D:/Videos/Movies/Avatar-poster.jpg".

Don't worry if you don't have any local posters yet - the script will download your remote artwork and create the files for you (alternatively, it will identify your local artwork and add those to your library if they are not already present - it can do this for any type of artwork so is good for adding manually downloaded logos and clearart etc.). The script will construct the artwork file names based on the movie file name, so as long as you have "nfs://192.168.1.7/share/Videos/Movies/Avatar.mkv" (or whatever) then the script knows where it needs to put or find your artwork (alongside your movie file).

If you have remote artwork then it will download that artwork and store it alongside the movie file using the moviename-poster.jpg naming format, and then at the end it will output (in fixup.dat) the changes needed to update your library so that your remote artwork urls are replaced with your new local artwork urls. You apply the changes using "cat fixup.dat | texturecache.py set", and then you can cache your new local artwork in the normal way (ie. "texturecache.py c movies", etc.)

If you're not sure about running the mklocal.py script, then include the "--read-only --dryrun" options so that it won't create or change any of your existing files.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
You are a machine. Cheers.

I'll start playing around with it first. Good inclusion "--read-only --dryrun".
Reply
Thank you Milhouse I will open a ticket as suggested.

I appreciate your time looking into this.
Reply
Hi,

I've found out that when using the script with the "R" option, my "Bookmarks" I've created (which are set when you use the "create bookmark" function while a video is playing and stored in "...userdata/thumbnails/video/bookmarks") are always removed!!!

That's not surprising, as the "Bookmarks" are stored in the bookmark table of the "MyVideos*" database, and not in the "Textures13.db" file which the script is investigating when using the "R" option.

So here are my suggestions for modifying the "R" option behaviour:

1. Ignore the "Bookmark" files that are stored in "...userdata/thumbnails/video/bookmarks" to prevent being deleted because of a lacking reference in the "Textures13.db" file
- or -
2. Find and delete only the "orphaned" "Bookmark" files that are stored in "...userdata/thumbnails/video/bookmarks" which have no reference in the "MyVideos*" database

Thank you in advance!
Reply
Yeah... I think I'll just ignore the contents of the Bookmarks folder for now. I'll make the change when I can.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
  • 1
  • 152
  • 153
  • 154(current)
  • 155
  • 156
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17