[Milhouse]

Check your kodi.log, as it looks like Kodi is experiencing a fairly major problem processing web server requests. Have you updated Kodi recently?

[RazorFR]

Yes I've updated to Helix Final yesterday.

I remembered I also installed the audio plugin spotlight, in replacement of spotimc that was suddenly crashing with Helix.
I tried removing spotlight from my user data folder and now the texture recache script is working again...
That's a bit annoying because I'd like to keep a way to access spotify in xbmc.

In addition sadly, I have noticed that enabling the web server with Helix causes crashes. It seems other users too:
http://forum.kodi.tv/showthread.php?tid=204647&page=6
I hope the team will find a fix for that.

[pgjensen]

Is there a way to do thumbnail caching, but not download the artwork file if it's missing from Textures13.db but DOES exist in the local Thumbnails directory (with the same hash of course)? If I start with a fresh Textures13.db, already have all artwork on the local filesystem, and run `texturecache.py c movies` it redownloads all of the same artwork (same names, sizes). If Textures13.db is fully populated, and the file exists on the filesystem, that's the only way it won't actually redownload the artwork.

Also, if I point the JSONRPC location to a different box (i.e. a headless server), and run caching on the client, no matter if it exists in Textures13.db AND on the local filesystem, it treats all thumbnails as missing. Although I am guessing this could be because it's computing the hash based on a local file path?

(Note: I use a shared MySQL db, samba shares for files, Helix final, and `texturecache.py update` reports latest version)

[Milhouse]

Is there a way to do thumbnail caching, but not download the artwork file if it's missing from Textures13.db but DOES exist in the local Thumbnails directory (with the same hash of course) If I start with a fresh Textures13.db, already have all artwork on the local filesystem, and run `texturecache.py c movies` it redownloads all of the same artwork (same names, sizes). If Textures13.db is fully populated, and the file exists on the filesystem, that's the only way it won't actually redownload the artwork.

Not without inserting rows directly into Textures13.db.

Also, if I point the JSONRPC location to a different box (i.e. a headless server), and run caching on the client, no matter if it exists in Textures13.db AND on the local filesystem, it treats all thumbnails as missing. Although I am guessing this could be because it's computing the hash based on a local file path?

Can you upload a logfile from "textutecach.py @logfile=".

[pgjensen]

Is there a way to do thumbnail caching, but not download the artwork file if it's missing from Textures13.db but DOES exist in the local Thumbnails directory (with the same hash of course) If I start with a fresh Textures13.db, already have all artwork on the local filesystem, and run `texturecache.py c movies` it redownloads all of the same artwork (same names, sizes). If Textures13.db is fully populated, and the file exists on the filesystem, that's the only way it won't actually redownload the artwork.

Not without inserting rows directly into Textures13.db.

Also, if I point the JSONRPC location to a different box (i.e. a headless server), and run caching on the client, no matter if it exists in Textures13.db AND on the local filesystem, it treats all thumbnails as missing. Although I am guessing this could be because it's computing the hash based on a local file path?

Can you upload a logfile from "textutecach.py @logfile=".

Logfiles in PM

[pgjensen]

Is there a way to get the .dds artwork created as you cache the jpg/png files? It does it on the host machine that adds files to the library, but nowhere else. That's one reason I wanted to have the thumbnail caching done through rsync and the ability for your script to recognized local files and just add them straight to the db without re-downloading the files (which takes a considerable amount of time compared to rsync).

[redglory]

On the other hand you only do it one time

[redglory]

Milhouse,

When I run vscan using texturecache.py with a movie path parameter, like this:

Code:

D:\TextureCache>python texturecache.py vscan "nfs://192.168.1.77/movies/Perfect.Sense.(2011)/"
Rescanning directory: nfs://192.168.1.77/movies/Perfect.Sense.(2011)/...

my kodi's log shows only:

Code:

15:33:01 T:5764  NOTICE: VideoInfoScanner: Starting scan ..
15:33:01 T:5764   DEBUG: CAnnouncementManager - Announcement: OnScanStarted from xbmc
15:33:01 T:5764   DEBUG: GOT ANNOUNCEMENT, type: 16, from xbmc, message OnScanStarted
15:33:01 T:5764  NOTICE: VideoInfoScanner: Finished scan. Scanning for video info took 00:00

but when i run vscan without path it works as expected.

Has JSON RPC v6 been updated?

Thanks

EDIT: after movie on library, if i re-run vscan with movie path it works correctly:

Code:

16:09:56 T:4936  NOTICE: VideoInfoScanner: Starting scan ..
16:09:56 T:4936   DEBUG: CAnnouncementManager - Announcement: OnScanStarted from xbmc
16:09:56 T:4936   DEBUG: GOT ANNOUNCEMENT, type: 16, from xbmc, message OnScanStarted
16:09:56 T:4936   DEBUG: NFS: Old context timed out - destroying it
16:09:56 T:4936   DEBUG: NFS: Context for 192.168.1.77/movies not open - get a new context.
16:09:56 T:4936   DEBUG: NFS: Connected to server 192.168.1.77 and export /movies
16:09:56 T:4936   DEBUG: NFS: chunks: r/w 32768/32768
16:09:56 T:3984   DEBUG: CImageLoader::DoWork - took 2301 ms to load special://masterprofile/Thumbnails/9/9b73886a.jpg
16:09:56 T:4936   DEBUG: VideoInfoScanner: Skipping dir 'nfs://192.168.1.77/movies/Perfect.Sense.(2011)/' due to no change (fasthash)
16:09:56 T:4936  NOTICE: VideoInfoScanner: Finished scan. Scanning for video info took 00:00

[Milhouse]

Logfiles in PM

From your PM:

ran both locally, against a Textures.db that's about halfway done with cached artwork. On the localhost it recognizes existing artwork in the db. Pointing to the server (only changing the texturecache.cfg file) causes texturecache to think nothing is cached. MySQL shared db, same samba location for files.

server: (log)
localhost: (log)

Thanks for looking into this!

This is looking like a Kodi bug of some sort. When the server is queried for TexturesDB information (ie. the contents of the Texture cache, Textures13.db) it always respond with no rows:

Code:

2014-12-28 13:22:48.225000:MainThread: libTextures.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "params": {"filter": {"operator": "startswith", "field": "cachedurl", "value": "0/"}, "properties": ["cachedurl", "url"]}, "method": "Textures.GetTextures", "id": "libTextures"}]
2014-12-28 13:22:48.230000:MainThread: libTextures.BUFFER RECEIVED (len 61)
2014-12-28 13:22:48.230000:MainThread: libTextures.PARSING JSON DATA: {"id":"libTextures","jsonrpc":"2.0","result":{"textures":[]}}

however when the same query is performed on localhost, the expected information is returned:

Code:

2014-12-28 13:21:41.088000:MainThread: libTextures.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "params": {"filter": {"operator": "startswith", "field": "cachedurl", "value": "0/"}, "properties": ["cachedurl", "url"]}, "method": "Textures.GetTextures", "id": "libTextures"}]
2014-12-28 13:21:41.287000:MainThread: libTextures.BUFFER RECEIVED (len 32768)
2014-12-28 13:21:41.288000:MainThread: libTextures.BUFFER RECEIVED (len 23392)
2014-12-28 13:21:41.288000:MainThread: libTextures.BUFFER RECEIVED (len 2564)
2014-12-28 13:21:41.296000:MainThread: libTextures.PARSING JSON DATA: {"id":"libTextures","jsonrpc":"2.0","result":{"textures":[{"cachedurl":"0/00ad4f83.jpg","textureid":1,"url":"image://smb%3a%2f%2fMEDIA%2fstorage%2fmovies/{HIDDEN}/"}]}}

Since the "network" query is returning an empty texture cache, the script proceeds to cache everything, which is entirely correct. I really don't know why the server is responding with an empty cache when queried over the network, but a populated cache when queried locally - this is perhaps better answered by one of the Windows Webserver/JSON-RPC developers (the webserver is currently known to crash on Windows, for some as yet undetermined reason).

Checking the kodi.log during these network queries may show some evidence of service failures, particularly if debug is enabled, but there's definitely something amiss with the Windows installation of Kodi.

Is there a way to get the .dds artwork created as you cache the jpg/png files? It does it on the host machine that adds files to the library, but nowhere else. That's one reason I wanted to have the thumbnail caching done through rsync and the ability for your script to recognized local files and just add them straight to the db without re-downloading the files (which takes a considerable amount of time compared to rsync).

If Kodi is configured to create dds files, and the build you are using supports DDS files (some dropped DDS support several months ago) then they'll be created as artwork is cached. If the build you are using no longer supports DDS files then enabling it will have no effect. If DDS files are not being generated when they should be, then that's another Kodi issue.

[Milhouse]

Milhouse,

When I run vscan using texturecache.py with a movie path parameter, like this:

Code:

D:\TextureCache>python texturecache.py vscan "nfs://192.168.1.77/movies/Perfect.Sense.(2011)/"
Rescanning directory: nfs://192.168.1.77/movies/Perfect.Sense.(2011)/...

Yeah, it's a bit hit and miss when you scan in a new "leaf" directory, and (un)fortunately nothing new.

My take on it is that you're asking the scanner to scan a folder ("Perfect.Sense.(2011)") that isn't yet known to the media library, so it is ignored. Scan the parent folder, however, with "vscan nfs://192.168.1.77/movies" and there's a very good chance that the "Perfect.Sense.(2011)" folder will be found and the contents added to the media library.

It's a bit like trying to scan in a new episode for a tv season. If it's a new season (say Season 7), then you need to scan against the *whole* library as running "vscan nfs://blah/tvshows/The Mentalist/Season 7" simply won't work, as Season 7 for The Mentalist isn't "registered" at this point with the media library, so the scan of the "unknown" Season 7 folder won't find anything.

Running "vscan nfs://blah/tvshows/The Mentalist" may also be effective (assuming The Mentalist folder has already been scanned/added), and might be a slightly faster version of the whole-library scan (it's been a while since I last tried to work out why the scanner behaved this way).

After Season 7 is added by a whole-library/tvshow scan, future new episodes in "Season 7" will be found by simply scanning only the "Season 7" folder. So the following week when "The Mentalist S07E02" is added, you CAN run "vscan nfs://blah/tvshows/The Mentalist/Season 7" and the new episode #2 (or whatever) will be added without a problem.

The automated library scanning procedure I use varies depending on movie or tvshow, and new season or existing season. If a movie is being added the whole library is scanned - trying to scan only the folder that contains the new movie will invariably fail as you've found out. If a TV show episode is being added then a whole library scan is performed if it's a new season, but if it's a new episode for an existing season then only the season folder needs to be scanned.

Not sure if this makes any sense, but the above is my take on what is happening. It's confusing behaviour, but what you're seeing now is not new behaviour.

[redglory]

Weird, I could swear i could run vscan with an unknown folder regarding video library...

I was trying to avoid running a complete re-scan as it takes lots of time and I'm using the vscan command to update video library and then set my local art and caching it with the use of texturecache, all in a sequence...

Going to adapt my script towards this...

Thanks anyway

[Mychale]

wow, what an awesome thread to stumble on. so much info.

Ok, so I have it installed as stated in post #1199, and it seems to be working. Any ideas on a test to run first?

Some info on my setup:
I have media server that holds all movies and tvshows. Seperate folders for all Movies and TV shows. Media has .nfo files and art saved in each folder.
I am using MySQL5.7 shared DB running Win2k8, and have 7 Win7 htpc's that currently run XBMC Gotham release, and a Pi. Local shares in XBMC use smb://MEDIA-SERVER/Videos/Kids 12 and Under/Movies/ paths. (4 others)

I understand that this script is not changing anything in MySQL, just the textures13.db file for the installation on that local machine I run this on. Correct?

Once this DB is cleaned up and fixed on this PC, I have a personal .bat file that updates all boxes with the new textures13.db file, and a restart of XBMC locally.

My last question (promise) - with a new setup, what would be a good place to start to run this script? The posts I see here have a lot of passing variables that seem unique to their setup.

This is a copy of the texturecache.cfg I wrote following the first few pages of posts:

Code:

userdata = C:\Users\Mychale\AppData\Roaming\XBMC\userdata
dbfile = Database/Textures13.db
thumbnails = Thumbnails/
xbmc.host = localhost
webserver.port = 8080
allow.recacheall = yes
rpc.port = 9090
download.threads = 6
qaperiod = 9999
checkupdate = yes
orphan.limit.check = yes

Thank you for your work, time, and continued assistance btw, I am amazed at how this started out as a simple tool for yourself and it grew into this.

[Milhouse]

Ok, so I have it installed as stated in post #1199, and it seems to be working. Any ideas on a test to run first?

Code:

texturecache.py c

is probably what most people run first.

I understand that this script is not changing anything in MySQL, just the textures13.db file for the installation on that local machine I run this on. Correct?

When caching, only Texture13.db is updated, although it is possible to update the media library (whether it be MySQL or SQLite) using the "set" option, for example:

Code:

texturecache.py imdb movies | texturecache.py set

would update your movies with the latest votes, ratings, and top250 from imdb - additional imdb fields can be configured - and these changes would then be applied against your media library, which as it's central means you'd only need to run this update on one client (potentially your Pi) for them all to see the benefit.

Once this DB is cleaned up and fixed on this PC, I have a personal .bat file that updates all boxes with the new textures13.db file, and a restart of XBMC locally.

So you're copying the textures13.db file to each PC? Just be aware that this could cause problems if you're not also copying/syncing the Thumbnails folder as well.

You can also run this script centrally, against remote clients by varying the @xbmc.host setting. Potentially you could run the script on the Pi against each of your Windows clients, waking them up ("wake") and putting them to sleep ("power suspend") as required. You'll need to configure the MAC address of each client to wake them up, assuming they support Wake-On-LAN. Best way to do this would be to add "sections" to your cfg for each client, ie.

Code:

allow.recacheall = yes
download.threads = 6
qaperiod = 9999
checkupdate = yes
orphan.limit.check = yes

[pc1]
xbmc.host = 192.168.0.123
network.mac = dd:27:eb:12:9e:7b

[pc2]
xbmc.host = 192.168.0.124
network.mac = dd:27:eb:12:9e:7d

[pc3]
xbmc.host = 192.168.0.125
network.mac = dd:27:eb:12:9e:8b

then simply add "@section=pc1" when running texturecache.py to run options against the pc1 client, add @section=pc2 to run against pc2 etc., cycling through all of your clients. Any setting in a section overrides the "global" settings you see at the top of the cfg file.

The only options that require direct access to the userdata/Thumbnails folder (ie. that need to be run locally, or against a mounted filesystem) are "f", "F", "r", "R", "S", "X" and "Xd". Everything else can be run centrally against remote clients using JSON for control and updates.

My last question (promise) - with a new setup, what would be a good place to start to run this script? The posts I see here have a lot of passing variables that seem unique to their setup.

Not sure I understand the question, are you asking what to run, where to run? I've fully automated my library management (scanning [vscan], caching [c], quality checking [qax], local artwork creation/loading [mklocal.py]) using this script, and run it centrally on a Pi with a MySQL library, pre-loading the caches ("lc movies"/"lc tvshows") of my non-Pi clients whenever new stuff has been added centrally.

This is a copy of the texturecache.cfg I wrote following the first few pages of posts:

Code:

userdata = C:\Users\Mychale\AppData\Roaming\XBMC\userdata
dbfile = Database/Textures13.db
thumbnails = Thumbnails/
xbmc.host = localhost
webserver.port = 8080
allow.recacheall = yes
rpc.port = 9090
download.threads = 6
qaperiod = 9999
checkupdate = yes
orphan.limit.check = yes

The first five settings you have are just the system defaults (rpc.port=9090 is also the default), so you can probably drop those. You really only need to specify the userdata/dbfile/thumbnails settings if they're in an unusual or non-standard location (for example, if you're mounting the userdata folder of a remote client on the local PC). The rest look fine.

I am amazed at how this started out as a simple tool for yourself and it grew into this.

You're amazed, I'm horrified.

[Mychale]

Thank you for the detailed info, and Happy New Year!

I ran the:

Code:

texturecache.py c

as you suggested and I had a lot to cache. I bit the bullet and ran:

Code:

texturecache.py C

and then a:

Code:

texturecache.py p

which showed a lot of extra files so I did the P after that, but not sure I'm seeing the results of all this. The folder did get smaller, now 9GB+ from 15GB+, but how do I update the missing art work and data (plot/plot outline/rating/...) for both movies and tv shows?

I'm now thinking of backing up my data, and starting fresh. I have .nfo files and images saved with the media is separate folders. I recently ran a separate file export before upgrading to Kodi. I just want to backup the watched flags and "date added to library" for movies and tv shows, playcount and resume times would be good too. What would be a good way to go about that? Would this script even help with that?

Your thoughts/suggestions? and thanks for taking the time to reply.

[redglory]

Why running caching and recaching after?