• 1
  • 107
  • 108
  • 109(current)
  • 110
  • 111
  • 197
[RELEASE] Texture Cache Maintenance utility
I'm planning to use diskless scheme to run xbmc,
google ccboot to see what i mean, basically all clients will use pxe boot and thus the thumbnails should not need synchronizing... Library updates will be done on the diskless server.. My question is.. will this script or mklocal.py help improve the speed for my case?
It was recommended several times but I wonder if it helps in diskless mode
Reply
(2015-09-04, 13:50)denywinarto Wrote: all clients will use pxe boot and thus the thumbnails should not need synchronizing

Why won't they need synchronizing? Each client should still be using their own independent Textures13.db and Thumbnails folders - if you're thinking of sharing a single Thumbnails folder between all your clients then don't, as that doesn't work.

(2015-09-04, 13:50)denywinarto Wrote: will this script or mklocal.py help improve the speed for my case?

As each of your clients should still be using their own texture cache, pre-loading the texture cache is still a good idea and will improve GUI performance by avoiding the delays that result when artwork needs to be cached.

mklocal.py is useful for converting remote artwork to local artwork and only needs to be run against one client if you are sharing the library (I assume you mean MySQL, and not sharing the SQLite databases, which would be a bad idea).

Amongst other things, the use of local artwork will result in less delays when artwork needs to be cached. Once you have converted remote artwork to local artwork, update the texture caches of all clients so that the new local artwork is cached.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-09-04, 14:57)Milhouse Wrote:
(2015-09-04, 13:50)denywinarto Wrote: all clients will use pxe boot and thus the thumbnails should not need synchronizing

Why won't they need synchronizing? Each client should still be using their own independent Textures13.db and Thumbnails folders - if you're thinking of sharing a single Thumbnails folder between all your clients then don't, as that doesn't work.

(2015-09-04, 13:50)denywinarto Wrote: will this script or mklocal.py help improve the speed for my case?

As each of your clients should still be using their own texture cache, pre-loading the texture cache is still a good idea and will improve GUI performance by avoiding the delays that result when artwork needs to be cached.

mklocal.py is useful for converting remote artwork to local artwork and only needs to be run against one client if you are sharing the library (I assume you mean MySQL, and not sharing the SQLite databases, which would be a bad idea).

Amongst other things, the use of local artwork will result in less delays when artwork needs to be cached. Once you have converted remote artwork to local artwork, update the texture caches of all clients so that the new local artwork is cached.

Thanks for the quick reply Milhouse,
I haven't tested ccboot with Kodi,
But i have tested it for an online game called pointblank,,
The game itself is being updated weekly online from its own server, the game size is 5GB, roughly the same with my thumbnails
And i have 25 clients running them simultaneously without any lags or issues

I'd imagine Kodi won't be too different with this game?
Kodi needs to be updated periodically as well
I think pre-loading the texture cache will still beneficial for my case...
But for mklocal.py i don't think i can use it since i don't use MySQL,

PXE booting, If i'm not mistaken it creates pre-environment (OS) on the network card
https://en.wikipedia.org/wiki/Preboot_Ex...nvironment
And CCBoot creates cache for each client,
So if i understand this correctly each client has its own thumbnail stored in the Diskless server.
Reply
Is the ccboot cache persistent? If so, each client will still benefit from texturecache.py. If not, then presumably the cache is cloned from a "master" each time the client is booted, and this "master" cache may still benefit from texturecache.py.

makelocal.py can still be used with SQLite, it's main purpose is simply to ensure your artwork is local rather than remote.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-09-04, 22:10)Milhouse Wrote: Is the ccboot cache persistent? If so, each client will still benefit from texturecache.py. If not, then presumably the cache is cloned from a "master" each time the client is booted, and this "master" cache may still benefit from texturecache.py.

makelocal.py can still be used with SQLite, it's main purpose is simply to ensure your artwork is local rather than remote.

It gets deleted after reaching the limit size,
Not sure about the deleting algorithm, i think it's depends on diskless program,
but it seems to be stable enough to run 500GBs of games on 25 clients without any issues,,

I guess for my case it's the same with local usage
So then i would have to run it via .bat script everytime kodi starts?
Reply
(2015-09-05, 03:58)denywinarto Wrote: It gets deleted after reaching the limit size,
Not sure about the deleting algorithm, i think it's depends on diskless program,
but it seems to be stable enough to run 500GBs of games on 25 clients without any issues,,

Then it sounds like it's somewhat persistent, so running texturecache.py against the client when required would work.

(2015-09-05, 03:58)denywinarto Wrote: I guess for my case it's the same with local usage
So then i would have to run it via .bat script everytime kodi starts?

Not sure what you are referring to by "it".

mklocal.py and texturecache.py would normally be run whenever you have loaded new media and associated artwork. You won't need to run either if the artwork hasn't changed on the client from a previous run of Kodi.

You wouldn't want to run these scripts in a .bat at Kodi start-up because they're not quick - in a MySQL environment you'd normally run mklocal.py on your scraper "master" after scraping new media, and then update the texture cache of each "slave" client in turn (waking the client if necessary using WOL etc.).

If each of your clients has independent SQLite databases (MyVideosXX, MyMusicXX, Textures13 etc.) then I'm not sure how you intend to scrape new stuff - scrape 25 times? However once all your artwork is local you shouldn't need to run mklocal.py a second time, unless you want it to load non-standard artwork which it can also do for you. Although if all your clients are meant to be identical then you should be able to apply the output from the first mklocal.py run against all your other clients. To be honest with 25 clients I don't know why you're not considering MySQL.

I suggest you try using the scripts and then you'll probably understand better when and why you need to run them.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-09-05, 05:38)Milhouse Wrote:
(2015-09-05, 03:58)denywinarto Wrote: It gets deleted after reaching the limit size,
Not sure about the deleting algorithm, i think it's depends on diskless program,
but it seems to be stable enough to run 500GBs of games on 25 clients without any issues,,

Then it sounds like it's somewhat persistent, so running texturecache.py against the client when required would work.

(2015-09-05, 03:58)denywinarto Wrote: I guess for my case it's the same with local usage
So then i would have to run it via .bat script everytime kodi starts?

Not sure what you are referring to by "it".

mklocal.py and texturecache.py would normally be run whenever you have loaded new media and associated artwork. You won't need to run either if the artwork hasn't changed on the client from a previous run of Kodi.

You wouldn't want to run these scripts in a .bat at Kodi start-up because they're not quick - in a MySQL environment you'd normally run mklocal.py on your scraper "master" after scraping new media, and then update the texture cache of each "slave" client in turn (waking the client if necessary using WOL etc.).

If each of your clients has independent SQLite databases (MyVideosXX, MyMusicXX, Textures13 etc.) then I'm not sure how you intend to scrape new stuff - scrape 25 times? However once all your artwork is local you shouldn't need to run mklocal.py a second time, unless you want it to load non-standard artwork which it can also do for you. Although if all your clients are meant to be identical then you should be able to apply the output from the first mklocal.py run against all your other clients. To be honest with 25 clients I don't know why you're not considering MySQL.

I suggest you try using the scripts and then you'll probably understand better when and why you need to run them.

Sorry, was referring to texturecache.py... So this script is strictly for MySQL usage only?

I wont need to scrape 25 times,
I just need to scrape once in the diskless server,
then after client restarts (to reload the image) they will have the same exact library.
It's been working that way for online games that has similar method of updating files,
so i'm sure it will work for Kodi as well

That's why for my case i think it's the same as local usage,
I just need to make the thumbnails load faster in the diskless server..
From your post i assume i have to run it everytime i update the library?
Reply
(2015-09-05, 06:50)denywinarto Wrote: Sorry, was referring to texturecache.py... So this script is strictly for MySQL usage only?

No, it doesn't matter what your backend database is, either MySQL or SQLite, as the script uses JSON to communicate with Kodi. It's just that MySQL is a typical use-case for texturecache.py as the non-scraper clients in a MySQL environment will benefit from having their cache pre-loaded after new media is scraped.

(2015-09-05, 06:50)denywinarto Wrote: I wont need to scrape 25 times,
I just need to scrape once in the diskless server,
then after client restarts (to reload the image) they will have the same exact library.
It's been working that way for online games that has similar method of updating files,
so i'm sure it will work for Kodi as well

That's why for my case i think it's the same as local usage,
I just need to make the thumbnails load faster in the diskless server..
From your post i assume i have to run it everytime i update the library?

Unlike games, Kodi will be updating its "local storage" which is not static, like perhaps a game would be. From this latest description it doesn't sound like the client is persistent at all, with the client receiving a new copy of the "master" environment each time it boots, so if you just scrape on the server then I don't think you'll need texturecache.py (at least, not to pre-load the cache as this usually isn't necessary on a scraper client, as the scraper client is able to maintain it's own texture cache while it scrapes), although it wouldn't do any harm to run it after a scrape just to be sure the texture cache is fully up to date.

Just make sure each client receives a *unique copy* of the environment as you don't want clients updating shared data (potential for SQLite data corruption, etc.). However since you refer to it as an "image" I'm assuming that any modifications written to the image by the client are not persisted and won't affect any other client, which should then be fine.

Also, the cached thumbnails (generated by your diskless server) are potentially platform specific, so all your clients should be identically configured including similar hardware (usually not a problem if all x86, but can be an issue when mixing x86 and ARM). Just something to be aware of.

You might still benefit from running mklocal.py after a scrape, it really depends how you are sourcing your artwork and whether you want them all to be local (in the absence of local artwork, Kodi will find and use remote artwork). If you do convert any remote artwork to local using mklocal.py, you will need to update the texture cache with the new local artwork so your diskless server workflow would be:
  1. Scrape new media
  2. mklocal.py ...
  3. texturecache.py c

Automating this is pretty straight forward so that it runs on a regular basis (cron), or as new media is detected (polling your library for changes), or just as a one-click script - the choice is yours (and for you to implement!) You can also use texturecache.py to initiate a video library scan with "texturecache.py vscan", so you don't need to go near the Kodi GUI.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-09-05, 08:29)Milhouse Wrote:
(2015-09-05, 06:50)denywinarto Wrote: Sorry, was referring to texturecache.py... So this script is strictly for MySQL usage only?

No, it doesn't matter what your backend database is, either MySQL or SQLite, as the script uses JSON to communicate with Kodi. It's just that MySQL is a typical use-case for texturecache.py as the non-scraper clients in a MySQL environment will benefit from having their cache pre-loaded after new media is scraped.

(2015-09-05, 06:50)denywinarto Wrote: I wont need to scrape 25 times,
I just need to scrape once in the diskless server,
then after client restarts (to reload the image) they will have the same exact library.
It's been working that way for online games that has similar method of updating files,
so i'm sure it will work for Kodi as well

That's why for my case i think it's the same as local usage,
I just need to make the thumbnails load faster in the diskless server..
From your post i assume i have to run it everytime i update the library?

Unlike games, Kodi will be updating its "local storage" which is not static, like perhaps a game would be. From this latest description it doesn't sound like the client is persistent at all, with the client receiving a new copy of the "master" environment each time it boots, so if you just scrape on the server then I don't think you'll need texturecache.py (at least, not to pre-load the cache as this usually isn't necessary on a scraper client, as the scraper client is able to maintain it's own texture cache while it scrapes), although it wouldn't do any harm to run it after a scrape just to be sure the texture cache is fully up to date.

Just make sure each client receives a *unique copy* of the environment as you don't want clients updating shared data (potential for SQLite data corruption, etc.). However since you refer to it as an "image" I'm assuming that any modifications written to the image by the client are not persisted and won't affect any other client, which should then be fine.

Also, the cached thumbnails (generated by your diskless server) are potentially platform specific, so all your clients should be identically configured including similar hardware (usually not a problem if all x86, but can be an issue when mixing x86 and ARM). Just something to be aware of.

You might still benefit from running mklocal.py after a scrape, it really depends how you are sourcing your artwork and whether you want them all to be local (in the absence of local artwork, Kodi will find and use remote artwork). If you do convert any remote artwork to local using mklocal.py, you will need to update the texture cache with the new local artwork so your diskless server workflow would be:
  1. Scrape new media
  2. mklocal.py ...
  3. texturecache.py c

Automating this is pretty straight forward so that it runs on a regular basis (cron), or as new media is detected (polling your library for changes), or just as a one-click script - the choice is yours (and for you to implement!) You can also use texturecache.py to initiate a video library scan with "texturecache.py vscan", so you don't need to go near the Kodi GUI.

Sorry for the late reply, was trying to figure out the best scenario for my case,
I found another alternative, I just remembered i could fetch the artwork using filebot and store them in library folders..
And name them according to xbmc standars, e.g posters.jpg, banners.jpg

My question is, which one give better performance for diskless case like me?
1. Store the artwork in the ubuntu server inside the library (inside the movie and tvshow folders) OR
2. Store them in thumbnails folder in userdata, the diskless program creates cache for each client,
so theoretically this should give less burden to the traffic compared to the first option..
But i'm not sure how artwork fetching works in xbmc.. (and how it will the affect to the performance)

I have roughly 7000 movies and 500 tvshows, i store them alphabetically
Lastly, can this script copy the artwork stored in library using filebot to thumbnails folder?
Reply
(2015-09-09, 14:07)denywinarto Wrote: My question is, which one give better performance for diskless case like me?
1. Store the artwork in the ubuntu server inside the library (inside the movie and tvshow folders) OR
2. Store them in thumbnails folder in userdata, the diskless program creates cache for each client,
so theoretically this should give less burden to the traffic compared to the first option..
But i'm not sure how artwork fetching works in xbmc.. (and how it will the affect to the performance)

I have roughly 7000 movies and 500 tvshows, i store them alphabetically

Thing is, *you* don't manage the Thumbnails folder - that's Kodi's domain. You store your artwork in your library, or reference it on remote web sites, then you leave it to Kodi to write a copy of that artwork - optimised for the client on which the artwork will be displayed - into the Thumbnails folder. The Thumbnails folder along with Textures13.db is known as the "texture cache".

Given that you are "diskless" and therefore accessing your cached artwork over the LAN, this will almost always be slower than if you had the texture cache on a local disk. Conceivably a really crap SD card or USB disk might be slower than a good wired GigE network connection but generally speaking the LAN will always be slower than average-speed local storage.

(2015-09-09, 14:07)denywinarto Wrote: Lastly, can this script copy the artwork stored in library using filebot to thumbnails folder?

You can use the script to populate the Thumbnails folder (texture cache) from your local/remote artwork - the script will request Kodi to store your local/remote artwork into the texture cache. At no point should anyone other than Kodi be writing directly in to the Thumbnails folder.
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-09-09, 18:00)Milhouse Wrote:
(2015-09-09, 14:07)denywinarto Wrote: My question is, which one give better performance for diskless case like me?
1. Store the artwork in the ubuntu server inside the library (inside the movie and tvshow folders) OR
2. Store them in thumbnails folder in userdata, the diskless program creates cache for each client,
so theoretically this should give less burden to the traffic compared to the first option..
But i'm not sure how artwork fetching works in xbmc.. (and how it will the affect to the performance)

I have roughly 7000 movies and 500 tvshows, i store them alphabetically

Thing is, *you* don't manage the Thumbnails folder - that's Kodi's domain. You store your artwork in your library, or reference it on remote web sites, then you leave it to Kodi to write a copy of that artwork - optimised for the client on which the artwork will be displayed - into the Thumbnails folder. The Thumbnails folder along with Textures13.db is known as the "texture cache".

Given that you are "diskless" and therefore accessing your cached artwork over the LAN, this will almost always be slower than if you had the texture cache on a local disk. Conceivably a really crap SD card or USB disk might be slower than a good wired GigE network connection but generally speaking the LAN will always be slower than average-speed local storage.

(2015-09-09, 14:07)denywinarto Wrote: Lastly, can this script copy the artwork stored in library using filebot to thumbnails folder?

You can use the script to populate the Thumbnails folder (texture cache) from your local/remote artwork - the script will request Kodi to store your local/remote artwork into the texture cache. At no point should anyone other than Kodi be writing directly in to the Thumbnails folder.

Thanks Milhouse, i think local thumbnails will be slightly faster because the diskless program creates different cache for each client..
Unlike the first option where clients have to access single shared thumbnails from the NAS

I got another issue related to Textures13.db,
It seems like my database is messed up and Kodi won't scrape fanarts and posters from the path i already scrap before.. any idea why?
http://forum.kodi.tv/showthread.php?tid=238310
I don't wanna rebuild the database if possible Sad
Reply
(2015-09-09, 18:57)denywinarto Wrote: Unlike the first option where clients have to access single shared thumbnails from the NAS

Sharing the Thumbnails folder between multiple clients is asking for problems - it's not a supported option (because it doesn't/can't work) and it will cause problems.

(2015-09-09, 18:57)denywinarto Wrote: I got another issue related to Textures13.db,
It seems like my database is messed up and Kodi won't scrape fanarts and posters from the path i already scrap before.. any idea why?
http://forum.kodi.tv/showthread.php?tid=238310
I don't wanna rebuild the database if possible Sad

You seem rather confused about the purpose of the texture cache - you shouldn't have deleted the local artwork (which normally resides alongside your video files) from your NAS as Kodi now has no source for the artwork it may need to cache and display.

You should upload a debug log (wiki) into your thread, maybe someone will be able to work out what has happened but you've obviously got something out of sync. My guess would be that you should probably delete the Textures13.db and Thumbnails folders, make sure you're not doing any Path Substitution/sharing of Thumbnails, and then re-scrape your TV Shows. Please continue discussion of this issue in your other thread, scraper issues are somewhat OT for this thread, although you can use this script to view the artwork urls for a tvshow and that might give you a clue what is happening:
Code:
./texturecache.py jd tvshows "tvshowname"
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
(2015-09-09, 19:19)Milhouse Wrote:
(2015-09-09, 18:57)denywinarto Wrote: Unlike the first option where clients have to access single shared thumbnails from the NAS

Sharing the Thumbnails folder between multiple clients is asking for problems - it's not a supported option (because it doesn't/can't work) and it will cause problems.

(2015-09-09, 18:57)denywinarto Wrote: I got another issue related to Textures13.db,
It seems like my database is messed up and Kodi won't scrape fanarts and posters from the path i already scrap before.. any idea why?
http://forum.kodi.tv/showthread.php?tid=238310
I don't wanna rebuild the database if possible Sad

You seem rather confused about the purpose of the texture cache - you shouldn't have deleted the local artwork (which normally resides alongside your video files) from your NAS as Kodi now has no source for the artwork it may need to cache and display.

You should upload a debug log (wiki) into your thread, maybe someone will be able to work out what has happened but you've obviously got something out of sync. My guess would be that you should probably delete the Textures13.db and Thumbnails folders, make sure you're not doing any Path Substitution/sharing of Thumbnails, and then re-scrape your TV Shows. Please continue discussion of this issue in your other thread, scraper issues are somewhat OT for this thread, although you can use this script to view the artwork urls for a tvshow and that might give you a clue what is happening:
Code:
./texturecache.py jd tvshows "tvshowname"

Sorry Milhouse, i figured it out, it was because the nfo generated by filebot Big Grin

I wasn't referring to the thumbnails folder,
what i meant was the artworks stored in each movies & tvshows,
what if they're shared to multiple clients...
compared to cached diskless userdata containing thumbnails folder unique to each client

It's strange, i just did some testing on 2 PC using diskless scheme

i used 52 tvshows stored in 2 A-E folders (intentionally duplicated to test the performance)
The first one has the artwork (posters, banners, etc) stored in NAS generated using filebot script
The second has the artwork fetched manually from Kodi and is stored within userdata folder

Both are restarted first before opening Kodi
Browsing the first one seems to be faster..
There's a noticable delay, especially in background poster for the second one..

But then again, this is only a small sample, can't tell if it'll be the same for larger sample and heavier network traffic
Reply
(2015-09-10, 07:33)denywinarto Wrote: I wasn't referring to the thumbnails folder,
what i meant was the artworks stored in each movies & tvshows,
what if they're shared to multiple clients...

Yes, sharing the artwork stored alongside your media is absolutely fine - it's read once by each client then written into the cache (Thumbnails folder + Textures13.db)

(2015-09-10, 07:33)denywinarto Wrote: compared to cached diskless userdata containing thumbnails folder unique to each client

As long as each client has their own copy of the databases in the Database folder (which includes Textures13.db) and their own copy of the Thumbnails folder, you shouldn't have a problem with a diskless setup. It will be slower than local storage, however.

(2015-09-10, 07:33)denywinarto Wrote: It's strange, i just did some testing on 2 PC using diskless scheme

i used 52 tvshows stored in 2 A-E folders (intentionally duplicated to test the performance)
The first one has the artwork (posters, banners, etc) stored in NAS generated using filebot script
The second has the artwork fetched manually from Kodi and is stored within userdata folder

Both are restarted first before opening Kodi
Browsing the first one seems to be faster..
There's a noticable delay, especially in background poster for the second one..

But then again, this is only a small sample, can't tell if it'll be the same for larger sample and heavier network traffic

It all depends on what artwork needs to be cached. In theory the caching process will result in a slower user experience the first time an item of artwork is displayed, but there should be no delay when the same artwork is displayed on subsequent occasions.

As far as your test is concerned, a big factor is the state of the texture cache available to each client before you begin browsing around the GUI - if artwork needs to be cached the GUI will be slower, as it needs to cache the artwork before it can be displayed. You can check what artwork needs to be cached by running "texturecache.py nc tvshows". To pre-load the cache use "texturecache.py c tvshows"

If you want to test the performance of local (ie. NAS) artwork compared with remote (ie. internet based) artwork then delete both texture caches from your two clients (ie. Textures13.db and Thumbnails folder), restart Kodi and repeat your test - both clients will now be caching their respective artwork into their respective texture cache. Once artwork is cached, any subsequent delay is due to something else (CPU/GPU performance, storage/network performance, etc. etc.)
Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
Reply
Is this possible to do music fanart? I entered texturecache.py c music and it didn't work...
Reply
  • 1
  • 107
  • 108
  • 109(current)
  • 110
  • 111
  • 197

Logout Mark Read Team Forum Stats Members Help
[RELEASE] Texture Cache Maintenance utility17