[Milhouse]

texturecache.py (Version 1.4.3)

• Add: Implement chunked queries to reduce memory consumption on client and server (which are often the same device). In testing, chunked queries significantly reduced XBMC memory consumption, by as much as 80MB or more (a big deal on a Pi with under 200MB free). This change will ensure that queries use a relatively fixed (and small) amount of server memory when responding to chunked queries, rather than unpredictable and sometimes very large amounts of memory when responding to unconstrained queries (for example when retrieving all movies with all cast members, or the entire texture cache database).
  
  Media library queries will be chunked using JSON API limits (start/end), with the chunk size varying according to the anticipated complexity of the query, eg. when caching cast thumbs a smaller chunk size will be used due to the significant increase in data per movie.
  
  The Textures13 db query will also be chunked, though not using limits as this isn't currently supported by the JSON API (nor obviously SQL, which is used when the Texture DB JSON API isn't available). Instead, data for each of the Thumbnail folders (0-9,a-f) will be retrieved in sequence (cachedurl like '0/%' etc.), which results in more manageable (but unfortunately still variable) amounts of data, and has required related functions (caching, pruning) to be re-written.
  
  In addition, chunking allows unnecessary query data to be eliminated sooner, reducing client memory consumption - for example, when caching artwork, details of cast members without thumbnails can be dropped as soon as each chunk is received which avoids storing often significant amounts of redundant information only to be processed and discarded later.
  
  Chunked queries is not currently enabled by default, but can be enabled with @chunked=yes - please do so and report any problems. If there are no problems reported in the next week or so I will enable chunked queries as the default setting. Other than a slight performance gain when not chunking (assuming you don't run out of memory), there should be no reason to disable this option and in fact I may also remove the non-chunked code at some point in the future.
  
  Other functions, such as texture database searches (s, S, x, X etc.) and orphan file checks (r/R) have not been updated to support chunked queries as it's not practical to do so, plus you are less likely to have memory problems when performing these operations.
  

  ---

• Add: readfile option added to return content of the named file, outputting contents to stdout when the output filename is "-" (not suitable for binary data) or to the named output file (which is suitable for binary data)
  

texturecache.py (Version 1.4.4)

• Fix: Cosmetic change
  

Chunked queries is a fairly major change in this release, one which should benefit anyone experiencing out of memory issues. Please test, as I intend for chunked queries to become the default (and possibly only) option in future.