[Levi59]

Little problem since last update:

Code:

xbian@xbian ~ $ ./texturecache.py P
Successfully updated from v1.3.5 to v1.4.0
[...]
Summary: 312 files; Total size: 4,506 KB

xbian@xbian ~ $ ./texturecache.py R
Orphaned file found: Name [e/e2c635f3.jpg], Created [Sat Jan 25 01:00:36 2014],                                                                                                                           Size [0], REMOVING...

Summary: 1 files; Total size: 0 KB

xbian@xbian ~ $ ./texturecache.py C all

Caching artwork: 59 items remaining of 59 (qs: 0, qm: 59), 0 errors, 2 threads a                                                                                                                          Caching artwork: 58 items remaining of 59 (qs: 0, qm: 58), 0 errors, 2 threads a                                                                                                                          Caching artwork: 57 items remaining of 59 (qs: 0, qm: 57), 0 errors, 2 threads a                                                                                                                          Caching artwork: 0 items remaining of 59 (qs: 0, qm: 0), 1 error, 0 threads active (00.11 downloads per second, ETA: 00:00:00)

The following items could not be downloaded:

[fanart    ] [Repositories Installer                  ] /home/xbian/.xbmc/addons/plugin.program.repo.installer/

Traceback (most recent call last):
  File "./texturecache.py", line 6391, in <module>
    main(sys.argv[1:])
  File "./texturecache.py", line 6247, in main
    extraFields=_extraFields, query=_query)
  File "./texturecache.py", line 3392, in jsonQuery
    cacheImages(mediatype, jcomms, database, data, title_name, id_name, force, nodownload)
  File "./texturecache.py", line 3567, in cacheImages
    item.status = MyMediaItem.QUEUED
AttributeError: type object 'MyMediaItem' has no attribute 'QUEUED'
xbian@xbian ~ $ ./texturecache.py C all

Caching artwork: 0 items remaining of 59 (qs: 0, qm: 0), 1 error, 0 threads active (00.10 downloads per second, ETA: 00:00:00)

The following items could not be downloaded:

[fanart    ] [Repositories Installer                  ] /home/xbian/.xbmc/addons/plugin.program.repo.installer/

Traceback (most recent call last):
  File "./texturecache.py", line 6391, in <module>
    main(sys.argv[1:])
  File "./texturecache.py", line 6247, in main
    extraFields=_extraFields, query=_query)
  File "./texturecache.py", line 3392, in jsonQuery
    cacheImages(mediatype, jcomms, database, data, title_name, id_name, force, nodownload)
  File "./texturecache.py", line 3567, in cacheImages
    item.status = MyMediaItem.QUEUED
AttributeError: type object 'MyMediaItem' has no attribute 'QUEUED'

[Milhouse]

@Levi59 - many thanks, I've just pushed a fix (v1.4.1).

[theowiesengrund]

Thank you, you were correct about the cast thumbs. I set the option to "no" and caching worked!

But do you have any idea, what's going wrong here?

Code:

OpenELEC:~ # ./texturecache.py qax movies
Traceback (most recent call last):
  File "./texturecache.py", line 6391, in <module>
    main(sys.argv[1:])
  File "./texturecache.py", line 6247, in main
    extraFields=_extraFields, query=_query)
  File "./texturecache.py", line 3394, in jsonQuery
    qaData(mediatype, jcomms, database, data, title_name, id_name, rescan)
  File "./texturecache.py", line 3847, in qaData
    files = data.get("result", []).get("files", [])
AttributeError: 'list' object has no attribute 'get'

[Milhouse]

But do you have any idea, what's going wrong here?

Sorry about that, can you try v1.4.2...

Thank you, you were correct about the cast thumbs. I set the option to "no" and caching worked!

Great. I'm still looking into reducing the memory footprint, there's one possible option which trades outright performance for reduced memory usage (not necessarily an issue if you're running overnight, right?). Will let you know if I get anywhere... Obviously it would be an option to "switch on" if you are having memory issues.

[Levi59]

@Levi59 - many thanks, I've just pushed a fix (v1.4.1).

seems to work! Thanks to you

[theowiesengrund]

But do you have any idea, what's going wrong here?

Sorry about that, can you try v1.4.2...

Thank you. But i spotted another problem(?):

Code:

OpenELEC:~ # ./texturecache.py imdb movies
libMovies.ERROR: JSON response has no result!
{u'jsonrpc': u'2.0', u'id': u'libMovies', u'error': {u'message': u'Invalid params.', u'code': -32602, u'data': {u'method': u'VideoLibrary.GetMovies', u'stack': {u'message': u'array element at index 3 does not match', u'property': {u'message': u'Received value does not match any of the defined enum values', u'type': u'string'}, u'type': u'array', u'name': u'Item.Fields.Base'}}}}

[Milhouse]

If you have a full logfile I should be able to work out the reason, unfortunately I can't from just the above. Have you specified any additional fields for imdb.fields.* ?

[theowiesengrund]

Code:

2014-01-27 18:44:40.100373:MainThread: Command line args: ['./texturecache.py', 'imdb', 'movies']
2014-01-27 18:44:40.104748:MainThread: Current version #: v1.4.2
2014-01-27 18:44:40.105440:MainThread: Current platform : linux2
2014-01-27 18:44:40.106107:MainThread: Python  version #: v2.7.3.0 (final)
2014-01-27 18:44:40.118291:MainThread: libVersion.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "method": "JSONRPC.Version", "id": "libVersion"}]
2014-01-27 18:44:40.120952:MainThread: libVersion.BUFFER RECEIVED (len 89)
2014-01-27 18:44:40.123853:MainThread: libVersion.PARSING JSON DATA: {"id":"libVersion","jsonrpc":"2.0","result":{"version":{"major":6,"minor":13,"patch":3}}}
2014-01-27 18:44:40.128270:MainThread: libVersion.PARSING COMPLETE, elapsed time: 0.004736 seconds
2014-01-27 18:44:40.129134:MainThread: libVersion.FINISHED, elapsed time: 0.010053 seconds
2014-01-27 18:44:40.130425:MainThread: libPVR.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "params": {"booleans": ["System.GetBool(pvrmanager.enabled)"]}, "method": "XBMC.GetInfoBooleans", "id": "libPVR"}]
2014-01-27 18:44:40.138432:MainThread: libPVR.BUFFER RECEIVED (len 85)
2014-01-27 18:44:40.139499:MainThread: libPVR.PARSING JSON DATA: {"id":"libPVR","jsonrpc":"2.0","result":{"System.GetBool(pvrmanager.enabled)":false}}
2014-01-27 18:44:40.140399:MainThread: libPVR.PARSING COMPLETE, elapsed time: 0.001018 seconds
2014-01-27 18:44:40.143237:MainThread: libPVR.FINISHED, elapsed time: 0.010020 seconds
2014-01-27 18:44:40.994360:MainThread: libMovies.JSON SOCKET REQUEST: [{"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "title"}, "properties": ["title", "art", "imdbnumber", "ratings", "votes"]}, "method": "VideoLibrary.GetMovies", "id": "libMovies"}]
2014-01-27 18:44:41.002592:MainThread: libMovies.BUFFER RECEIVED (len 334)
2014-01-27 18:44:41.004086:MainThread: libMovies.PARSING JSON DATA: {"error":{"code":-32602,"data":{"method":"VideoLibrary.GetMovies","stack":{"message":"array element at index 3 does not match","name":"Item.Fields.Base","property":{"message":"Received value does not match any of the defined enum values","type":"string"}," (truncated)
2014-01-27 18:44:41.005268:MainThread: libMovies.PARSING COMPLETE, elapsed time: 0.001679 seconds
2014-01-27 18:44:41.014488:MainThread: libMovies.FINISHED, elapsed time: 0.015374 seconds
2014-01-27 18:44:41.017615:MainThread: Successful completion

Code:

imdb.fields = ratings, votes

[Milhouse]

It's rating not ratings.

Just realised my original post contained the "ratings" error - now corrected, apologies.

[dio_free]

Curl is a utility, like wget, to download files from web servers. If you want to use this script, copy & paste those installation commands into your ssh window (ie. PuTTY if you're on Windows, connecting to your OpenELEC client).

thanks millhouse, i will try this out.

[Milhouse]

texturecache.py (Version 1.4.3)

• Add: Implement chunked queries to reduce memory consumption on client and server (which are often the same device). In testing, chunked queries significantly reduced XBMC memory consumption, by as much as 80MB or more (a big deal on a Pi with under 200MB free). This change will ensure that queries use a relatively fixed (and small) amount of server memory when responding to chunked queries, rather than unpredictable and sometimes very large amounts of memory when responding to unconstrained queries (for example when retrieving all movies with all cast members, or the entire texture cache database).
  
  Media library queries will be chunked using JSON API limits (start/end), with the chunk size varying according to the anticipated complexity of the query, eg. when caching cast thumbs a smaller chunk size will be used due to the significant increase in data per movie.
  
  The Textures13 db query will also be chunked, though not using limits as this isn't currently supported by the JSON API (nor obviously SQL, which is used when the Texture DB JSON API isn't available). Instead, data for each of the Thumbnail folders (0-9,a-f) will be retrieved in sequence (cachedurl like '0/%' etc.), which results in more manageable (but unfortunately still variable) amounts of data, and has required related functions (caching, pruning) to be re-written.
  
  In addition, chunking allows unnecessary query data to be eliminated sooner, reducing client memory consumption - for example, when caching artwork, details of cast members without thumbnails can be dropped as soon as each chunk is received which avoids storing often significant amounts of redundant information only to be processed and discarded later.
  
  Chunked queries is not currently enabled by default, but can be enabled with @chunked=yes - please do so and report any problems. If there are no problems reported in the next week or so I will enable chunked queries as the default setting. Other than a slight performance gain when not chunking (assuming you don't run out of memory), there should be no reason to disable this option and in fact I may also remove the non-chunked code at some point in the future.
  
  Other functions, such as texture database searches (s, S, x, X etc.) and orphan file checks (r/R) have not been updated to support chunked queries as it's not practical to do so, plus you are less likely to have memory problems when performing these operations.
  

  ---

• Add: readfile option added to return content of the named file, outputting contents to stdout when the output filename is "-" (not suitable for binary data) or to the named output file (which is suitable for binary data)
  

texturecache.py (Version 1.4.4)

• Fix: Cosmetic change
  

Chunked queries is a fairly major change in this release, one which should benefit anyone experiencing out of memory issues. Please test, as I intend for chunked queries to become the default (and possibly only) option in future.

[Milhouse]

Either nobody has tested @chunked=yes, or it's all working wonderfully... maybe we'll find out when I make it the default next week!

@theowiesengrund I'd appreciate your feedback as you were experiencing the OOM problem, cheers.

[redglory]

Milhouse I'll test it when i get the chance at home and will let you know my results

[Milhouse]

Cheers, much appreciated. I really don't want to have to support two access methods for long, particularly when the non-chunked method is prone to exhaust all available memory.

[theowiesengrund]

Sorry, didn't have the time before to check it out.

Just started

Code:

./texturecache.py c all

with

Code:

cache.castthumb = yes
cache.extrafanart = yes
cache.extrathumbs = yes
dbjson = no
chunked = yes

But will not be able to report till tomorrow morning (unless it is finished in the next half hour).