[charrua]

It seems this has to be extracted from the media on the fly, if it's not already specified in the NFOs, and this is only done the first time the item is listed (or played back)... once that information is extracted, and stored in the media library, it should be much quicker to calculate the total runtime of a given list of items.

Sorry, I thought he was talking only about the total duration of all the media being displayed in a list, but now with your explanation I've realized that he was referring to each media stream details (stored in the DB table streamdetails).

[un1versal]

I found this
6.20.4 Video.Details.File
JSON-RPC_API/v6#Video.Details.File (wiki)
6.20.20 Video.Streams
JSON-RPC_API/v6#Video.Streams (wiki)

idk if its any help as I dont understand what it does exactly but the code examples seem to point to the flags and total duration which covers both points.

uNi

[Milhouse]

Yep, tried both, no luck. As confirmed by Montellese (the resident JSON expert), it's simply not possible to kickstart the stream details extract via JSON for the time being. So although JSON promises the possibility of providing this useful information, unless the scraper has pulled in the stream detail information at the time the media is scraped in you're going to be sh1t out of luck as the dynamic stream detail import process is a GUI only feature (and thus of no use from a JSON perspective).

The only other option right now is be to populate the local NFOs with the relevant stream/run-time information before scraping/updating the library. I typically use Sickbeard for my TV Shows, I'll ask there tomorrow to see if there is any possibility of improving the information present in the NFOs that Sickbeard creates but suspect the answer will be no...

[un1versal]

I understand.

But no, nfo is not an option though either way, I never needed a nfo and I suspect that need is not going to change soon. thanks again.

uNi

[User 34959]

Thanks for the script, really makes a fresh library setup much smoother experience with the precaching feature.

[hydra75]

Thx for your work, MilhouseVH...

I want to speed up my Raspberry running on Openelec 3.0.0 and a MySQL DB.
But...

Code:

openelec:~ # ./texturecache.py c movies
Querying localhost:9090 for all Movies...
Movie: #9                                        [ . fanart     ]  [ . poster     ]
Movie: ...denn sie wissen nicht, was sie tun     [ . fanart     ]  [ . poster     ]
Movie: ...und dann kam Polly                     [ . fanart     ]  [ . poster     ]
Movie: .45                                       [ . fanart     ]  [ . poster     ]
Movie: 3 Idiots                                  [ . fanart     ]  [ . poster     ]
Movie: 3 Schweinchen und ein Baby                [ . fanart     ]  [ . poster     ]
Movie: 3D Sex and Zen: Extreme Ecstasy           [ . fanart     ]  [ . poster     ]
Movie: 4.3.2.1 - 4 Girls, 3 Tage, 2 Städte, 1   [ . fanart     ]  [ . poster     ]
Movie: 7 Psychos                                 [ . fanart     ]  [ . poster     ]
Movie: 7 Sekunden                                [ . fanart     ]  [ . poster     ]
Movie: 7 Zwerge - Der Wald ist nicht genug     Traceback (most recent call last):
  File "./texturecache.py", line 1869, in <module>
    main(sys.argv[1:])
  File "./texturecache.py", line 1815, in main
    libraryQuery("cache", argv[1])
  File "./texturecache.py", line 617, in libraryQuery
    libraryAllMovies(action, filter, force, extraFields, rescan, decode, isTag=False)
  File "./texturecache.py", line 871, in libraryAllMovies
    loadImage(force, artwork, art[artwork], imagesCache, imgErrors)
  File "./texturecache.py", line 1303, in loadImage
    URL = getDownloadURL(filename)
  File "./texturecache.py", line 1250, in getDownloadURL
    data = sendJSON(REQUEST, "1")
  File "./texturecache.py", line 491, in sendJSON
    data = sendWeb("POST", "/jsonrpc", json.dumps(request), {"Content-Type": "application/json"})
  File "./texturecache.py", line 466, in sendWeb
    web.request(request_type, url, request, headers)
  File "./Lib/httplib.py", line 958, in request
  File "./Lib/httplib.py", line 992, in _send_request
  File "./Lib/httplib.py", line 954, in endheaders
  File "./Lib/httplib.py", line 814, in _send_output
  File "./Lib/httplib.py", line 776, in send
  File "./Lib/httplib.py", line 757, in connect
  File "./Lib/socket.py", line 571, in create_connection
socket.error: [Errno 111] Connection refused

... got this error.

Any idea? A hint?

Thx

[Milhouse]

My guess would be that you haven't enabled the web server in XBMC.

Go to System -> Settings -> Services -> Webserver, enable "Allow control of XBMC via HTTP" The default port (for texturecache) is 8080 but you can use whatever port you like as long as you specify the new setting as webserver.port=n (where n is your port) in the properties file.

You can also use username and password now, again set them in the properties file (see first post for more details).

[Milhouse]

v0.3.1 - a pretty major update.

• Multiple thread support added to the c and C options. Default is 2 threads, set your own value using the download.threads = n property. This is a "global" thread setting for all media classes (albums, movies, tvshows etc.), and you can individually fine-tune the number of threads by specifying additional per-class properties, eg. download.threads.albums = 2, download.threads.songs = 5, download.threads.tvshows = 10, and for everything else downloads.threads will be used.
  
  
• New option: nc. This is essentially the same as c but doesn't download anything - think of this as a dry run version of c, it will show you what needs to be cached. Isn't multi-threaded, as there is no benefit when just parsing data.
  
  
• Lots of code re-factoring. I may have broken something, and some things that used to work may no longer work on older versions of Python. Sorry. Only able to test with v2.7.3, though a very quick spin on a Dharma/Python v2.6.6 machine didn't highlight anything obvious.
  
  
• Sanity checks added at script initialisation, so if your web server isn't running and it's needed for the option you're trying to use, then the script will inform you, and abort. Same for the JSON-RPC interface, JSON API version, and the sqlite3 Texture13 database.
  
  
• Source code now hosted on github: https://github.com/MilhouseVH/texturecache.py. Just download the latest version of texturecache.py rather than the whole repository, though you can do that too if you like.
  

Multi-threaded download has been tested on OpenELEC/Raspberry Pi, Ubuntu 12.10/x86 and also Windows (by charrua - many thanks).

Running with 10 threads is not a problem, not even for the Raspberry Pi (though see [1], below)! Caching performance with additional threads is significantly improved, though obviously beyond a certain number of threads it's a case of diminishing returns, so don't go too mad...

Enjoy.

---

1. Just one caveat... and it's a fairly major one... with the Raspberry Pi I'm seeing the web server fall over and effectively die - no further web requests accepted - and then a short while later taking the whole Pi with it, necessitating a power cycle.

This problem seems to be due to OMX errors (log of failure - nothing more appears after this - and below it, from a separate run, this time successful)... So far I'm only seeing this problem with tvshows on my Raspberry Pi, but that could just be random luck as it may be image dependent, and I've not seen this problem on Ubuntu, or heard of an issue on Windows. For all other media classes - albums, artists, songs, sets, movies, tags - these can pre-load with 10 threads, not a problem, but tvshows, just using one thread: power cycle. Hmm.

It's definitely not multi-thread related as I can provoke the problem with the old v0.3.0 "serial" version, and with the new version when download.threads.tvshows = 1, so effectively single-threaded. It even happens on the very first web request (which is for Files.Preparedownload), and sometimes after 900+, so definitely looks like an internal XBMC issue on Raspberry Pi (tested both stock OE release and latest Rbej build, unfortunately no difference).

Edit: Looking at that debug log again, it might not be Files.Preparedownload but the actual download - the incoming request isn't obvious in the log - and subsequent caching that is causing the webserver to fall over, certainly after this situation has occurred, no further requests will be serviced by the web server, and eventually the entire Pi will hang. When this problem occurs, the GUI will freeze immediately.

Edit2: Now fixed, this caveat no longer applies. See here - make sure Pi has 29 March firmware or later.

[Milhouse]

It's quite interesting monitoring free memory on a Raspberry Pi while pre-loading an entire media class, say movies, with almost 1200 items, as there seems to be a quite significant memory leak.

I'm using this shell script on a 512MB Raspberry Pi with 128MB GPU memory split. Pre-loading with 10 threads.

See this pastebin of the memory stats and period in question. Tested on latest OpenELEC built from git this morning.

Straight after booting, and without loading SYSTEM into RAM on OpenELEC (add noram to cmdline.txt), the Pi has 150MB RAM free, and I've left XBMC idle at the menu (actually in the Videos -> Files list, so as not to waste CPU cycles sat on the main menu). The pre-load process is started and as you'd expect, the free memory drops... but it keeps on dropping, and by the time the process is complete, the Pi is down to 22MB free memory. Almost none of the memory used is ever returned by the time the script terminates - I'm guessing the only memory that is returned, about 16MB, is that used by the Python script.

Memory leak, or can someone give a reasonable explanation for this loss of 110MB RAM?

Update: I just tried running the same "C movies" test again, without rebooting or doing anything on the Pi, starting from 19MB free memory and expecting the Pi to crash mid-test OOM (I have no swap), but bizarrely the amount of free memory is now increasing as the test proceeeds (see pastebin for before/during/after). The "C movies" pre-load started with 19MB of free memory, then increased throughout the entire test, as though memory were being released from some hidden reserves... finally finishing with 166MB free (not including ~14MB for Python?)

Bizarre!

[awsic]

... can't delete post.
Thanks for the script.

[Milhouse]

You've downloaded a web page, not the script.

Try:

Code:

wget https://raw.github.com/MilhouseVH/texturecache.py/master/texturecache.py

and for anyone on OpenELEC which has a pretty basic wget that doesn't support https downloads, use curl:

Code:

curl https://raw.github.com/MilhouseVH/texturecache.py/master/texturecache.py -o texturecache.py

I'll add these links to the original post.

[awsic]

Sorry, MilhouseVH for the previous post. It was a "D'OH!~" moment on my end. I later went to the site and found the correct file. Thanks for the updated wget ^^ above.

[un1versal]

@MilhouseVH

Congrats on Github move.

Also wanted to update you on texturecache.py used it with P, and while it emptied a lot of /userdata/Thumbnails content my Texturs#db satyed same size, so I guess it was a success though I half expected my Texturs.db to shrink, I guess I dint have anything worth removing inthere.

So far thumbs up and looking forward for this multithreaded thing with other switches too if that is ever a plan.

Congrats and thanks again for this tool and your dedication to its continued development.

uNi

[Milhouse]

@MilhouseVH

Congrats on Github move.

Thanks!

Also wanted to update you on texturecache.py used it with P, and while it emptied a lot of /userdata/Thumbnails content my Texturs#db satyed same size, so I guess it was a success though I half expected my Texturs.db to shrink, I guess I dint have anything worth removing inthere.

It should certainly be deleting both the database row and the corresponding cached file - you can confirm this by enabling the logfile option (logfile = filename) and then looking to see if any SQL deletions took place. Though I'm not sure how sqlite3 manages its database size as rows are removed - maybe it never shrinks the database (potentially expensive and counter productive) and holds on to the now free space to be re-used in the future? You can get a row count by running the x option and piping the results through "wc -l", or use the "f" option - there's a rowcount summary at the end, along with the amount of disk space used by the corresponding files.

So far thumbs up and looking forward for this multithreaded thing with other switches too if that is ever a plan.

Again thanks. Though not sure what other switches to add right now, I'm fresh out of ideas... currently just tweaking the code and hoping to get to the bottom of this Pi killer issue. If you've got any suggestions for the future, fire away!

Congrats and thanks again for this tool and your dedication to its continued development.

Not sure how long it will last - I only started it as a way to learn a new language, and to more easily extract rows from sqlite3 database - and now look at it...

[charrua]

Though not sure what other switches to add right now, I'm fresh out of ideas... currently just tweaking the code and hoping to get to the bottom of this Pi killer issue. If you've got any suggestions for the future, fire away!

Just two small cosmetic suggestions:
-show the processing time in the statistics
-put the TOTAL column as first one instead of last
(other than that, I believe that the script has reached the first stage of maturity with v0.3.1 )
Congrats on a job well done, my friend!

The next step (just in case you want it to gain popularity) is transforming the script to an ADD-ON (as previously suggested)