Login at Kodi Home

**scott967** · 2014-10-13, 06:29

Here is a problem I have. I tried using texturecache.py on it but maybe I'm not using the right options.

This is for music library. When new files are scanned, if a new album is found the first album artist will be assigned and the scanner will look for (local art) folder.jpg and fanart.jpg in the album's parent folder. If it exists it will be added to the album artist and in the database "art" table. The problem is if there is more than one album artist or compilation only (song) artists. There is no mechanism to assign local art for these during a scan. IF the artist is scraped, then it is possible to use the "artist info" screen to browse for local thumb and fanart. If the artist didn't scrape, "artist info" isn't accessible, without setting "lastScraped" in the artist table, either by "import library" or directly updating the artist table. Even when the "artist info"screen is available, it is very tedious to use the UI to browse and set local art.

What I am looking for is something that can go through the "artist" and "art" table and find all artists where thumb and or fanart are not set and provide some means of updating the art table. It is sort of possible to do this with the music library import, but the only way I get it to work is to use the

Code:
<thumb>...</thumb>

<fanart><thumb>...</thumb></fanart>

elements, but it seems like if you do this yes you can get local art into the art table but it locks out any url/online art that might have scraped.

Any ideas?

Milhouse · 2014-10-13, 09:36

(2014-10-13, 06:29)scott967 Wrote: Here is a problem I have. I tried using texturecache.py on it but maybe I'm not using the right options.

This is for music library. When new files are scanned, if a new album is found the first album artist will be assigned and the scanner will look for (local art) folder.jpg and fanart.jpg in the album's parent folder. If it exists it will be added to the album artist and in the database "art" table. The problem is if there is more than one album artist or compilation only (song) artists. There is no mechanism to assign local art for these during a scan. IF the artist is scraped, then it is possible to use the "artist info" screen to browse for local thumb and fanart. If the artist didn't scrape, "artist info" isn't accessible, without setting "lastScraped" in the artist table, either by "import library" or directly updating the artist table. Even when the "artist info"screen is available, it is very tedious to use the UI to browse and set local art.

What I am looking for is something that can go through the "artist" and "art" table and find all artists where thumb and or fanart are not set and provide some means of updating the art table. It is sort of possible to do this with the music library import, but the only way I get it to work is to use the
Code:
<thumb>...</thumb> <fanart><thumb>...</thumb></fanart>
elements, but it seems like if you do this yes you can get local art into the art table but it locks out any url/online art that might have scraped.

Any ideas?

I don't really know much about updating the media library database directly, particularly the music database - you might have more luck asking in the Universal Music Scraper thread. I've only concerned myself with JSON queries when extracting data from the media library (or making changes), but unfortunately there are no JSON methods that would allow you to make artwork changes in the music database.

wrxtasy · 2014-10-14, 09:51

Milhouse, could you help with this interpretation after running the script with the c option.

Is it correct on my large library I have gained only 28 items that needed cache-ing ?

Cheers

Milhouse · 2014-10-14, 10:48

Yes, it would look that way. "Skipped" would be items already in your cache, and "Duplicates" would be artwork used more than once (which only needs to be skipped/cached once). You've got some "Undefined" artwork that you might want fix - this would be library entries such as fanart="" (a qa check might reveal them, remember to set @qaperiod=9999).

wrxtasy · 2014-10-14, 12:17

Thanks, as expected a bit of missing mpaa's and obscure docos with no fanart, and lots of vgenres [tvshow, ***** and "vgenres [movie, *****]
This looks skin related and is confirmed by no genres subfolders in the extras folder contained inside the individual skin add-on directory.

What Milhouse's very useful script does confirm, is that using a good external media manager. One that parses filenames to store media in XBMC preferred individual folders and scrapes and stores full metadata within. This makes managing a large media library very easy. My RPi library has over 5000 media files and only 28 textures were not cached.

Happy as.

I would go so far as to suggest this should be mainlined into XBMC, with a some of simple GUI options. Wish I was a skilled coder.

Wink

un1versal · 2014-10-14, 12:34

For any of that to work, first the artwork and meta has to exist in the databases this gets scraped from, obscure stuff is hardly excluded from this.

Milhouse · 2014-10-16, 00:19

Version 1.7.7

Add: Support for @clean.showdialogs and @Scan.showdialogs, default no. With JSON API 6.19.0, JSON initiated library scans and cleans will not show any GUI progress or dialogs. Enable GUI progress/dialogs by setting these new parameters to yes.
Add: play, stop and pause options to start, stop and pause media playback. pause will toggle playback. Examples:

Code:
./texturecache.py play http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4 ./texturecache.py pause ./texturecache.py stop

redglory · 2014-10-16, 18:13

Thanks Milhouse!

This Clean feature was a must have for me as cleaning my kodi library was a PITA with that dialogs Smile

Milhouse · 2014-10-16, 23:33

Version 1.7.8

Add: playw option. Same as play, but wait until end of playback.

Forage · 2014-10-17, 19:18

After restoring an OpenElec backup I lost most of my fanart and posters. When trying this script I get errors exactly for those movies that lost it's images.

Code:
OpenELEC:~/downloads # ./texturecache.py C movies "2 Guns" @allow.recacheall=yes @download.threads=1

Caching artwork: 0 items remaining of 2 (qs: 0, qm: 0), 2 errors, 0 threads active (00.57 downloads per second, ETA: 00:00:00)

The following items could not be downloaded:

[fanart    ] [2 Guns                                  ] http://image.tmdb.org/t/p/original/wZsX7160l5feO646shRx3liMutx.jpg

[poster    ] [2 Guns                                  ] http://image.tmdb.org/t/p/original/30lM3Uvzs6HOG5l4hzhwxYTWgd3.jpg

Cache pre-load activity summary for "movies", filtered by "2 Guns":

              |    fanart   |    poster   |    TOTAL    

--------------+-------------+-------------+-------------

Cached        |      -      |      -      |      0      

Deleted       |      -      |      -      |      0      

Duplicate     |      -      |      -      |      0      

Error         |      1      |      1      |      2      

Ignored       |      -      |      -      |      0      

Skipped       |      -      |      -      |      0      

Undefined     |      -      |      -      |      0      

========================================================

TOTAL         |      1      |      1      |      2      

Download Time | 00:00:01.74 | 00:00:01.74 | 00:00:03.48 

  Threads Used: 1

   Min/Avg/Max: 0.57 / 0.57 / 0.57

       Loading: 00:00:00.24

       Parsing: 00:00:00.01

     Comparing: 00:00:03.60

   Downloading: 00:00:04.00

TOTAL RUNTIME: 00:00:07.90

OpenELEC:~/downloads #

There have been more post with similar issues, but those displayed a network or relative url instead of a plain accessible internet url.
Manually downloading one of the images with the following on the same system works fine so it's not the internet connection that's causing issues:

Code:
OpenELEC:~/downloads # curl http://image.tmdb.org/t/p/original/wZsX7160l5feO646shRx3liMutx.jpg -o wZsX7160l5feO646shRx3liMutx.jpg

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 1043k  100 1043k    0     0  3402k      0 --:--:-- --:--:-- --:--:-- 3442k

OpenELEC:~/downloads #

Enabling debugging, including component specific for curl, gives me nothing related in the xbmc.log. The following is all that's logged during the time the script is running:

Code:
54:20 317.615112 T:2755408976   DEBUG: Thread JobWorker 2755408976 terminating (autodelete)

54:35 332.493042 T:2773480528   DEBUG: webserver: request received for /jsonrpc

54:35 332.532013 T:2730243152   DEBUG: JSONRPC Server: New connection detected

54:35 332.532440 T:2730243152    INFO: JSONRPC Server: New connection added

54:35 332.556763 T:2730243152    INFO: JSONRPC Server: Disconnection detected

54:36 333.596283 T:2730243152   DEBUG: JSONRPC Server: New connection detected

54:36 333.596741 T:2730243152    INFO: JSONRPC Server: New connection added

54:36 333.710236 T:2730243152   DEBUG: RunQuery took 8 ms for 1 items query: select * from movieview  WHERE ((movieview.c00 LIKE '%2 Gun$

54:36 333.816772 T:2730243152   DEBUG: JSONRPC Server: New connection detected

54:36 333.817169 T:2730243152    INFO: JSONRPC Server: New connection added

54:39 337.169250 T:2730243152    INFO: JSONRPC Server: Disconnection detected

54:39 337.227020 T:2830906448   DEBUG: webserver: request received for /jsonrpc

54:43 341.207306 T:2730243152   DEBUG: Previous line repeats 7 times.

54:43 341.207581 T:2730243152    INFO: JSONRPC Server: Disconnection detected

Manually refreshing the movie also doesn't restore the images.
What I do get in the log when manually refreshing is:

Code:
15:21 1578.453003 T:2830906448  NOTICE: Thread JobWorker start, auto delete: true

15:21 1578.464722 T:2830906448   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found

15:21 1578.465088 T:2830906448  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg

15:29 1586.925171 T:2679911504  NOTICE: Thread VideoInfoDownloader start, auto delete: false

15:44 1602.097046 T:2679911504  NOTICE: Previous line repeats 1 times.

15:44 1602.097412 T:2679911504  NOTICE: Thread JobWorker start, auto delete: true

15:47 1604.883545 T:2679911504   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/685e3b42.jpg not found

15:47 1604.883911 T:2679911504  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/685e3b42.jpg

15:47 1604.886963 T:2830906448   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found

15:47 1604.887329 T:2830906448  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg

15:47 1605.216797 T:2830906448   ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found

15:47 1605.217163 T:2830906448  NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg

What could be the cause of the issue and how can I fix it?

Milhouse · (This post was last modified: 2014-10-17, 19:34 by Milhouse.)

Did you try enabling CURL component debugging? The problem is with the Files.PrepareDownload call or the download of the artwork by XBMC itself. If you enable texturecache.py logging with @logfile= you should see which step is failing, though why it would be failing is harder to answer - sometimes it's just one of those things that XBMC (annoyingly) does.

(2014-10-17, 19:18)Forage Wrote: What I do get in the log when manually refreshing is:

Code:
19:15:21 1578.453003 T:2830906448 NOTICE: Thread JobWorker start, auto delete: true 19:15:21 1578.464722 T:2830906448 ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found 19:15:21 1578.465088 T:2830906448 NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg 19:15:29 1586.925171 T:2679911504 NOTICE: Thread VideoInfoDownloader start, auto delete: false 19:15:44 1602.097046 T:2679911504 NOTICE: Previous line repeats 1 times. 19:15:44 1602.097412 T:2679911504 NOTICE: Thread JobWorker start, auto delete: true 19:15:47 1604.883545 T:2679911504 ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/685e3b42.jpg not found 19:15:47 1604.883911 T:2679911504 NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/685e3b42.jpg 19:15:47 1604.886963 T:2830906448 ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found 19:15:47 1604.887329 T:2830906448 NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg 19:15:47 1605.216797 T:2830906448 ERROR: COMXImageFile::ReadFile special://masterprofile/Thumbnails/6/6c1ec998.jpg not found 19:15:47 1605.217163 T:2830906448 NOTICE: LoadJpeg: unable to load special://masterprofile/Thumbnails/6/6c1ec998.jpg

I would advise running "./texturecache.py Xd" as it looks like your Textures13.db is out of sync with the contents of the Thumbnails folder, with the db rows referencing thumbnails that don't exist.

Forage · 2014-10-17, 19:42

(2014-10-17, 19:28)Milhouse Wrote: I would advise running "./texturecache.py Xd" as it looks like your Textures13.db is out of sync with the contents of the Thumbnails folder, with the db rows referencing thumbnails that don't exist

Worked like a charm! Muchas gracias!
I would have expected this to be unneeded when you are already forcing a re-cache/download with the C option.

Milhouse · 2014-10-17, 20:15

(2014-10-17, 19:42)Forage Wrote: Worked like a charm! Muchas gracias!
I would have expected this to be unneeded when you are already forcing a re-cache/download with the C option.

Indeed, however this is what happens during a force cache "C" option:

Code:
Acquire the url of the artwork to be cached by calling Files.PrepareDownload

If the url is obtained successfully in step #1, remove the existing artwork from cache (deleting the row and corresponding thumbnail file)

Use the url to download the new artwork, creating a new entry in the cache (new row and new thumbnail)

In normal (ie. working circumstances) this sequence acts as a safety net - if the artwork you are attempting to re-cache is no longer available to be cached then texturecache.py won't delete the existing thumbnail you already have in your cache, so at least you'll be left with "something" at the end of the process rather than nothing at all. Users might become rather hacked off if their cache is wiped out because of a temporary networking issue.

However in your case, the db is confused as it thinks it has a thumbnail when it doesn't and this is what causes step #1 to fail so steps #2 and #3 are never attempted. As the db is the main driver when determining if the artwork is "in the cache" or not, once your db is out of sync with respect to the thumbnails folder, "bad things" will start to happen.

It would be possible to add some sort of sanity check to determine if the reason for the failure in step #1 is because the thumbnail file doesn't exist (and if not to then remove the db row and repeat step #1), but this will slow things down and should not in normal circumstances be required. Also, there are several legitimate reasons why step #1 should fail, and querying the Thumbnails folder could become quite an expensive process.

I'll think about it though... I don't want to make it any messier than it already is. Smile

In the meantime, running "Xd" from time to time is probably the best idea, or just when "bad things" start to happen.

Milhouse · (This post was last modified: 2014-10-17, 22:24 by Milhouse.)

Update: I can add a fix, but it's only going to work if texturecache.py has direct file system access to the Thumbnails folder as the JSON Textures API "lies" about thumbnail file existence - it responds with details from Textures13.db even though the underlying cached file is missing, so the fix is not going to be ideal. Those that are maintaining remote systems won't see any change, but those running the script on a local system (or that have mounted the remote Thumbnails folder locally) will have the benefit of automatic "fixing" of any missing thumbnails.

Milhouse · 2014-10-17, 21:42

Version 1.7.9

Chg: When direct file system access to the Thumbnails folder is available, sanity check Files.PrepareDownload failures to determine if the thumbnail file is missing and if so delete the database row before calling Files.PrepareDownload again. See http://forum.xbmc.org/showthread.php?tid...pid1815613