[pilluli]

Hi,

I've been thinking for several weeks about it now and I think XBMC needs a standard way to read item info in the background. Right now, if I am not mistaken (possibly I am) thumbnails are the only thing read/created on the background for items in the library. In any case, if more data is read in the background, I could not see a unified or unique place in the code where this is done.

I also think these other data could be set/read in the background:

• Availability (see thread): File path is checked to see if it is available and marked if not [useful in the file is in removable USB disks, or offline nas, etc.].
  
• Subtitles (see ticket): So streamdetails can be updated with subtitle info in external files [useful if user has new external subtitles available].
  
• Hash check to re-read media info if file is different or the first time [useful if user change the file keeping the name, I guess to update to a better version, don't know if this is very common though as I rarely have done it].
  

Ideally this can be read each time the database is pulled. The thing I've got in my mind is that when a list of items is pulled from the database then the SQL does its magic and then foreach item a background process reads all the above information (so no delay is added).

I wanted to start trying to code something for this. So I'd like to ask to the devs, what do you think? Having a unified/common way to read background info for items is useful? Are these options outlined before ok?

Regards and happy new year,

[jmarshall]

See ThumbLoader.cpp. Both stream details and thumbs are handled there - no reason other stuff couldn't be.

For updating stuff you'd need a hash per file which is something the new library intends to supply.

Cheers,
Jonathan

[pilluli]

See ThumbLoader.cpp. Both stream details and thumbs are handled there - no reason other stuff couldn't be.

As always thanks for the info Jonathan. The name was a bit misleading but I see it now. What do you think is the best course of action now?

- Create a new class similar to CThumbExtractor only to check existance of file. Actually, I think three classes (thumb + stream details + existance) will be even better but haven't really checked if thumb extraction and stream details need each other...

- Modify the above class to do all thumb + stream details + existance? However, as it is now at least a class name change should be done.

Regards,

For updating stuff you'd need a hash per file which is something the new library intends to supply.

OK, I'll wait for the new library to be up and running before adding that.


Regards,

[jmarshall]

Currently we add 2 jobs to the job manager for an item with no info (one for stream details, one for thumb extraction). No reason you couldn't add a third.

I'd be tempted to do the exists check directly in the main thumbloader thread though.

Cheers,
Jonathan

[spiff]

for the sake of correctness; we only add one job, either a job that only extract stream details or a job that extracts thumb and stream details.

[pilluli]

I'd be tempted to do the exists check directly in the main thumbloader thread though.

I didn't know that the main Thumbloader is a different thread. If so and based on my tests I think the Exists() checking can be easily done in the main ThumbLoader thread. In my tests, checking the existence of 1000 files in a nfs shared folder over my home wireless only took 40ms.

I will try that solution and post a patch if I manage Modifying the current Thumbextractor job was "almost" working but I think adding a job just for that was overkill.

My only concern now is what happens when a file item is found to be unavailable. I see in the ThumbLoader code that a message is sent to notify that an item is modified (and I guess the GUI reacts). Does this message also refresh the entire video list? What I am thinking is, if we filter by availability (similar to by watched status), then an item found unavailable must be deleted from the list. Does the existing message accomplish that? Or do I need to send a different one?

Regards, and feel free to ignore me if I ask too many questions for a simple patch!

[spiff]

the GUI_MSG_UPDATE_ITEM just updates a single item inplace.
send GUI_MSG_UPDATE to force an update of the entire fileitemlist.

[pilluli]

I am still struggling to make this patch to work.

I have modified the CThumbExtractor class to check existance (with FileItem::Exists()) with a new bool variable and change an item property accordingly. Also, I have modified CVideoThumbLoader::OnJobComplete() function to send a GUI_MSG_UPDATE instead of GUI_MSG_UPDATE_ITEM when the availability is changed. However:

- List does not get updated in the GUI. I have created a new filter that does not show unavailable items. However, even if they are set to be unavailable the list does not change. (i was working before when I set for FileItemExists() directly in the VideoDatabase.cpp file though). Maybe it is related to the next point as I think it might be due to using GUI_MSG_UPDATE message only on one fileitem pointer and not the entire list ...

- Performance is noticeable slower than checking it in the main thread. I think that may be due to sending a gui message for each unavailable item. The solution might be sending only one message at the end of all files but.. how can I modify the CVideoThumbLoader class to send a message at the end of all items?

In summary, is it possible to send a message after all thumb/details/availability jobs have finished? If so, where is the best place?

Again, sorry for the questions.


In case anyone wants to take a look at the patches here is a first version where everything works but availability is done synchronously when reading items from the database:

http://pastebin.com/ma42bad5

And here the version I am trying to make using the thumbextractor job.

http://pastebin.com/m788ec281

In both of them I mapped "SendClick(6,20)" into any key to filter items by availability.

[pilluli]

Hi,

I've made some progress on this but I need some help now

I have been able to create a Job that checks for file existence in all files and set a property accordingly. It is done in the background and seems to work fine. What I am doing is setting the "OFFLINE" property to false when an item is read from the database (in functions in VideoDatabase.cpp) so by default all items are online/available and then the job changes the property to true if not found.

However, I am not able to notify the GUI to refresh the list when the job has finished. If I use the MSG_GUI_UPDATE in the entire file list then the list is read again from the database and another Job (in VideoThumbExtractor) is created and as the "OFFLINE" property is reset it changes again and it gets in a loop.

Is it possible to refresh the GUI (call the OnPrepareFileItems() function) without re-reading items in the database instead of sending the MSG_GUI_UPDATE message?

If not, is it possible to somehow set the "OFFLINE" property to be "stuck" to a FileItem between reads in the database?

PS: As you can see I am a little lost ...

[jmarshall]

I'd be tempted to just set a property and not worry about refreshing the list - instead, refresh the item and leave it up to the skinner to colour code the item differently or otherwise flag that it's unavailable.

Cheers,
Jonathan

[pilluli]

I'd be tempted to just set a property and not worry about refreshing the list - instead, refresh the item and leave it up to the skinner to colour code the item differently or otherwise flag that it's unavailable.

Thanks Jonathan, I've got that working fine finally. I even modified the confluence skin to display an icon and dim the item if unavailable. I am sure Jezz can do a much better job at that though

Here is a small screenshot with the result. The icon is the little triangle at the right.



However, call me stubborn, but right now it is a personal challenge but I am going to give it one more try to create a filter for unavailable items and be able to refresh that list. To be honest it might be just me but we find it very useful to stick one of the USB disk or DVDs and easily see what movies are available there (within the library views).

I am going to try defining a new GUI_MSG but instead of calling a normal Update() I will try to do the same calls Update() does except the database re-read.

[jmarshall]

If you want them filtered out, then filter them out in OnFilterItems. I think there's already a message that calls this for you.

Cheers,
Jonathan

[pilluli]

If you want them filtered out, then filter them out in OnFilterItems. I think there's already a message that calls this for you.

Finally, I've got it working. Couldn't find a message to call OnFilterItems so I created one, also I separated OnFilerItems and OnFilterUnavailableItems for clearness but could be easily merged.

I'll post a patch in trac with all changes to see if it is ok...

regards and thanks for the help.

[pilluli]

Ok, I have found some time to post the patch. It is in this ticket.

If you devs think the patch is OK I might give a try in supporting loading stream details for external subtitles.

Regards,

[jmarshall]

Cheers for the patch, will take a look.