[Montellese]

This issue has been brought up in the forum many many times and now I'm running into a similar problem with the UPnP client import work I'm doing. We really need to find a way to handle the case where xbmc runs a library cleanup and a whole path (e.g. a NAS) is temporarily not available. Ideally we would recognize that not only a single media item is missing but a whole source (or a path in a source) and then throw a dialog on the screen asking the user whether to remove that source/path and all the media items it contains from the library or not.

The question is how do we best figure out that a whole source/path/server is missing? Do we just go up one level and do an exists check until we end up at the root of the path? Or do we directly jump to the root of the path or to the path defined in the source? IMO we should do as few of those "exists" checks as possible as they take quite some time when the source is accessed over a network and is not available.

PS: We should probably allow the clean library to be run in "quite" mode where we assume that all the questionable removals are denied by the user so that the clean library functionality can still be triggered by a remote application (e.g. a cron job) without blocking until the user finally looks at the screen.

[Martijn]

iirc this has been added by night199uk for music library.
https://github.com/xbmc/xbmc/commit/bd38...6505c8926d

at least do server is online and source if the folder set as source is still there. any folder below that could be that user removed the entire subfolder.

[Koying]

+1000

A way would be to go through thru the local sources and check their availability, asking the user what to do if they're not available.
We could also add a "removable" flag to the sources (for e.g. usb drive) which forces "ignore if not present".

If a path is encountered which do not belong to a source, we should assume ignore, for mysql/multi-homes systems.
For "quiet" mode, just assume ignore for all unavailable paths.

Then, in the cleanup, if a file is not available, and its path is not marked "ignore" -> remove

[Montellese]

I like the idea of a "removable" flag but the problem with that might be that you can add several paths to a source but we only have per-source settings and not per-path-in-source settings. So I could have a "movies" source with some paths to local hard drives and a path to a USB drive which is removable.

[Koying]

I like the idea of a "removable" flag but the problem with that might be that you can add several paths to a source but we only have per-source settings and not per-path-in-source settings. So I could have a "movies" source with some paths to local hard drives and a path to a USB drive which is removable.

Mmm... Adding paths to different devices under the same source? Can we do that?
AFAIK, 1 source = 1 root path, no?

[Martijn]

I like the idea of a "removable" flag but the problem with that might be that you can add several paths to a source but we only have per-source settings and not per-path-in-source settings. So I could have a "movies" source with some paths to local hard drives and a path to a USB drive which is removable.

Mmm... Adding paths to different devices under the same source? Can we do that?
AFAIK, 1 source = 1 root path, no?

i have one source added for movies with 4 different smb:// paths.
i.e.:
Source name: Movies
--smb:/ip_address/media1/
--smb:/ip_address/media2/
--smb:/ip_address/media3/
--smb:/ip_address/media4/

and if i wanted i can add any local or removable drive too

[Koying]

Oh, right. Never noticed that

Well, in this case, up to the user to organize his sources so that "ignorable" ones stick together, I guess, which seems logical to me, anyway.
That'll come naturally when they will bored being nagged about unavailable paths

[natethomas]

Out of curiosity, is there any way to do a "hide if not present"? I'm more thinking for future cases where a person might regularly add and drop library items (e.g. on a tablet they take back and forth from home to school or something). It'd be nice if XBMC saw that the material wasn't present and simply hid it, rather than requiring the user to delete the data and then re-scan at some future point.

This would probably make more sense after Jonathan has duplication figured out.

[jmarshall]

I'd tend to use parentpathid. That's the folder that the media item resides in (media item might be a show or a movie within it's own folder, or a movie.avi file).

i.e. generally there'll be more than one media item in the path corresponding to a given items' parentpathid. If that doesn't exist, it's likely that the user has either moved a bunch of stuff, or it's just offline. I'd mark it as unavailable and optionally prompt (certainly prompt in the case of a clean library being performed in the foreground where the progress dialog is up).

Cheers,
Jonathan

[Montellese]

I'd tend to use parentpathid. That's the folder that the media item resides in (media item might be a show or a movie within it's own folder, or a movie.avi file).

i.e. generally there'll be more than one media item in the path corresponding to a given items' parentpathid. If that doesn't exist, it's likely that the user has either moved a bunch of stuff, or it's just offline. I'd mark it as unavailable and optionally prompt (certainly prompt in the case of a clean library being performed in the foreground where the progress dialog is up).

Cheers,
Jonathan

Are you talking about an additional column in the "path" table that either points to the "root" path haven't been added to a source or to the actual parent path? How would you build that information on existing installations? Just go through the "path" table for every item and see if there's a parent path based on string compares?

[Ned Scott]

Regarding how we manage sources that sometimes have one path and sometimes have multiple paths:

I have no idea how the programming/technical side of this would go, so if this is a stupid suggestion then please feel free to ignore it. From a user/GUI standpoint, I think it would make a lot of sense to migrate away from using file sources as how users manage library paths.

A dedicated sources manager that reads directly from the database would be cool. It would help users identify situations where they have two paths that point to the same files, causing duplicates, as well as orphaned paths/sources that don't exist in the GUI but still populate the library. It would also be a way to manage library paths from "Settings", which has been talked about in the past.

We could even still let users enter/set scraper info from a file source, but that would just be a way to automatically set that info on multiple paths. They could then set a "removable" flag in the "library path manager" on an individual path.

[jmarshall]

Montellese: No, I meant that the movies table, tvshows table, episodes table has a parent path ID column. That specifies the path that the object will be listed in - it's used for loading metadata when listing the path in Video->Files

(we know the path -> we know the pathid from the table -> we can find all movies/shows/episodes that are items in that path -> we can assign metadata to those items).

This is effectively the parent path of any of those items, whether they be a folder containing a movie or a folder for a tvshow (or even a folder containing a tvshow episode).

i.e. ideal for determining a parent during library clean: ATM we check file existence of each of the items in the database. We'd then also check the existence of it's parent path. If neither are there, don't clean. If parent is but file isn't, clean.

Cheers,
Jonathan

[Koying]

Unless mistaken, the path table contains each and every path containing a library file, so that won't work.

Say someone no as much attached to the law as we are download the Wolverine torrent. It will have the .mkv and the .nfo inside a created directory. This directory will be in the path table.
Say this someone deletes the torrent. This will delete both the mkv and the directory so according to your rule, the file will never be cleaned.

Seems more logical to go the sources route to me. As outlined above, that would allow high-level control on what to clean and how.

[jmarshall]

If the directory in question is "Wolverine.2013.x264.DTS5.1.FeLoNy.RoXerS" then if they scan it with "movies in separate folders named after the movie" it will be cleaned, as it's parentpathid will point to the parent folder "downloaded_movies" which still exists.

The problem with going via sources is not everything is a source. You can set content on any folder you like - it doesn't have to be a source - indeed, you can have content set on folders that aren't part of sources at all (and can have no sources at all and still have content set on paths in the database). I agree that some cleanup work here could be done, but that's a pretty large intrusive change if we wish to remedy this for Gotham.

The parentpathid is the logical place to start as the first path up the tree that's likely to still be present if the item has been deleted.

You might wish to go further up the filesystem tree if the parent path also doesn't exist - you could do something similar to what GetScraperForPath() does for example and stop once you reach that point (as content has been set on that folder). Indeed, just refactoring that might be all that is required? Note that this makes assumptions that the filesystem tree is hierarchical - it's not in general, but probably is for the cases we need to care about here.

Cheers,
Jonathan

[Martijn]

merged in PR3708