Kodi Community Forum

Introducing merge.sh version 0.1 [Linux only]

What this does:

This detects when you add any hard drive or other mass storage, or open a network share, and if it finds a directory named Music, Pictures or Videos, it overlays it onto the corresponding folder in your home directory.

It can do this for any number of drives at once. If you add, say, three drives and they all have a Videos folder, then ALL of the files in the Videos folders from ALL three drives will show up, at once, in the Videos folder in your Home folder.

If you then write to the Videos folder on the home folder, it will actually write the file to the drive with the most free space (or at least, the one that had the most free space the last time you added or removed a drive/share).

If you remove a drive or share, only the files on that drive/share are no longer there. All of the files from other drives/shares still show up. Removing a drive/share that you're not reading from or writing to should have no effect on the other drives/shares.

If you have sub-folders, such as TV Shows or Movies under Videos, these work too. If they exist on two or more drives/shares, the all of the files from both drives/shares show up under the one folder.

This is a simple script that gets a couple of programs to do all of the heavy lifting. It's not so clever by itself, and there's lots of room for improvement.

You'll need inotifywait (inotifytools package in all major distributions) and unionfs-fuse (package in all major distributions). For non-Linux, it might work if it's Linux-like enough (OSX, Solaris, BSD). Windows is right out, even with cygwin. You'll also need a fairly up-to-date box, but nothing special; I tested this mostly on Ubuntu 10.04.

Q Does this work for NTFS drives?
A This works for any filesystem that the OS can mount. It operates on the mount points and not the filesystems themselves, so if you can see the drive/share, it will work. So yes, if you have ntfs-3g or whatever.

Q How do I add another drive/share?
A Easy! Mount the drive/share (just by plugging it in, or navigating to the share, like with any other drive/share). If it has a Music, Pictures, or Videos folder, they will be merged. If not, you can just create those folders, safely remove the drive/share, and re-attach/re-navigate to them; now that they have those folders, they'll work.

Q I've lost all of my Music/Pictures/Videos that were in my home folder before the merge!
A They're not gone; they just don't show up when other folders are mounted up over the top of them. Remove all other drivers/shares and they will be visible again.
Really, this is intended to have a boot drive that you don't use except for the OS, and all of your storage on portable hard drives or network shares.

Q It didn't add my folder!
A The folder must be in the root of the drive/share; the script won't go hunting for it. But you could always mount the lower folder manually into /media/, and add it that way.

Q How can I run this every time I start up?
A In Debian/Ubuntu, go to the System menu, select Preferences, Startup Applications. Anywhere else, I'm not too sure, but it should be easy enough. I'll improve this answer as I test on more platforms.

Q I unmounted a drive or share, and it caused problems!
A This can happen, this is a pretty simple script.

• It's best to cease all drive activity before you try to remove a drive or share. But (especially for reads/playback), it should actually keep working, as long as the data wasn't on the drive/share in question.
  
• If you remove a drive or share that you're currently reading from or writing to (because a file that you've opened happens to physically be stored there), then obviously that will cause problems.
  
• If you are writing multiple files to another drive, the re-configuration when you remove a drive might cause them to be written to the home folder, which of course is invisible once you run the script. You'll have to end the script, possibly remove the shares manually, and then copy those files to another drive/share/folder.
  

Q What do I do if I want to make sure that a file I create/copy/move goes to the drive/share I want?
A There are a few ways:

• Obviously, before you begin, you could remove all other drives/shares, so that there is only one.
  
• Alternately, you could just open the drive itself, rather than the Home folder. It's still mounted and can be used normally.
  

Q It says it can't write, but I have lots of space!
A When you add or remove a drive/share, it remounts all of the available folders, and by default, any writes will go to whichever one had the most free space at this point. There are two exceptions, though:

• If a sub-folder exists on only one drive, writes to that sub-folder will go only to that drive. For example, if you had Drive A which had Videos/TV Shows/, and Drive B which had Videos/Movies/, then even though both folders would show up in ~/Videos, writing to ~/Videos/TV Shows/ would always go to Drive A, and writing to ~/Videos/Movies/ would always go to Drive B. The simple solution is to go to the drive which does not have whichever folder, and create it there.
  
• At the moment, network shares mounted through the shell show up as having zero bytes free, so you can never write to them unless all drives/shares have zero bytes (and by coincidence they overlay earliest). This could perhaps be fixed; I'm working on it. The only workaround is to remove all other drives, or possibly to mount the share into /media/ manually using cifs.
  

Q CD-ROM/DVD-ROM, etc. drives don't get added.
A That is specially disabled at this time. It's not impossible, but for the problem I was personally trying to solve, it's not useful because you can't write to them, and they're slow, and hard drive storage is cheap.

Q I deleted/renamed a file, and it's still there!
A It probably existed on more than one drive/share. Delete it again; once it's gone from all drives/shares, it's gone altogether.

Q I created/copied a new version of a file, but I keep seeing the old one.
A Again, it probably exists on more than one drive/share. Delete/rename the old one, and then the new one should show up.

Q How can I manually unmount the merged folders?
A fusermount -u -z ~/Music ; fusermount -u -z ~/Pictures ; fusermount -u -z ~/Videos

Q It doesn't work.
A Could be for a lot of reasons, including:

• You have xbmc standalone, without the main OS shell. I haven't even thought about trying to make this work, and it's not likely to.
  
• You're running a user account that is not a member of the fuse group.
  
• All of your drives and shares are completely full, and you're trying to write more to them. Free some space (or add another drive)!
  

Q What does this have to do with XBMC?
A Strictly speaking, nothing. I wrote it for a machine that was going to be used as a HTPC running XBMC, and am talking about it here because I thought others might find it useful too. But in principle, there's no reason it wouldn't work for any media center, or in fact any purpose at all.

Q How can I contact you?
A PMs through the XBMC forum, or posts in this thread. I'm not ready to put my name to it yet, so there will be no other contact details until I do.

POSSIBLE FUTURE ENHANCEMENTS

• Use xdg-user-dirs so that it uses the local name for the folder depending upon the language (for example, for German instead of Music, Pictures, Videos you would have Musik, Bilder, Filme).
  
• Arguments to close or open shares explictly (rather than waiting for a device event), or end the script.
  
• Let users specify what folders they want this behaviour for.
  
• Let users specify what folders to mount, and where.
  
• Let users specify a different method of determining the order of folders (so that reads/writes happen for the media of their choice).
  
• etc.
  

Download the script: [url="http://rapidshare.com/files/449649498/merge-1.0.zip]merge-1.0[/url]

I'm not quite sure what this script does over a simple one line fuse command, but I digress. Wouldn't it be better instead of relying purely on gvfs (which isn't a given in most linux setups, especially stand-alone htpc setups) wouldn't it be better to have a config file which would work with any internal hard drive or external hard drive path?

Also XBMC has "fuse-like" functionality by simply adding multiple paths to a library / video file path.

To make it start up automatically without relying on a GUI, look up how /etc/init.d or /etc/rc.d work depending on which linux flavour you're using. There is also an ncurses based text tool called sysv-rc-conf which can be installed in ubuntu or debian by doing
sudo apt-get install sysv-rc-conf

Also as another note, using unionfs on external drives and not using proper unmount procedures which isn't mentioned in your tutorial can quite easily cause drive corruption. With unionfs even if you don't think there are writes happening on your drive there is no guarantee, by design it makes multiple mount points operate as one, giving the user little control over what device is accessed or written to at any one time. This method really should be used with caution

ProphetVX Wrote:I'm not quite sure what this script does over a simple one line fuse command

Not much. It basically is a couple of one-line fuse commands, with some mountpoint-detection logic to automate it. So that you don't have to modify the script when you want to include new drives.

ProphetVX Wrote:Also XBMC has "fuse-like" functionality by simply adding multiple paths to a library / video file path.

Sure, but then you have folders like "Movies 1", "Movies 2", etc. or "TV Shows A-M", "TV Shows N-Z", etc.
This merges everything together. Some people prefer that. They're not interested in the physical limitations of storage.

ProphetVX Wrote:To make it start up automatically without relying on a GUI, look up how /etc/init.d or /etc/rc.d work

The script has to be run by the logged on user, because fuse drives mount for users, after logon. You could, of course, have the drives fixed in the system fstab and mount with the system, but then you could just add these merge commands there. (If you knew enough to do that, you would). But you'd have to manually reconfigure it every time you wanted to add or remove storage.

ProphetVX Wrote:Also as another note, using unionfs on external drives and not using proper unmount procedures which isn't mentioned in your tutorial can quite easily cause drive corruption. With unionfs even if you don't think there are writes happening on your drive there is no guarantee, by design it makes multiple mount points operate as one, giving the user little control over what device is accessed or written to at any one time. This method really should be used with caution

This script takes already-mounted filesystems and merges them. To unmount a filesystem, you use the normal procedure (safe remove). This script will then detect that the drive has been unmounted, and reconfigure the merge based on what's still there. This has nothing to do with the drives themselves; it relies on the OS (or shell) to do everything. All it does is update the merge based upon what mountpoints are visible.

If you unmount a drive normally, it will be safely removed, as usual. If you try to write to the merged directory while this is happening, the next available drive is used. This is all handled at a much lower level than what I'm doing. I'm just saying that IF a drive dies, or loses power, or a network share goes down, etc. then it won't have any impact except upon that one drive. Just like if you weren't using this script.

If there are writes happening that you don't know about, the normal handling kicks in. Either the write is cut off and fails, because the unmount happened; or else the unmount is blocked, because a write is in progress. Just as if you were writing directly. There shouldn't ever even be writes that you don't know about, though. It only merges Music, Pictures and Videos folders. It doesn't affect other locations, so caching and metadata and OS program loading so on will all work unless they need to write to the media files themselves. Again, if this happens, then it would be the same problem without this script.

Also, it uses lazy unmount, which means it just starts sending future reads/writes to the next merged drive, and when all current reads/writes are complete, then the unmount actually happens. All totally transparent to any application. That's why there is no problem unmounting a drive while you're doing something, if what you're doing does not affect that drive. If it can't unmount because it's in use, then that's the same as without this script. Stop what you're doing and try again!

Media Centers tend to be much more modal than general-purpose computers. That is, people perform one task at a time. If you're watching a movie, you're probably not doing anything else. Okay, so sometimes, you're ripping a CD at the same time. But, basically, you're not multitasking. So if you remove a drive deliberately (by using safe remove), you're very probably neither reading from nor writing to ANY location. Even if you are, as long as the location isn't physically the drive that you're removing, you're unaffected by the way that this script works (with a few possible edge cases that I'm testing for). And even then, it'll be the normal OS errors (disk full; share inaccessible, etc). that you'll get. By the time you've clicked OK and are trying again, it's probably reconfigured and working. In any case, there's no more chance ; the worst case is that things might get put on another drive than you expected, which in turn could perhaps result in a disk full error. That's the worst of it.

All this script really does is blur the line between what is stored where. All Videos on all drives look like they're in your Videos folder in your home folder. You don't have to care where they really are. Most people don't care about the artificial distinctions between multiple storage locations. Modern filesystems are trying to abstract away the details of the storage; it's the general direction that they're all heading in, it's just that it's still a field of ongoing research because, basically, people are sort-of used to hierarchical filesystems now. This lets you use a simplified approach for this special, uncomplicated, predictable case of a media center PC. There's really only a few predictable categories of file; Music, Pictures and Videos, so it's easy.

Thanks for the feedback, though!