Linux - Episodes on mhddfs source not being added reliably during library update

  Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
nickr Offline
Donor
Posts: 12,827
Joined: May 2009
Reputation: 244
Location: Christchurch NZ
Post: #16
Thank you, looks interesting!

If I have helped you or increased your knowledge, click the 'thank user' button to give thanks :) (People with less than 20 posts won't see the "Thank you" button.)
find quote
Wintersdark Offline
Senior Member
Posts: 103
Joined: Apr 2012
Reputation: 2
Post: #17
I'll look into that in the future, if I run into any problems with my current system.

I'm running aufs now rather than mhddfs - as a kernel level system, it's much faster. Built into the Ubuntu Server kernel, so there was no screwing around to get it up and running. Still have the fasthash issue with aufs, but it's been seamless since.
find quote
eskay993 Offline
Senior Member
Posts: 142
Joined: Nov 2014
Reputation: 1
Post: #18
Thanks for the mergerfs tip! Been testing it and worked great with latest Helix.

However, I'm having trouble with Isengard Beta 1, and latest nightly as of today. New episodes are not being picked up.

Fasthash advancedsetting doesnt work eitehr. Used to work in Helix, but also doesn't in Isengard. Have tried mhddfs and mergerfs. Not touched aufs in a while since I was having some issues with it.

Reason for using Isengard is to use my fav skin so keen to get it to work. Anyone else having this issue in Isengard?

Edit:
This is the output of "xattr -l .mergerfs". I set category.search=newest in fastab, but noticed func.getattr=newest is also set. Woudl these two conflict with each other?

Code:
user.mergerfs.srcmounts: /mnt/disk1:/mnt/disk2:/mnt/disk3:/mnt/disk4
user.mergerfs.category.action: all
user.mergerfs.category.create: epmfs
user.mergerfs.category.search: newest
user.mergerfs.func.access: newest
user.mergerfs.func.chmod: all
user.mergerfs.func.chown: all
user.mergerfs.func.create: epmfs
user.mergerfs.func.getattr: newest
user.mergerfs.func.getxattr: newest
user.mergerfs.func.link: all
user.mergerfs.func.listxattr: newest
user.mergerfs.func.mkdir: epmfs
user.mergerfs.func.mknod: epmfs
user.mergerfs.func.open: newest
user.mergerfs.func.readlink: newest
user.mergerfs.func.removexattr: all
user.mergerfs.func.rename: all
user.mergerfs.func.rmdir: all
user.mergerfs.func.setxattr: all
user.mergerfs.func.symlink: epmfs
user.mergerfs.func.truncate: all
user.mergerfs.func.unlink: all
user.mergerfs.func.utimens: all
(This post was last modified: 2015-05-20 14:43 by eskay993.)
find quote
jerms415 Offline
Junior Member
Posts: 8
Joined: Jul 2011
Reputation: 1
Post: #19
(2015-05-20 14:39)eskay993 Wrote:  Thanks for the mergerfs tip! Been testing it and worked great with latest Helix.

However, I'm having trouble with Isengard Beta 1, and latest nightly as of today. New episodes are not being picked up.

Fasthash advancedsetting doesnt work eitehr. Used to work in Helix, but also doesn't in Isengard. Have tried mhddfs and mergerfs. Not touched aufs in a while since I was having some issues with it.

Reason for using Isengard is to use my fav skin so keen to get it to work. Anyone else having this issue in Isengard?

Edit:
This is the output of "xattr -l .mergerfs". I set category.search=newest in fastab, but noticed func.getattr=newest is also set. Woudl these two conflict with each other?

Code:
user.mergerfs.srcmounts: /mnt/disk1:/mnt/disk2:/mnt/disk3:/mnt/disk4
user.mergerfs.category.action: all
user.mergerfs.category.create: epmfs
user.mergerfs.category.search: newest
user.mergerfs.func.access: newest
user.mergerfs.func.chmod: all
user.mergerfs.func.chown: all
user.mergerfs.func.create: epmfs
user.mergerfs.func.getattr: newest
user.mergerfs.func.getxattr: newest
user.mergerfs.func.link: all
user.mergerfs.func.listxattr: newest
user.mergerfs.func.mkdir: epmfs
user.mergerfs.func.mknod: epmfs
user.mergerfs.func.open: newest
user.mergerfs.func.readlink: newest
user.mergerfs.func.removexattr: all
user.mergerfs.func.rename: all
user.mergerfs.func.rmdir: all
user.mergerfs.func.setxattr: all
user.mergerfs.func.symlink: epmfs
user.mergerfs.func.truncate: all
user.mergerfs.func.unlink: all
user.mergerfs.func.utimens: all

Hmm. That's strange that it's not working in Isengard.

category.search is a superset of func.getattr, so if you set category.search to something, func.getattr will also be set to that same value. So your settings look correct. However, my settings are slightly different. I also set category.create to "mfs", since I was running into an issue where mergerfs was thinking I was out of disk space when I really wasn't. I don't know if you are also having that issue. Here's my output:

Code:
user.mergerfs.srcmounts: /media/btrfs1:/media/btrfs2:/media/btrfs4:/media/btrfs5                                   :/media/btrfs6
user.mergerfs.category.action: all
user.mergerfs.category.create: mfs
user.mergerfs.category.search: newest
user.mergerfs.func.access: newest
user.mergerfs.func.chmod: all
user.mergerfs.func.chown: all
user.mergerfs.func.create: mfs
user.mergerfs.func.getattr: newest
user.mergerfs.func.getxattr: newest
user.mergerfs.func.link: all
user.mergerfs.func.listxattr: newest
user.mergerfs.func.mkdir: mfs
user.mergerfs.func.mknod: mfs
user.mergerfs.func.open: newest
user.mergerfs.func.readlink: newest
user.mergerfs.func.removexattr: all
user.mergerfs.func.rename: all
user.mergerfs.func.rmdir: all
user.mergerfs.func.setxattr: all
user.mergerfs.func.symlink: mfs
user.mergerfs.func.truncate: all
user.mergerfs.func.unlink: all
user.mergerfs.func.utimens: all

If disabling fasthash doesn't work, then it must be another issue with Isengard.
find quote
eskay993 Offline
Senior Member
Posts: 142
Joined: Nov 2014
Reputation: 1
Post: #20
Thanks jerms415. I matched your setup exactly and stil no luck Sad I've asked in the original thread where this issue was raised about usefasthash in Isengard (here). I'll stick with Helix for now and use my fav skin in a functional albeit slightly broken state Smile
(This post was last modified: 2015-05-22 15:02 by eskay993.)
find quote
trapexit Offline
Junior Member
Posts: 4
Joined: Jun 2015
Reputation: 0
Post: #21
Quote:I also set category.create to "mfs", since I was running into an issue where mergerfs was thinking I was out of disk space when I really wasn't.

Hi. trapexit here. Author of mergerfs. Can you provide more information about this statement? epmfs only considers drives with the path available. mfs considers all drives and will clone the path appropriately.
(This post was last modified: 2015-06-03 22:51 by trapexit.)
find quote
jerms415 Offline
Junior Member
Posts: 8
Joined: Jul 2011
Reputation: 1
Post: #22
(2015-06-03 22:48)trapexit Wrote:  
Quote:I also set category.create to "mfs", since I was running into an issue where mergerfs was thinking I was out of disk space when I really wasn't.

Hi. trapexit here. Author of mergerfs. Can you provide more information about this statement? epmfs only considers drives with the path available. mfs considers all drives and will clone the path appropriately.

Sure, I can explain it more. The epmfs mode works the way that you say it does, but it might be confusing for those of us coming from mhddfs. On mhddfs, if a drive fills up it starts using the next drive, even if the next drive is completely empty (i.e. it has no directories or files). On mergerfs, by default it will not use the empty drive (unless the files you are writing are at the top level of the shared filesystem, I suppose). Instead, it will report that the shared filesystem is out of space, even though there is a drive that is completely empty.

Switching to mfs makes mergerfs behave more similarly to mhddfs, in that it uses the empty drive. However, it doesn't behave quite the same way. Here's a quote from https://romanrm.net/mhddfs describing how mhddfs works:
Quote:But what if you try to add new files somewhere inside that /mnt/virtual? Well, that is quite tricky issue, and I must say the author of mhddfs solved it very well. When you create a new file in the virtual filesystem, mhddfs will look at the free space, which remains on each of the drives. If the first drive has enough free space, the file will be created on that first drive. Otherwise, if that drive is low on space (has less than specified by “mlimit” option of mhddfs, which defaults to 4 GB), the second drive will be used instead. If that drive is low on space too, the third drive will be used. If each drive individually has less than mlimit free space, the drive with the most free space will be chosen for new files.

Personally I prefer the way mhddfs handles it, because I believe it tends to keep files a little more organized on the underlying disks than mfs does. Files that are written at similar points in time tend to be located on the same disk. However, it's not that big of a deal. I'm still extremely happy with mergerfs because it can be configured to correctly report the mtime, which solved this scanning bug in kodi. Now my scans are super fast!

Thanks for all your hard work on it!
find quote
trapexit Offline
Junior Member
Posts: 4
Joined: Jun 2015
Reputation: 0
Post: #23
Understood.

I explicitly didn't want to do the "drive A is full now as I write... let me move it to drive B and continue" behavior due to the possible failure conditions in doing so. I'm not opposed to exploring it again but it does complicate things.

An alternative which would give fairly close behavior is one which was recently requested by another user. The behavior would be in effect... least free space (vs most free space). The intent being to fill a drive before moving to the next. This would probably need a buffer size similar to mhddfs so when one drive is "full" we select the other. This could still result in errors as a result of trying to write a 5GB file to a drive with only 4GB of buffer but would largely behave as expected. Existing path least free space is probably a good permutation of the same.

If you have any feature requests or catch any bugs please submit them to the issue tracker at github.

Interesting about the mtime issue. I hadn't even considered that situation. I knew splitting it up so that each function could have a separate policy would be useful for someone Smile
find quote
trapexit Offline
Junior Member
Posts: 4
Joined: Jun 2015
Reputation: 0
Post: #24
BTW... If you look on github you can download deb packages for x86 64bit. No need to compile it from source if you don't want to.
(This post was last modified: 2015-06-04 03:39 by trapexit.)
find quote
trapexit Offline
Junior Member
Posts: 4
Joined: Jun 2015
Reputation: 0
Post: #25
I've added a feature to mergerfs which is pretty much the same as mhddfs in regard to creation policy which will pick the drive with the least space or just the first drive with at least X bytes.
find quote
Wintersdark Offline
Senior Member
Posts: 103
Joined: Apr 2012
Reputation: 2
Post: #26
Trying mergerfs now, using category.search=newest.

I switched to aufs some time ago, and while it worked very well I found I was having a lot of headaches with whiteout files everywhere and the resulting persistent empty directories on the source drives. Pool access was great, but there was occassional really bizarre behavior that was incredibly difficult to diagnose.

So, I'm going to give MergerFS a try - the upside is switching pooling solutions is inherently seamless to your installed applications, just mount the new pool where the old one was and everything should Just Work.

The downside, of course, is this exposes all those empty folders and .wh files scattered *everywhere*. If it works well, though, I'll just purge 'em all.


Edit: Erf, hope this works with Isengard now. Well, I'll find out shortly Smile
(This post was last modified: 2015-08-14 23:24 by Wintersdark.)
find quote
Post Reply