Kodi Community Forum

Running Kodi 14.1 on my Ubuntu HTPC.
Library is on my server, MySQL.

Logs: http://xbmclogs.com/peiapxwbb#line-1479

Probably contibutory setup:

I'm running 4 storage drives pooled with mhddfs. I've only added the pooled mount, /mnt/pool, as an NFS share.

Kodi is running
The underlying disks (/mnt/disk1 through disk4; /mnt/pool) are all assigned to user "server" as well, with full read/write permissions for everyone.

As an example, ls /mnt/pool/TV/Bubble.Guppies/Season.3 shows (in addition to everything else):

When I do a library update, that episode is not picked up and scanned in, and as can be seen in the log it reports "no change". When I go to the library, that episode is not there.

However - and this is where it gets baffling - I can go to Video>Files>TV(my one TV video source)>Bubble Guppies>Season 3> and see all the scanned in episodes as names, and the unscanned episode as a filename. I can then manually add it by refreshing the info for that episode, after which it appears in the library just fine. So, Kodi can clearly see the file in /mnt/pool/TV/Bubble.Guppies/Season.3.

Now, this episode actually resides on /mnt/disk2, while the remainder of the episodes in that folder are in /mnt/disk1. If I manually copy that episode to /mnt/disk1 alongside the other episodes, it will scan in just fine on a regular update.

My confusion here is that Kodi should not be aware that they are on different drives, as mhddfs is presenting /mnt/pool as a single filesystem, and Kodi CAN see the file and add it manually without issue.

This appears to only be an issue with TV shows spread between multiple folders, as movies are scattered between all four drives as well and scan in on updates just fine, and they're added in the same manner (/mnt/pool/Movies source).

There's no kodi-end multi-path setup - as I said, there's just one TV source added, and it's just a regular source, /mnt/pool/TV.

If all else fails, I can ditch mhddfs, spend a couple days moving files around (to move each TV series onto a single drive) and add all four drives as individual sources which will fix this problem, but at the same time it'll remove the wonderful joy of being able to just throw more drives into my system and be unconcerned about how much space is free on individual drives (leading to constant shuffling and the resulting potential library screwups). Clearly, I'd rather not do that.

Any ideas?

Further information.

It turns out, if I go Video>Files>TV (my TV source)>Scan For New Content, Kodi picks up everything as it should.

A normal "Update Library" from elsewhere in the system, however, does not, and misses everything not on the first drive.


--------------


Also: Removed mhddfs, installed aufs. Generally the same problem persists - whole TV series are simply missing, though they can be added manually as above or by drilling down through Video>Files>TV>ShowName> then selecting individual files and pulling up their information.

Can anyone suggest why these files are visible to Kodi, addable to the library, but not picked up in a regular library scan? I'm really completely baffled.

try with a nightly build and

in advancedsettings.xml

I too use mhddfs but I share it via SMB. I don't think I have seen this problem, although something similar happened last night - for the first time that I can recall.

I do not that some of your files have an exclamation point (!) in the filename. I think that is an illegal character for filenames in some OSes or filesystems. Does that lead anywhere?

I would bet you have some bad NFO's

I have also run into this problem before.

What program creates the NFO? Sickbeard? Sickrage? Is so, make sure you have XBMC 12+ or KODI 12+ checked in the options for metadata.

(2015-03-19, 04:21)FishOil Wrote: [ -> ]I would bet you have some bad NFO's

I have also run into this problem before.

What program creates the NFO? Sickbeard? Sickrage? Is so, make sure you have XBMC 12+ or KODI 12+ checked in the options for metadata.

I hadn't considered this; thanks for suggesting it. It's not the problem in my case, but it's a good troubleshooting step in the future - deleting the existing NFO's just in case. In my case, I could move the file (NFO and all) to the same physical disk as the remainder of the series and Kodi would scan it in on a normal update. That's a wierd situation, though, because to Kodi they're all in the same directory structure - aufs and mhddfs pools present themselves as single, unified file systems.

(2015-03-19, 02:51)nickr Wrote: [ -> ]I too use mhddfs but I share it via SMB. I don't think I have seen this problem, although something similar happened last night - for the first time that I can recall.

I do not that some of your files have an exclamation point (!) in the filename. I think that is an illegal character for filenames in some OSes or filesystems. Does that lead anywhere?

Nah, I've always had !'s in the filenames. Fine on Windows and Linux both, and it's never been an issue before. No correlation between such characters and whether or not they get scanned in.

The only common factor I can tell as to whether or not a newly added episode gets added to the library on a normal update is which physical disk it resides on.

(2015-03-18, 22:23)wsnipex Wrote: [ -> ]try with a nightly build and

Code:

<videolibrary>
    <usefasthash>false</usefasthash>
</videolibrary>

in advancedsettings.xml

Just upgraded to nightly, testing this now.

A regular (and sadly, very slow) library update saw a huge swaft of episodes found and added. This seems to have fixed the problem.

It then appears that the fasthash function is not playing nice with the union filesystems (mhddfs and aufs). It'd be very easy to use these filesystems and not encounter the problem, however, depending on which physical disks held your episodes, and the specific method you use to scan them in (as directed source scans through Video>Files>*source* still scan things correctly).

So! I'll tentatively call this solved, and suggest a look at how the fasthash function works when processing those sorts of file systems... or NFS shares?

And wsnipex, nickr and FishOil: Thanks so much for your time! That was a particularly strange problem, and definitely new to me.

can you test if the m(modify)time of the parent dir gets updated when you create a file?

e.g.:

Sure:


However, that's not all the picture, as that's made entirely in the pool. It stands to reason the folder's mtime gets updated, because the file will certainly be placed within that directory.

The way these pools work, is they create identically named directories on multiple drives, and combine the contents. So:

Disk1/dirA has a mtime of 0200
Disk1/dirA/FileA has a mtime of 0200 as well

There's no dirA on any other drive.

Thus, Pool/dirA has an mtime of 0200.

I add FileB at 0202. Three different things can happen:

1) It gets placed in Disk1/dirA/FileB. This causes dirA's mtime to be updated, and is reflected in Pool/dirA's mtime.
2) The pool makes Disk2/dirA/FileB. Pool/dirA still shows Disk1/dirA's mtime, dispite Disk2/dirA having a different mtime.
3) The pool makes Disk2/dirA/FileB. Pool/dirA shows Disk2/dirA's mtime, as it's more recent? Dunno what logic is employed here.

I will test this further right now.

Ok, I've been unable to ascertain exactly what causes the pool folders' mtimes to update, but apparently only the relevant individual disk's folder mtime is updating; that is, if the pool writes to disk1/dir, then disk1/dir's mtime updates, but not disk2/dir or disk3/dir.

Further, pool/dir may or may not update. Results have so far been inconclusive as to what exactly does and does not trigger mtime modification of the aufs pool.

Note that in normal use, I'm never accessing the drives directly, but rather always writing to and reading from the pool


Further simple test to prove:


I chose this particular folder because it resides only on a mostly full drive ( /mnt/disk1/TV/12.Monkeys ) and I know aufs will prefer to write to /mnt/disk3/ as it's mostly empty. This is shown here:



Regardless, mtime is clearly not a good indicator in this instance.

Thanks for testing and reporting back. Scanning should be back to normal when using 14.2 or 15.x with the following advancedsettings.xml

Tried that yet?

yes he did, and it works, read above.

Update: after a couple days use, and new files being added in a variety of methods, the fasthash disabling has worked reliably. New shows are added correctly with no problems.

Thanks again for your help, everyone!

(2015-03-21, 10:55)mkortstiege Wrote: [ -> ]Thanks for testing and reporting back. Scanning should be back to normal when using 14.2 or 15.x with the following advancedsettings.xml

Code:

<videolibrary>
    <usefasthash>false</usefasthash>
</videolibrary>

Tried that yet?

I was having the same problem, I can't exactly put my finger on it, but it must have been an openelec update. I am on oe 5.0.6 which has kodi 14.2rc1-git e7ba06f. From the earlier talk about nightlies I thought I might need 15 to fix this. Your post made me think that fasthash solution also applied to 14.2, so I tried it and it seemed to work. Thank you.

If you want to take advantage of the fast hash functionality, there's an alternative to mhddfs called mergerfs which can be configured so that it doesn't have this problem.

You need to use one of the following configuration options:
category.search=newest
or
func.getattr=newest

I prefer the category.search=newest way of doing it, since it prevents more functions from using the first found method. First found doesn't really make sense from a correctness standpoint (which is the cause of this bug in the first place), although it will be slightly faster in most cases.

I've been using mergerfs for a couple days now, and everything seems to be working fine.

FYI here's the steps I used to install on ubuntu: