Kodi Community Forum

Full Version: [RELEASE] Library watchdog
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hi,

I don't remember if I already reported the issue but I just tried the addon again on frodo and still had the same issue.

I saw, some other user reported the issue too.

The issue I have is, that I get a really high cpu usage when I had enabled the addon. Process Explorer shows that its more a kernel cpu usage instead of a xbmc process usage

My setup.

I have all stuff on another computer (both windows 7). I use CIFS and had mapped all the volumes on the client where xbmc is running.
I never shutdown my client. I only reboot it when necessary. I only use standby to ram. I use Eventghost to exit xbmc before enter standby and start it again after waking up the client.
I think its only after waking up the client and start xbmc that the addon or the system went crazy.

Last time i switched back to the other addon xbmc auto update library but that gave me some problems on frodo after the migration, so I retried this addon.

I don't know if this is fixable but I would like to try to give you all the info I can get.

Do you or anyone watching this thread had seen this issue too?
I don't saw any entries regarding this issue in the debug log of xbmc.
which version should I be rolling back to? I was on 7 so rolled back to 6.4 but its done it again and cant seem to roll back to 6.3
Ok got back to 6.3 now, should this one be ok?
Lets say I have two xbmc clients, both with watchdog installed, sharing a MySQL library and sources on a nas. If both the xbmc clients are open at the same time and a movie is added to the source on the nas would that cause a problem? I'm guessing they would both start a scan, but would the movie be added to the library twice? I haven't been able to test this setup but I was wondering if you had an idea of how it would be handled?
About the high cpu load issue.
I still had polling enabled with infinite depth. And I have some folders with a lot of files and folders.

Now that we just can use Auto in Frodo for mounted CIFS shares, I set Auto with 2 subfolders (tv with season folders) and all is fine for now.
As far as I understand it does not use polling for my network shares but some Windows Api:

2013-02-09 08:02:39:0954 887 08:02:39 T:6428 DEBUG: service.watchdog: watching <N:\vids\tv\> using <class 'watchdog.observers.read_directory_changes.WindowsApiObserver'>

Thanks for this addon.
Thanks for this great plugin. I'm having some troubles with it auto refreshing my library though. I have a NAS where my media and mysql database are hosted. My XBMC client is a Windows 7 box and the media source is added using smb:// format (not a mounted drive). I'm using Frodo and the latest watchdog with default settings. I use Sickbeard to download torrents and it places the finished files in my sources directory. I did four different trials to try to pinpoint the problem. I downloaded the first 4 episodes of American Idol Season 12. I closed xbmc between each one to grab the full debug logs. The first episode was auto-refreshed by watchdog and the following three were not. I'm thinking the difference is that for the first one, Sickbeard created a season directory that didn't exist and maybe that is what triggered watchdog. For each episode it moved the video file and a nfo file and a thumbnail icon inside the source directory being watched.

First episode Only one that triggered a watchdog library refresh.

Second Episode

Third Episode

Fourth Episode
I seem to be having an issue with the latest version of Watchdog. Whenever I delete a file or a movie it automatically cleans my database even though this option is set to off in the configuration. This is not good because if I do for some reason lose my network connection watchdog will delete every thing in the database.

The other issue I am having is when I add something new, it does not automatically scan.

Are these known issues? I had no issues with the previous version. I am currently using OpenELEC 3 (Beta 5).

Any suggestions?

Thanks
Shedrock
(2013-02-10, 03:26)xombiemp Wrote: [ -> ]Thanks for this great plugin. I'm having some troubles with it auto refreshing my library though. I have a NAS where my media and mysql database are hosted. My XBMC client is a Windows 7 box and the media source is added using smb:// format (not a mounted drive). I'm using Frodo and the latest watchdog with default settings. I use Sickbeard to download torrents and it places the finished files in my sources directory. I did four different trials to try to pinpoint the problem. I downloaded the first 4 episodes of American Idol Season 12. I closed xbmc between each one to grab the full debug logs. The first episode was auto-refreshed by watchdog and the following three were not. I'm thinking the difference is that for the first one, Sickbeard created a season directory that didn't exist and maybe that is what triggered watchdog. For each episode it moved the video file and a nfo file and a thumbnail icon inside the source directory being watched.

First episode Only one that triggered a watchdog library refresh.

Second Episode

Third Episode

Fourth Episode

They're all detected by the addon and the scan is triggered. The problem is that xbmc scans but don't see that anything has changed. Probably something in between xbmc and your server that don't properly invalidate cache. Try setting a longer delay in the addon setting. That's the delay the addon will wait after the new file was detected to telling xbmc to scan.
(2013-02-09, 09:10)brotbuexe Wrote: [ -> ]About the high cpu load issue.
I still had polling enabled with infinite depth. And I have some folders with a lot of files and folders.

Now that we just can use Auto in Frodo for mounted CIFS shares, I set Auto with 2 subfolders (tv with season folders) and all is fine for now.
As far as I understand it does not use polling for my network shares but some Windows Api:

2013-02-09 08:02:39:0954 887 08:02:39 T:6428 DEBUG: service.watchdog: watching <N:\vids\tv\> using <class 'watchdog.observers.read_directory_changes.WindowsApiObserver'>

Thanks for this addon.

Auto only works for local drives and network locations set in xbmc. If you have network shares mounted by OS then you need to set method to Polling. There's no way to know where the drive for "N:\vids\tv" is physical located. (i''ll update the faq)
(2013-02-10, 03:49)shedrock Wrote: [ -> ]I seem to be having an issue with the latest version of Watchdog. Whenever I delete a file or a movie it automatically cleans my database even though this option is set to off in the configuration. This is not good because if I do for some reason lose my network connection watchdog will delete every thing in the database.

The other issue I am having is when I add something new, it does not automatically scan.

Are these known issues? I had no issues with the previous version. I am currently using OpenELEC 3 (Beta 5).

Any suggestions?

Thanks
Shedrock

Fix for the cleaning is underway. In the meantime, rollback to v0.6.3 or manually download v0.7.1 from here. As for the other issue, see two posts above or the first post
Fix is up now.
(2013-02-10, 17:22)takoi Wrote: [ -> ]
(2013-02-09, 09:10)brotbuexe Wrote: [ -> ]About the high cpu load issue.
I still had polling enabled with infinite depth. And I have some folders with a lot of files and folders.

Now that we just can use Auto in Frodo for mounted CIFS shares, I set Auto with 2 subfolders (tv with season folders) and all is fine for now.
As far as I understand it does not use polling for my network shares but some Windows Api:

2013-02-09 08:02:39:0954 887 08:02:39 T:6428 DEBUG: service.watchdog: watching <N:\vids\tv\> using <class 'watchdog.observers.read_directory_changes.WindowsApiObserver'>

Thanks for this addon.

Auto only works for local drives and network locations set in xbmc. If you have network shares mounted by OS then you need to set method to Polling. There's no way to know where the drive for "N:\vids\tv" is physical located. (i''ll update the faq)

I recently experienced this high load CPU issue. By viewing my log I found out xbmc was scanning my music library all the time in a sort of infinite loop. I figured out it was the watchdog plugin. I don't know where that issue came from (i think it was recent, maybe from Frodo RC3 ?), but my shared folders are identified by xbmc as smb:// shared folder, on another computer, so watchdog should be aware of this I suppose.
For now, I simply disabled watchdog in order to be able to watch seamless HD movies. I'll check on some other time, but in the meantime, I just wanted to give some feedback about this.

And, while I'm at it, great job on that plugin, because appart from that one time, it always gave me plain satisfaction Wink

(2013-02-10, 17:09)takoi Wrote: [ -> ]
(2013-02-10, 03:26)xombiemp Wrote: [ -> ]Thanks for this great plugin. I'm having some troubles with it auto refreshing my library though.

They're all detected by the addon and the scan is triggered. The problem is that xbmc scans but don't see that anything has changed. Probably something in between xbmc and your server that don't properly invalidate cache. Try setting a longer delay in the addon setting. That's the delay the addon will wait after the new file was detected to telling xbmc to scan.


I have stripped out as many variables as possible. I did a fresh default install of XBMC (no mysql, path redirection, symlinking or anything) and added a smb source. The full path of the tv show I am testing is:
data
----Media
--------TV Shows
------------Modern Family
----------------Season 02
--------------------mkv, nfo, tbn files

The source I added in xbmc was smb://192.168.120.178/Media/TV Shows/
I installed the watchdog addon and left the default settings, enabled debug logging, and added a keymap of g to XBMC.UpdateLibrary(video).

So for the first test I just moved episode 19 mkv, nfo, and tbn to the Season 02 directory. I saw a very quick flash of the updating notification in the top right but the episode was not added to the library. I then pressed g to initiate a video library update and the updating notification appeared and the episode was added to the library. See the full log of this test log

For the second test, I modified the delay in watchdog, as suggested, to 10. I did the same test as above with episode 20 and had the exact same results. See the log for that here: log

You can compare the 2 different library updates in each of these two files and see that they look a bit different. Any idea what is going on?
(2013-02-10, 17:22)takoi Wrote: [ -> ]Auto only works for local drives and network locations set in xbmc. If you have network shares mounted by OS then you need to set method to Polling. There's no way to know where the drive for "N:\vids\tv" is physical located. (i''ll update the faq)

Hm, but it does work fine for me with set to auto.
+ 978 14:46:06 T:5092 DEBUG: service.watchdog: watching <P:\vids\tv\> using <class 'watchdog.observers.read_directory_changes.WindowsApiObserver'>
2013-02-11 20:05:24:0553 + 201820 20:05:24 T:6116 DEBUG: service.watchdog: <deleted> <P:\vids\tv\Sons of Anarchy\Season 05\Sons of Anarchy - 5x01 - Sovereign - 720p WEB-DL.mkv>
2013-02-11 20:05:24:0553 + 201829 20:05:24 T:6116 DEBUG: service.watchdog: <modified> <P:\vids\tv\Sons of Anarchy\Season 05>
2013-02-11 20:05:24:0553 + 201830 20:05:24 T:6116 DEBUG: service.watchdog: <deleted> <P:\vids\tv\Sons of Anarchy\Season 05\Sons of Anarchy - 5x01 - Sovereign - 720p WEB-DL.nfo>
2013-02-11 20:05:24:0553 + 201831 20:05:24 T:6116 DEBUG: service.watchdog: <modified> <P:\vids\tv\Sons of Anarchy\Season 05>
2013-02-11 20:05:24:0553 + 201832 20:05:24 T:6116 DEBUG: service.watchdog: <deleted> <P:\vids\tv\Sons of Anarchy\Season 05\Sons of Anarchy - 5x01 - Sovereign - 720p WEB-DL.tbn>
2013-02-11 20:05:24:0553 + 201833 20:05:24 T:6116 DEBUG: service.watchdog: <modified> <P:\vids\tv\Sons of Anarchy\Season 05>
2013-02-11 20:05:33:0116 + 201922 20:05:32 T:6116 DEBUG: service.watchdog: <created> <P:\vids\tv\Sons of Anarchy\Season 05\Sons of Anarchy - 5x01 - Sovereign - 720p WEB-DL.mkv>
2013-02-11 20:05:33:0116 + 201923 20:05:32 T:6116 DEBUG: service.watchdog: <modified> <P:\vids\tv\Sons of Anarchy\Season 05>
2013-02-11 20:05:33:0116 + 201924 20:05:32 T:6116 DEBUG: service.watchdog: <modified> <P:\vids\tv\Sons of Anarchy\Season 05\Sons of Anarchy - 5x01 - Sovereign - 720p WEB-DL.mkv>
2013-02-11 20:05:33:0116 + 201925 20:05:32 T:6116 DEBUG: service.watchdog: <created> <P:\vids\tv\Sons of Anarchy\Season 05\Sons of Anarchy - 5x01 - Sovereign - 720p WEB-DL.nfo>
2013-02-11 20:05:33:0116 + 201926 20:05:32 T:6116 DEBUG: service.watchdog: <modified> <P:\vids\tv\Sons of Anarchy\Season 05>
2013-02-11 20:05:33:0116 + 201927 20:05:32 T:6116 DEBUG: service.watchdog: <modified> <P:\vids\tv\Sons of Anarchy\Season 05\Sons of Anarchy - 5x01 - Sovereign - 720p WEB-DL.nfo>
2013-02-11 20:05:33:0116 + 201928 20:05:32 T:6116 DEBUG: service.watchdog: <created> <P:\vids\tv\Sons of Anarchy\Season 05\Sons of Anarchy - 5x01 - Sovereign - 720p WEB-DL.tbn>
2013-02-11 20:05:33:0116 + 201929 20:05:32 T:6116 DEBUG: service.watchdog: <modified> <P:\vids\tv\Sons of Anarchy\Season 05>
2013-02-11 20:05:33:0116 + 201930 20:05:32 T:6116 DEBUG: service.watchdog: <modified> <P:\vids\tv\Sons of Anarchy\Season 05\Sons of Anarchy - 5x01 - Sovereign - 720p WEB-DL.tbn>

p: is a mount to \\server\f$, Os on both sides are win7.

But something is still wrong:

+ 201954 20:05:32 T:6116 DEBUG: service.watchdog: <modified> <P:\vids\tv\Sons of Anarchy\Season 05\Sons of Anarchy - 5x01 - Sovereign - 720p WEB-DL.tbn>
+ 202248 20:05:58 T:6532 DEBUG: service.watchdog: scanning N:\vids\tv\ (video)
+ 202273 20:05:59 T:6532 DEBUG: service.watchdog: cleaning video library
+ 202412 20:06:09 T:7344 DEBUG: service.watchdog: <modified> <N:\sab\tv>
+ 202737 20:06:38 T:6532 DEBUG: service.watchdog: scanning P:\vids\tv\ (video)
(delay set to 20)

Looks like it does select the wrong path for scanning?

and n:\sab is excluded from scans.

+ 981 14:46:07 T:5092 DEBUG: service.watchdog: watching <N:\sab\> using <class 'watchdog.observers.read_directory_changes.WindowsApiObserver'>
+ 75024 17:00:35 T:8172 DEBUG: Testing Existence (N:\sab\)

If its excluded (its also set to content none), shouldnt it excluded from watching too?

(version 0.7.1)
Okay, restarted xbmc and tried a little bit and it does select the correct folder for scanning.

And it works with all the shares I have (all on only this one computer).