Login at Kodi Home

pr0xZen · (This post was last modified: 2016-05-03, 21:55 by pr0xZen.)

My library scan time has been steadily increasing, and library cleanup takes forever. I've been going through the logs and the forum, and it seems to me that (partly) the cause of this is non-existant SMB paths.

log file from library scan - bare with it, its huge.

I had to abort startup full library scan and just do TV series during log, or else xbmclogs.com would throw a fit and sieze on me.

Log file from Library Cleanup - cut after cleanup complete due to size.

Looking from line 1794 and out (during scan), it starts throwing up a sh*t-ton of "does not exist - skipping scan" entries for folders on SMB mounts. These are not errors, these folders have been deleted - but Library Cleanup doesn't remove these.

The same goes for non-existant UPNP file and external (internet) stream locations. These have been streams either played back "normally", cast to Kodi using Kodi as an UPNP renderer or links parsed to Kodi plugins for playback. I have no idea why these have been added to the database in the first place - they are not and never were, added to "the library" or Sources.

My first impression here is that Kodi is unwilling to remove/"clean up" the (actually) non-existant directories/paths, as they are external and its "safety feature" in case an SMB or other source is unreachable (doesn't explain the old plugin entries though). If that is so, is there any way to override this? And if my thoughs are wrong - any good ideas as how to clean up these entries?

Yeah, I know, old Kodi. I'll get on that soon - but the issue still remains (and accumulates).

-

EDIT: This has now progressed to add-on development, thread here: http://forum.kodi.tv/showthread.php?tid=272112

**black_eagle** · 2016-04-12, 16:13

Have you tried removing the source that points to the non-existent directories ? That should give you the option of removing everything in that path from the library. If it removes stuff you still have, you will have to re-add the source and scan it back again, but if you have local nfo files, that won't take long.

pr0xZen · (This post was last modified: 2016-04-12, 16:52 by pr0xZen.)

If you look at the logs, especially the one I just added (OP edited) when performing a library cleanup - you'll see thousands of entries from external locations (e.g. http://10.0.0.78 ). This has been an DHCP allocated IP to a visitor, whom has cast a video to Kodi for it to render. Never added to library, never run a library scan during the casting playback. Looking at the cleanup log - it starts at line 1709. The "first" 1000(!) lines is every youtube video (link) ever cast to Kodi for playback using Yatse or Kore (parsing link to Kodi's youtube plugin), from different phones and tablets. Now apparently there is no timeout for this so its done in a few seconds, but its rather troublesome that they are there. Looking closer, it seems it does this with every playback every plugin has ever played, not just youtube links.

Next, from line 2756 and on, it goes on for 25 minutes with trying to access non-existant locations and files (with 5 second timeout for LAN locations) and other previously played stream through plugins, without "cleaning out" any of them. So I'm reluctant to remove and re-add video sources (yet), as it is very clear that this issue definitely does not just apply to my added video sources. The majority of these entries, even the non-plugin related ones, have never been an added source.

BatterPudding · 2016-04-12, 22:05

/subscribes as I am also interested in this. A database scrubber would be handy. Something that can do a validation check NOW. What is visible NOW is kept, anything else is trimmed and deleted.

I have whole trees abandoned in there when I moved a TV series... yet the old paths are still in the database. Weird bits of junk that will never return.

Like pr0xZen I had assumed videos watched once via a stream or File Mode were not added... and now I see KODI has a record of EVERYTHING I have ever watched.

I've opened the database to read it and am surprised to see a record of every path I have ever watched a video from!! Every video?!?! A little worrying... I wasn't planning on keeping a full record of my pr0n habits.

Ah... yeah... there they all are in the PATH and FILES table. (And I assume elsewhere)

Along with paths to every stream... streams that will never be named the same again.

This is bonkers. So watching a video ONCE with KODI means it gets stored in the database never to be removed? That could be a very serious issue for some people.

I don't really want to delete the whole database and rebuild as I know a few videos I have changed titles, images, etc. A puzzle... we need a tool or some rules of how to delete entries from the database.

Or even better - a switch in KODI that tells it NOT to store one-off videos.

**black_eagle** · 2016-04-12, 22:26

Fully agree - I was also surprised by some of the stuff in my database that dates back over three years !!

I noticed that the 'date added' field always appears to be 'null' so that looks like it would be a starting point to build a query from. Unfortunately, at this time I don't know how to construct the rest of the query to join everything together but if I delete something out of my database with debugging turned on, I should be able to see the query Kodi uses in there.

* black_eagle prepares to potentially destroy his video database !!

pr0xZen · (This post was last modified: 2016-04-12, 23:21 by pr0xZen.)

(2016-04-12, 22:26)black_eagle Wrote: * black_eagle prepares to potentially destroy his video database !!

For the cause ! \o/

(2016-04-12, 22:05)BatterPudding Wrote: [...]
Ah... yeah... there they all are in the PATH and FILES table. (And I assume elsewhere)

This is one of the reasons I revisited this - as I took this to the forums once before (unfruitful), at a time I knew much less of what to look for. I seem to remember, when searching the DB for keywords unique to such a non-existant entry, I found it in more than just PATH and FILES. Being several thousands of entries, and with limited competance in regards to DBs, I decided to not go that route at the time.

The plot thickens. I guess the reason I didn't accidentally post my pr0n habits all over this place is that I haven't deleted any of it in years. Good catch !

This stuff really needs to be looked into - IMO its completely unneccessary, and a major privacy, potential security flaw. At the very best, it sincerely hampers Kodi performance.

**PatK** · 2016-04-12, 23:43

Interesting subject

Quote: /subscribes as I am also interested in this.

pr0xZen · (This post was last modified: 2016-04-13, 17:48 by pr0xZen.)

Anyone else seeing this when logging a library cleanup, or did it just happen to be us?

What Kodi version are you running, OS, local or external DB? Maybe we'll get more eyes on this with more logs.

BatterPudding · 2016-04-13, 19:49

(2016-04-13, 17:45)pr0xZen Wrote: Anyone else seeing this when logging a library cleanup, or did it just happen to be us?

(2016-04-12, 22:05)BatterPudding Wrote:
What Kodi version are you running, OS, local or external DB? Maybe we'll get more eyes on this with more logs.

Want more eyes in the thread? Go Tabloid Stylee and change the title to be "Why does KODI keep a record every video and stream I have ever watched?" or "Why is KODI remembering my p0rn viewing habits?" or "How can I get KODI to forget one off videos I watch through File Manager?" Should poke people more to life.

I also expect that this is not unique to Windows. Seems more like feature someone added without fully realising the implications.

@Black Eagle... Looking into my database the dateAdded == NULL in the FILES table rule doesn't work. I have NULL alongside many videos that legitimately are in the Library. But the more I look the weirder it gets. This database is telling me I added files in 1986, 1988 and various years this century... I didn't think I had had KODI running for *that* long. Big Grin

Something has been putting some data in the wrong columns... (I this is a "show aired" date for that file)

The oldest Video database I still have is from 2014 (Videos78.db) which is back in XBMC days I think? Same issue in there. All the streams and "watch once" files are listed in the FILES table. So this really does mean that KODI has a record of everything I watched dating back to install date...

FYI: I am walking through my KODI database with this. http://sqlitebrowser.org/ If anyone else wants to have a look, do make sure you have KODI closed first and BACKUP...

BatterPudding · 2016-04-13, 20:23

Okay.... so does this mean that during a library scan it is diving back down all of those streams to check them as well? Attempting to reopen connections that are long gone?

**black_eagle** · 2016-04-13, 22:11

Hmm, maybe dateAdded == NULL in the FILES table isn't the way to go the, although it seems to look OK to me in my MySQL db.

Given that (in my case) this looks right, then I have over 3000 entries that don't need to be in there !!! The earliest added date I have is 17th June 2009 (which is wrong) but there is stuff in there that was added in 2013 and those files are correct. I don't think that it is a 'show aired' date as otherwise I would have dates in the 1960's & 70's and I haven't.

It's clearly cross-platform because all my Kodi instances are (and always have) run on Linux as does the MySQL server.

pr0xZen · 2016-04-13, 22:14

To me it appears so, according to the logfile. Not sure why they're so quick, but its "checking" every one of them in some manner, just 1000+ in a few seconds (in my instance). Local network ones though, there's a 5 second timeout - causing a swift job taking half an hour.

Is this really happening? It just seems so absurd.

**beeswax** · 2016-04-14, 18:46

Also subscribing to this as I've noticed this before. A few times in the past I have had to nuke my MySQL database as it gets into a state where performing a database cleanup is impossible as it gets stuck at 99% and never completes. I don't have the logs from last time this occurred but I remember very distinctly seeing all sorts of weird crap in the log during the cleanup where Kodi seemed to be trying to connect to iPads and all sorts of streams and devices which had become listed as sources in the database.

BatterPudding · 2016-04-15, 21:13

How do we find the SQL Devs? Do we need to post a link over to a dev forum?

What I would really like to know is two things.

What happens if we deleted something from the FILES database thereby leaving a "gap" in the numbering?

Ditto with the PATHS database?

Now I could manually extract some of these items - especially the streams - as I assume they will not be referenced in the Library side of the database. But will this cause problems if suddenly row 1789 is missing?

BatterPudding · 2016-04-15, 21:26

(2016-04-13, 22:14)pr0xZen Wrote: To me it appears so, according to the logfile. Not sure why they're so quick, but its "checking" every one of them in some manner, just 1000+ in a few seconds (in my instance). Local network ones though, there's a 5 second timeout - causing a swift job taking half an hour.

Is this really happening? It just seems so absurd.

I wonder if this explains the odd "Clean Library" effect I get? When I hit the "Clean Library" button it starts and almost instantly gives up after a few seconds and the progress bar has hardly moved. I press it a second time and now it gets right through a scan (well, the progress bar moves a lot further)