[ReplayHarry]

Here is a pastebin of the whole log. http://pastebin.com/SZX16Dn8

I have to run now, but when I get back will check and if you need the log again with debug turned on, I'll do it.

[ReplayHarry]

Got to leave now, but decided to enable the addons debug and run again before leaving. To my amazement it worked perfectly.
Go figure.
I ran "clean" again (just for fun) and it took about 5 secs. Usually took about 20 minutes.

Thanks much. I have no idea why it ran ok this 2nd time, but thanks again.

[BatterPudding]

@ ORBY Exclusions.... A simple way to tell our addon which paths to *not* remove

Go hunting for the settings file. Assuming you are on a PC you want C:\Users\Username\AppData\Roaming\Kodi\userdata\addon_data\script.database.cleaner\

You will see a settings.xml file in there. You want to make a textfile called excludes.xml and place it along side. In this file you list the paths to protect.

PHP Code:

<excludes>
     <exclude>nfs://192.168.1.7/Movies/TVShows/</exclude>
     <exclude>nfs://192.168.1.7/Movies/Kids Films/</exclude>
     <exclude>nfs://192.168.1.7/Documentaries/</exclude>
     <exclude>nfs://192.168.1.7/Cartoons/</exclude>
 </excludes> 


Or just this would be enough to protect BOTH the TV Shows and Kids Films branches but drops the Documentaries and Cartoons:

PHP Code:

<excludes>
     <exclude>nfs://192.168.1.7/Movies/</exclude>
 </excludes> 


[BatterPudding]

Got to leave now, but decided to enable the addons debug and run again before leaving. To my amazement it worked perfectly.
Go figure.

Shh... but I think you'll find that our 0.5.1 release only worked when debugging was on... a few too many debug statements had been removed leaving a test wrapped round an important bit of code as noticed by Dalton63841 above . That has been fixed now. I am also about to do a code read through while I drink this cup of coffee.

I ran "clean" again (just for fun) and it took about 5 secs. Usually took about 20 minutes.

Thanks much. I have no idea why it ran ok this 2nd time, but thanks again.

Hehe.... YAY!! Quick Cleaning again...

If you had ever looked into your logs with debugging on before we wrote this addon you would have seen dozens and dozens of errors being spat out where that built in cleaner was running. There must have been so many different crashes and stalls being handled due to the crud in the database. Now with our lovely pre-wash scrub KODI Clean whizzes along again.

[ReplayHarry]

I'm back from shopping. Yes, every time I looked at the debugging log (for other reasons) I noticed all the crud, but could never figure out how to get rid of it. Thanks so much again for this addon.
I have a timer event on my PC which sends a "library clean" command to Kodi every night. It actually sends it to all 3 of my XBMC/Kodi boxes separated by 30 mins. They all have the same sources (my main Windows 7 HTPC). I was doing this to try to keep it clean, but obviously never really worked. I guess I will leave that timer active, but run your addon once a week to really clean it up.
Hopefully the Kodi developers will see that a change is needed to keep this from happening. As I was describing what was going on to my wife (while driving to the shopping center this morning), often programmers put things in their code to help them diagnosis a problem. Then once they fix it they may forget to remove it. That may very well be what happened here. I am retired now, but was in the IT business for over 35 years with a very large company and thousands of IT professionals. So I have seen my share of stuff over the years.

Again, many thanks for your efforts and very helpful addon. I appreciate it.

[black_eagle]

Glad it works for you

For reference, if you leave 'prompt before actually deleting anything' enabled then your version should run with no issues, but that would prevent you running it on a timer once per week.

As PatK reported his gotham installation worked fine, I think maybe the easiest solution is to push a fresh update out that is compatible from Gotham onwards and ensures that both you and PatK have the latest version.

I will speak to BatterPudding regarding this.

[BatterPudding]

@ReplayHarry - you only want to run our script on ONE of your KODI boxes. As the database is common to all, then the database only needs cleaning the once.

Do any of those PCs have a "correct" list of sources? That is the PC I'd pick to run our addon from. Otherwise knock up an exceptions.xml for a chosen KODI box and let the script run on there.



As to your theories where the mess has come from. We have done a fair bit of research into this, and can see various reasons causing it. Much of it is going to not be from debugging left behind in the core KODI code. It is more about functions written with a wider effect than expected. Developers come and go, but don't always document what or why they do things. Like the function that stores a list of OSD settings for EVERY video EVER played, even if you didn't change any of those settings from the default. Now, IMHO, that function in the core KODI is plain wrong. By storing all the pointless default values in there - even when they have not been changed from default - means it actually makes the table less useful. Now if the user has 500 films in his collection, and then tries to change a default value... that change will never get propogated to the original 500 films as they have their settings already "fixed" in the database.

Similar issue with the "watched" flag. As each Internet stream was viewed in KODI, it was marked as "watched"... which is clearly an odd thing to mark on a live stream that you will not watch again.

The videos database has this split personality between being somewhere to keep full details of your library of films. AND trying to keep track of settings for every type of video ever played. The main theme of our addon has been to thin your Videos database back to being a database of your library files.


I can also see other bugs and errors in the database... but this may only be in my one. Not sure yet. Once we have settled down the main pre-wash scrubber I see other areas that could be due a repair \ clean. I'm not confusing the discussion yet... those thoughts are for v2.

[ReplayHarry]

I plan to install your addon on my 2 bedrooms Kodis (Rasp Pi 2 and Zbox, both OpenElec), which are both Jarvis. I don't use a common database for all, they each have their own standard Kodi databases but share their sources. I have thought about making a common shareable database, but just haven't got around to it. Some of us retired guys are busy. After-all, I have to take my wife shopping once in awhile. Ha Ha!

So for now, I will just run your addon manually about once a week. If you are able to make a Gotham version that does not require any user feedback, then I will set that up to automatically run on a weekly schedule in the future.

Again, thanks very much.

[black_eagle]

If you are able to make a Gotham version that does not require any user feedback, then I will set that up to automatically run on a weekly schedule in the future.

This is definitely in the pipeline I think

[orby]

had to play husband last night will hopefully get to mess around tonight to figure out what was going on

just had a quick look into the new database and under movies the entries are all similar to below, am assuming this is correct for the location to be ignored now


X:\Guardians of the Galaxy (2014)\\ 1 10 null

[BatterPudding]

had to play husband last night will hopefully get to mess around tonight to figure out what was going on

just had a quick look into the new database and under movies the entries are all similar to below, am assuming this is correct for the location to be ignored now


X:\Guardians of the Galaxy (2014)\\ 1 10 null

I am not sure what you are trying to show us there? What is that bit of text you have pasted supposed to be?

More importantly - what is X:\ ? Where does this point?

[orby]

sorry done quickly at work

X:\ is one of my mapped drives for movies which was being cleaned out when running the addon, what i pasted is a portion of an entry from the movies table in my database so i am assuming that is the location kodi now uses for that movie so should be excluded now when running the cleaner

if not i can delve deeper into the database to find out where it stores sources and file locations

[BatterPudding]

I'll try that another way - where does X: map to? What is the *REAL* path? (I'll assume nfs://server/share/ for the description below)


If you are comfy looking into your database, what you want to do is look at two tables really. The PATH table will show you the paths that KODI really uses to access the files. It is these paths that we need to match in our excludes.xml list.

It is also the FILES table you look into if you want the file names of the videos. strFilename holds the filename, and idPath will be the initial parent folder.

Our searches are working with the files and folders they are living in. Checking the consistency of the files from a location point of view. We don't touch the MOVIES table that you are looking in. That table gets a good clean out by the standard KODI Clean Library function. Our pre-cleaning allows KODI's own Clean Library to clear out that movies table a lot better.


Personally I use the default SQLite database on my KODI setup. I then use the DB Browser for SQLite (sqlitebrowser.org) to poke around in there. This lets me open up the PATH table and then click on the column header of the strPath field. Now that the column is alpha sorted it makes it easier to have a good old nose at what is in there. This is when the scrappy paths start to stand out. And you soon get to see what your real paths are.

In your case you may have loads of files starting X: or your may have lots of files starting nfs://server/share/. What I am hoping you don't see in that table is both. Though even then we could handle it by just making sure that both X:/ and nfs://server/share/ are listed in your excludes.xml

If you decide you have folders list in the PATH table in X:/ and in nfs://server/share/ then we need to make sure these are both in the excludes.xml file.

Give me some *real* path examples and I'll start giving you a *real* excludes.xml file example in return.

[orby]

don't know how i missed the path's table but had a poke around in there and it looked good with all tv shows using the NFS path as set in sources and all movies linked to the corresponding drive letter and away she ran, thank you all for your time with me

[BatterPudding]

don't know how i missed the path's table but had a poke around in there and it looked good with all tv shows using the NFS path as set in sources and all movies linked to the corresponding drive letter and away she ran, thank you all for your time with me

Excellent... and I'm happy to spend the time to explain this stuff as you guys are helping to make this more bullet proof for other people. In turn it then helps us improve the instructions.