Beta - Database Pre-Wash Scrub - remove old junk links and privacy issues

  Thread Rating:
  • 5 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
orby Offline
Senior Member
Posts: 163
Joined: Nov 2012
Reputation: 8
Post: #31
will try and get that done tonight, do you want me to upgrade to the newest version also?
find quote
black_eagle Offline
Posting Freak
Posts: 1,815
Joined: Sep 2013
Reputation: 79
Location: East Yorkshire, UK
Post: #32
(2016-05-03 10:33)orby Wrote:  will try and get that done tonight, do you want me to upgrade to the newest version also?

If you don't mind yes please.

I ran your sources.xml through my checker and the produced SQL statement looks like it should have correctly excluded your paths, but my checker is based on the updated add-ons code, not the previous version so the issue could lie there. I'll try and test against the older code and see if it shows anything different. Might give me a clue as to what's happened before I see the debug log.

Thanks for testing this by the way. It's much appreciated.

Learning Linux the hard way !!
find quote
BatterPudding Offline
Posting Freak
Posts: 893
Joined: Jan 2015
Reputation: 52
Post: #33
(2016-05-02 23:39)MacGyver Wrote:  I just tried 0.5.1 and it worked great, and the built-in kodi clean is now 10x faster. I haven't noticed any weird side effects, so far it looks good here.

Thanks guys. The built-in kodi speed up alone was worth the first restore.

I'll do a more thorough look when I get off work tomorrow, but it seems to have worked perfect this time.

Thanks again guys. Smile
Yay!! Thanks for laying your database down on the line for us. Very useful, and even better to see we have had the planned effect this time. When I started looking into this issue I was surprised at the amount of hidden mess in my database. I know it was hassling other addons too as they would often spit errors into my debug logs caused by some of the non-existent paths.

(2016-05-03 10:33)orby Wrote:  will try and get that done tonight, do you want me to upgrade to the newest version also?
Newest versions are always a good idea. The puzzle with your sources is they are just plain normal. And we tested with normal plenty of times. I have a number of C:\ paths in my test script, which all behaved.

Are those real hard drives in the PC? Or are they mapped network paths?
find quote
orby Offline
Senior Member
Posts: 163
Joined: Nov 2012
Reputation: 8
Post: #34
mine are mapped drives to an NFS server that replicate on all 3 machines running Kodi
find quote
PatK Offline
Team-Kodi Member
Posts: 8,500
Joined: Mar 2010
Reputation: 207
Location: Canada
Post: #35
Just ran it on my mess, and it worked well, video clean-up is super fast, might have a gained a bit of speed. Great stuff guys.

windows fresh start -> delete C:\Users\User Name\AppData\Roaming\Kodi First time user (wiki) Log file (wiki) Video management (wiki) Artwork (wiki) Supported hardware (wiki) Releases (wiki) Database versions (wiki) Skin features compatibility list (wiki) http://xbmc-skins.com
find quote
black_eagle Offline
Posting Freak
Posts: 1,815
Joined: Sep 2013
Reputation: 79
Location: East Yorkshire, UK
Post: #36
(2016-05-03 17:06)orby Wrote:  mine are mapped drives to an NFS server that replicate on all 3 machines running Kodi

I have tried to replicate your issue by mounting some shares on my Windows laptop as mapped drives and running both versions of the add-on. I also ran your sources.xml against my test program to see if there were any issues revealed but unfortunately I have so far been unable to get it to misbehave.

It is possible that your full sources.xml file is malformed somewhere although I would have expected this to cause a script error, rather than a wipe of the database because I haven't written any error handling for that situation - an oversight that I shall rectify.

As I can't replicate the issue, we will need that debug log to progress with it any further.

Learning Linux the hard way !!
find quote
ReplayHarry Offline
Donor
Posts: 113
Joined: Feb 2014
Reputation: 0
Location: California, USA
Post: #37
(2016-05-01 23:23)BatterPudding Wrote:  Thanks ReplayHarry for letting us know. Should be no problems with Gotham as the concept is the same for older KODI\XBMC versions.

If you know how to look into your database, the main difference to look for are in the Paths and Files tables. My "production" machine was full of so much old addon and old stream crud that it caused errors in a normal Clean Library. Things are so much nicer now.


We're expecting a few common addons we need to look for and build some exception rules for. So any suggestions, we are here to listen.

Finally got a chance today to try it on my Windows 7 pro Gotham machine. This is my primary HTPC in the living room.
No luck. I installed the repository with a fresh download without issue, but when I try to install the addon, it just says "Incompatible" and will not even try to install.
find quote
black_eagle Offline
Posting Freak
Posts: 1,815
Joined: Sep 2013
Reputation: 79
Location: East Yorkshire, UK
Post: #38
(2016-05-03 21:32)ReplayHarry Wrote:  Finally got a chance today to try it on my Windows 7 pro Gotham machine. This is my primary HTPC in the living room.
No luck. I installed the repository with a fresh download without issue, but when I try to install the addon, it just says "Incompatible" and will not even try to install.

Yeah, the python version dependency for the add-on is set to "2.19.0" which I believe is Helix 14.x. I don't think that there is anything the add-on uses in it's code that hasn't been available since Frodo 12.x so I can't see it not running on Gotham, although I don't have any means to test this.

I can create a zipped version for you with a lower dependency that should install on V13 if you would like me to ?

Learning Linux the hard way !!
find quote
BatterPudding Offline
Posting Freak
Posts: 893
Joined: Jan 2015
Reputation: 52
Post: #39
(2016-05-03 17:06)orby Wrote:  mine are mapped drives to an NFS server that replicate on all 3 machines running Kodi
Have they always been mapped drives? I wonder what is actually in that database for the files. Could be a confusion of paths. If the database has the full ntfs://server/blah/path and the sources.xml only show mapped drive letters then this will be the issue.

Our algorithm doesn't go looking for files. It is about matching paths. It rattles through your FILES table in your database, checking each file and confirming it is in a "keep it" path. When we get to see that debug log we can see if it looked for the right things.

If you turn on debugging in KODI and in our addon, it is possible to run our addon up to the point you see the SQL statement and then say NO there. That will cleanly quit the addon without making any changes to your database. This would be enough to give us some useful debugging without you needing to restore your database a second time.
find quote
BatterPudding Offline
Posting Freak
Posts: 893
Joined: Jan 2015
Reputation: 52
Post: #40
(2016-05-03 19:07)PatK Wrote:  Just ran it on my mess, and it worked well, video clean-up is super fast, might have a gained a bit of speed. Great stuff guys.
I have to admit, yours was one of the KODI installs I was interested in... I get a feeling you have been around here for a while and no doubt had a right old mess in that database. Seeing it clean through without hiccups.... nice.


I think we also need to point out that we are all saving you wasted processor cycles now. Saving you from having KODI stall on confused searches. And plain reducing your electricity bills as KODI now needs to do less work. Saving you raw CASH that you can spend on extra BEER whilst saving the PLANET with your reduced power usage. [Image: Beer%20Smiley.gif]
find quote
PatK Offline
Team-Kodi Member
Posts: 8,500
Joined: Mar 2010
Reputation: 207
Location: Canada
Post: #41
dup.

windows fresh start -> delete C:\Users\User Name\AppData\Roaming\Kodi First time user (wiki) Log file (wiki) Video management (wiki) Artwork (wiki) Supported hardware (wiki) Releases (wiki) Database versions (wiki) Skin features compatibility list (wiki) http://xbmc-skins.com
(This post was last modified: 2016-05-03 22:58 by PatK.)
find quote
PatK Offline
Team-Kodi Member
Posts: 8,500
Joined: Mar 2010
Reputation: 207
Location: Canada
Post: #42
(2016-05-03 21:57)black_eagle Wrote:  
(2016-05-03 21:32)ReplayHarry Wrote:  Finally got a chance today to try it on my Windows 7 pro Gotham machine. This is my primary HTPC in the living room.
No luck. I installed the repository with a fresh download without issue, but when I try to install the addon, it just says "Incompatible" and will not even try to install.

Yeah, the python version dependency for the add-on is set to "2.19.0" which I believe is Helix 14.x. I don't think that there is anything the add-on uses in it's code that hasn't been available since Frodo 12.x so I can't see it not running on Gotham, although I don't have any means to test this.

I can create a zipped version for you with a lower dependency that should install on V13 if you would like me to ?
I've got a 13.2 build which balks, with dependency not met, would like access to a 'test' versions, the clean up take on the order of 2 hours! I'm quite impressed with the job it did on my 16.1 Jarvis build, someone should have done this sooner.

windows fresh start -> delete C:\Users\User Name\AppData\Roaming\Kodi First time user (wiki) Log file (wiki) Video management (wiki) Artwork (wiki) Supported hardware (wiki) Releases (wiki) Database versions (wiki) Skin features compatibility list (wiki) http://xbmc-skins.com
find quote
ReplayHarry Offline
Donor
Posts: 113
Joined: Feb 2014
Reputation: 0
Location: California, USA
Post: #43
(2016-05-03 21:57)black_eagle Wrote:  
(2016-05-03 21:32)ReplayHarry Wrote:  Finally got a chance today to try it on my Windows 7 pro Gotham machine. This is my primary HTPC in the living room.
No luck. I installed the repository with a fresh download without issue, but when I try to install the addon, it just says "Incompatible" and will not even try to install.

Yeah, the python version dependency for the add-on is set to "2.19.0" which I believe is Helix 14.x. I don't think that there is anything the add-on uses in it's code that hasn't been available since Frodo 12.x so I can't see it not running on Gotham, although I don't have any means to test this.

I can create a zipped version for you with a lower dependency that should install on V13 if you would like me to ?

Thanks, I would appreciate a chance to try a zipped version which would work on Gotham.
find quote
black_eagle Offline
Posting Freak
Posts: 1,815
Joined: Sep 2013
Reputation: 79
Location: East Yorkshire, UK
Post: #44
Link for a gotham installable zip version.

At least, I think it should install, but I'm unable to test it for you guys.

Learning Linux the hard way !!
(This post was last modified: 2016-05-04 11:01 by black_eagle.)
find quote
Dalton63841 Offline
Senior Member
Posts: 123
Joined: May 2014
Reputation: 4
Post: #45
The first time I ran it, I disabled the "Prompt before deleting" and debugging, and received this error, which appears fairly cut and dry:

Code:
-->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.NameError'>
                                            Error Contents: name 'i' is not defined
                                            Traceback (most recent call last):
                                              File "/storage/.kodi/addons/script.database.cleaner/default.py", line 391, in <module>
                                                if i:
                                            NameError: name 'i' is not defined
                                            -->End of Python script error report<--

Once I turned re-enabled those 2 settings, it worked perfectly.

EDIT: After a read through of the code, my admittedly elementary understanding of python leads me to believe that the entirety of the actual cleanup code is currently wrapped inside an if statement that requires debugging to be set to true. Is that correct? Makes sense for a beta, just curious.
(This post was last modified: 2016-05-04 12:13 by Dalton63841.)
find quote
Post Reply