Beta - Database Pre-Wash Scrub - remove old junk links and privacy issues

  Thread Rating:
  • 5 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
orby Offline
Senior Member
Posts: 163
Joined: Nov 2012
Reputation: 8
Post: #46
hopefully this will assist, let me know if you need anything else, ran it again with the latest version and again cleared out all the movies, tv shows unaffected

http://pastebin.com/8cNHXaiM
find quote
black_eagle Offline
Posting Freak
Posts: 1,815
Joined: Sep 2013
Reputation: 79
Location: East Yorkshire, UK
Post: #47
(2016-05-04 14:43)orby Wrote:  hopefully this will assist, let me know if you need anything else, ran it again with the latest version and again cleared out all the movies, tv shows unaffected

http://pastebin.com/8cNHXaiM

Yes, I don't think your sources.xml matches what you actually have in the database. As you are using a MySQL db, this needs to be set up (and probably was originally) with the full network path to the files.

EG, this is correct
Code:
nfs://192.168.1.7/Movies/TVShows/
whereas
Code:
W:\
is a mapped network drive, but likely doesn't correspond to a path in your database which will start (most likely) with smb://

The SQL the add-on is producing is correct for the supplied sources, but it appears that those paths are not in your database. Excluding them therefore deletes your movies because the actual paths in the database likely start with 'smb://' or 'nfs://'. This is borne out by the fact that your TV shows are untouched and the path in your sources.xml is a correct network path.

Learning Linux the hard way !!
find quote
BatterPudding Offline
Posting Freak
Posts: 893
Joined: Jan 2015
Reputation: 52
Post: #48
@orby - what else on that PC is using those mapped drives? Any reason why you can't put the sources back at their nfs:// equivalents? As the TVShows path is clearly working on that PC, I assume everything else about nfs is setup on that PC and would work equally as well. (If not a little better)
find quote
orby Offline
Senior Member
Posts: 163
Joined: Nov 2012
Reputation: 8
Post: #49
i have just nuked the database and sorting it out ready for a fresh backup, then will try again, has been many many changes for my box over the years and is most likely user error and i love to break things so possibly a lazy hack i did which has been clouded by alcohol by now,
find quote
black_eagle Offline
Posting Freak
Posts: 1,815
Joined: Sep 2013
Reputation: 79
Location: East Yorkshire, UK
Post: #50
(2016-05-04 15:34)orby Wrote:  i have just nuked the database and sorting it out ready for a fresh backup, then will try again, has been many many changes for my box over the years and is most likely user error and i love to break things so possibly a lazy hack i did which has been clouded by alcohol by now,

I think you had better remove your current sources.xml as well, otherwise you may still end up with a scenario where the database paths are correct but the sources are not.

Learning Linux the hard way !!
find quote
orby Offline
Senior Member
Posts: 163
Joined: Nov 2012
Reputation: 8
Post: #51
i can't think what in my sources might be wrong though, tv shows are shared by an NFS server using the built in kodi client and the mapped drives are all blu-ray rips which need to be setup as a shared drive for the path to be sent to TMT6 and powerdvd
find quote
orby Offline
Senior Member
Posts: 163
Joined: Nov 2012
Reputation: 8
Post: #52
willl have a good poke around once everything is imported and look at the path's in the database also this go around
find quote
pr0xZen Offline
Fan
Posts: 491
Joined: Feb 2014
Reputation: 20
Post: #53
Finally back at home base after travels and got to try out this thing. AFAIK, looking through the library and logs, no errors here. Also shaved a good 0.5 MB and hunks of thousands of lines off my scan logs. Took my library cleanup from ~30 minutes down to a healthy 90 seconds.

I love this thing! Big Grin

In orbys case here (others might very well benifit) - what are your thougts on possibly enabling a (manual) addendum to the sources.xml that the script fetches (e.g. excludedpaths.xml kind of ordeal)? In my mind, this adds quite a bit of flexibility in terms of various similar user scenarios and path hacks, without neccesarily having to adapt the script itself to a (potentially vast) variety of scenarios. This way, users can manually add the "correct" / resolved paths of their mounted drives, symblinks etc - to this file, and the script adds these paths to the "cleanup exclusions" in sources.xml..

Is this feasable?

If I have helped you or increased your knowledge - please click the plus to the left below to give thanks
find quote
black_eagle Offline
Posting Freak
Posts: 1,815
Joined: Sep 2013
Reputation: 79
Location: East Yorkshire, UK
Post: #54
I think the problem may be that originally the database was set up with network paths. Once these are scanned in, its completely possible for Kodi to run correctly without even having a sources.xml file as all the paths are contained within the database.

EG if your movie is at smb://myserver/mymovies/movie1/movie1.mkv then that entire path is added to the database. When you come to browse your movies, picking movie1 selects the full path 'behind the scenes' so Kodi can play it.

@pr0xZen - there is already the option for a user generated 'excludes.xml' that the script reads if it exists. Details should be in the first post, but if anything is not clear, please don't hesitate to ask.

Glad it worked for you as you were the original instigator of the idea Big Grin

Learning Linux the hard way !!
find quote
pr0xZen Offline
Fan
Posts: 491
Joined: Feb 2014
Reputation: 20
Post: #55
(2016-05-04 16:09)black_eagle Wrote:  @pr0xZen - there is already the option for a user generated 'excludes.xml' that the script reads if it exists. Details should be in the first post, but if anything is not clear, please don't hesitate to ask.
Really, that was mostly my bad for not re-reading TFM before calling tech support Wink

(2016-04-30 20:46)BatterPudding Wrote:  The filename is "excludes.xml" and the contents are as follows :-
Code:
<excludes>
    <exclude>plugin://plugin.video.youtube</exclude>
    <exclude>plugin://plugin.video.iplayerwww</exclude>
</excludes>

Any number of <exclude></exclude> tags can be added and can be as precise or vague as is required by the user eg "plugin://plugin.video" would exclude all video plugin data from being deleted. "http://" would keep all the http info, whereas "http://my_favourite_streaming_site.com" would retain just the info for that site.

So in orbys case, this is where he'd be wanting to add his resolved network paths..(?)
"[...] can be added and can be as precise or vague as is required by the user" - this means in case of paths, it will include any and all subdirs?

If I have helped you or increased your knowledge - please click the plus to the left below to give thanks
find quote
PatK Offline
Team-Kodi Member
Posts: 8,493
Joined: Mar 2010
Reputation: 207
Location: Canada
Post: #56
(2016-05-04 10:24)black_eagle Wrote:  Link for a gotham installable zip version.

At least, I think it should install, but I'm unable to test it for you guys.
Worked like a 'treat', many thank-yous. Gotham 13.2 clean up is under 5 seconds now.

windows fresh start -> delete C:\Users\User Name\AppData\Roaming\Kodi First time user (wiki) Log file (wiki) Video management (wiki) Artwork (wiki) Supported hardware (wiki) Releases (wiki) Database versions (wiki) Skin features compatibility list (wiki) http://xbmc-skins.com
find quote
ReplayHarry Offline
Donor
Posts: 113
Joined: Feb 2014
Reputation: 0
Location: California, USA
Post: #57
(2016-05-04 10:24)black_eagle Wrote:  Link for a gotham installable zip version.

At least, I think it should install, but I'm unable to test it for you guys.

No go. I didn't have debug turned on when I tried to run it. Here is the error I got. I'll turn on debug and try again to get complete info if you need it.

Code:
7:04:58 T:5460   ERROR: EXCEPTION: argument "path" for method "XBMCAddon::xbmcvfs::exists" must be unicode or str
07:04:58 T:5460   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.TypeError'>
                                            Error Contents: argument "path" for method "XBMCAddon::xbmcvfs::exists" must be unicode or str
                                            Traceback (most recent call last):
                                              File "C:\Users\Harry Kious\AppData\Roaming\XBMC\addons\script.common.plugin.cache\default.py", line 40, in <module>
                                                run()
                                              File "C:\Users\Harry Kious\AppData\Roaming\XBMC\addons\script.common.plugin.cache\default.py", line 35, in run
                                                s.run()
                                              File "C:\Users\Harry Kious\AppData\Roaming\XBMC\addons\script.common.plugin.cache/lib\StorageServer.py", line 263, in run
                                                if self.xbmcvfs.exists(self.socket):
                                            TypeError: argument "path" for method "XBMCAddon::xbmcvfs::exists" must be unicode or str
                                            -->End of Python script error report<--[code]

After running it, the movie database was gone.  Gone in that it seemed to have nothing in it. But when I looked at the file size, it was the same as my backup. The TV Shows (PVR) were all fine.
I restored and am none the worse for wear.
Let me know if you need anything else and/or when you want me to give it a go again. Will be off on a shopping trip with the wife, but will be back home in the afternoon.
Thanks again for your efforts.
(This post was last modified: 2016-05-04 17:00 by ReplayHarry.)
find quote
black_eagle Offline
Posting Freak
Posts: 1,815
Joined: Sep 2013
Reputation: 79
Location: East Yorkshire, UK
Post: #58
I should note that this was added to preserve plugin data originally, and I haven't tested it with any other paths.

Theoretically however it would exclude any valid database path, or part thereof.

So, 'nfs://' would exclude any path starting with that including all sub-directories. Same for 'smb://'.

'nfs://MyServer/my-tvshows/' Would exclude anything in that path and any sub-directories of it. It would not exclude 'nfs://MyServer/My-recordings/' For that you could use 'nfs://MyServer/' which would exclude both paths.

HTH

Learning Linux the hard way !!
find quote
black_eagle Offline
Posting Freak
Posts: 1,815
Joined: Sep 2013
Reputation: 79
Location: East Yorkshire, UK
Post: #59
(2016-05-04 16:58)ReplayHarry Wrote:  
(2016-05-04 10:24)black_eagle Wrote:  Link for a gotham installable zip version.

At least, I think it should install, but I'm unable to test it for you guys.

No go. I didn't have debug turned on when I tried to run it. Here is the error I got. I'll turn on debug and try again to get complete info if you need it.

Code:
7:04:58 T:5460   ERROR: EXCEPTION: argument "path" for method "XBMCAddon::xbmcvfs::exists" must be unicode or str
07:04:58 T:5460   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.TypeError'>
                                            Error Contents: argument "path" for method "XBMCAddon::xbmcvfs::exists" must be unicode or str
                                            Traceback (most recent call last):
                                              File "C:\Users\Harry Kious\AppData\Roaming\XBMC\addons\script.common.plugin.cache\default.py", line 40, in <module>
                                                run()
                                              File "C:\Users\Harry Kious\AppData\Roaming\XBMC\addons\script.common.plugin.cache\default.py", line 35, in run
                                                s.run()
                                              File "C:\Users\Harry Kious\AppData\Roaming\XBMC\addons\script.common.plugin.cache/lib\StorageServer.py", line 263, in run
                                                if self.xbmcvfs.exists(self.socket):
                                            TypeError: argument "path" for method "XBMCAddon::xbmcvfs::exists" must be unicode or str
                                            -->End of Python script error report<--[code]

After running it, the movie database was gone.  Gone in that it seemed to have nothing in it. But when I looked at the file size, it was the same as my backup. The TV Shows (PVR) were all fine.
I restored and am none the worse for wear.
Let me know if you need anything else and/or when you want me to give it a go again. Will be off on a shopping trip with the wife, but will be back home in the afternoon.
Thanks again for your efforts.

That error is not our add-on, it's script.common.plugin.cache .

Seems to have worked OK for PatK, so unsure without a debug log what may have happened with yours Harry Sad

Learning Linux the hard way !!
find quote
pr0xZen Offline
Fan
Posts: 491
Joined: Feb 2014
Reputation: 20
Post: #60
(2016-05-04 15:34)orby Wrote:  [...]
I belive this above (see black_eagle's posts), might be your ticket home.

If I have helped you or increased your knowledge - please click the plus to the left below to give thanks
(This post was last modified: 2016-05-04 17:07 by pr0xZen.)
find quote
Post Reply