[jbinkley60]

@HeresJohnny   I ended up making great progress today on the Video DB Data Analyzer code.  I posted version 0.0.9d which should work on all tables up through the musicvideo table with MySQL.   All three options, Analyze, Analyze / CSV Export and Analyze / Clean Table should work.  I highly recommend making a database backup before  using the cleaner function.  I've tested it on my test setup but backups are always good.


Thanks,

Jeff

[HeresJohnny]

Great work! Ran it against my database... over 30.000 actor_link entries were cleaned, that took a good chunk of time. Maybe a progress bar would help show the user that Kodi is still alive :-)

[jbinkley60]

Great work! Ran it against my database... over 30.000 actor_link entries were cleaned, that took a good chunk of time. Maybe a progress bar would help show the user that Kodi is still alive :-)

Thanks for the update.  I'll give the progress bar some thought.  I would suspect normally it will be much faster after an initial cleaning.  You've helped prove my point about how messy the databases can become.  I have the remainder of the tables done, I just need to finish some final testing.  I added / modified a few hundred lines of code.  I should post a version tomorrow with all tables in the menu working with MySQL.


Thanks,

Jeff

[jbinkley60]

Great work! Ran it against my database... over 30.000 actor_link entries were cleaned, that took a good chunk of time. Maybe a progress bar would help show the user that Kodi is still alive :-)

I've posted version 0.0.9e which has support for all tables with MySQL.  I looked at what it would take to add a progress dialog box during cleaning.  It's a small amount of work so I'll look to implement in the next release.  I also see that some of the slowness is due to me opening / closing a MySQL cursor on each clean action on every item.  I'll see what it would take to reuse the cursor for all clean actions in a single run which should speed things up significantly.  SQLite  doesn't do this.  It does them as a single batch.   


Thanks,

Jeff

[jbinkley60]

Great work! Ran it against my database... over 30.000 actor_link entries were cleaned, that took a good chunk of time. Maybe a progress bar would help show the user that Kodi is still alive :-)

I was able to punch out a shared MySQL cursor solution with version 0.0.9f .  It should be much faster now cleaning large tables.  Progress bar is for another day but I don't think it may be needed now.


Thanks,

Jeff

[HeresJohnny]

All videodatabase tables cleaned out nicely with 0.0.9f. However, I triggered another error while browsing movies (obfuscating login data):

xml:
2023-03-14 14:26:29.211 T:9624    debug <general>: Kodi selective cleaner movie menu selection is: 1

2023-03-14 14:26:29.211 T:9624    debug <general>: KC Cleaner Movies Menu selection: 1

2023-03-14 14:26:29.213 T:9624    debug <general>: KS Cleaner parse: mysql 127.0.0.1 3306 xxx xxx myvideos121

2023-03-14 14:26:29.295 T:9624    error <general>: KS Cleaner Movies error. 

2023-03-14 14:26:29.296 T:9624    error <general>: EXCEPTION IN (C:\Kodi\portable_data\addons\plugin.program.kscleaner\resources\lib\movies.py, LINE 78 "if len(movie[2]) < 1:                               # Handle blank movie names"): object of type 'int' has no len()

Also, strangely enough, there seems to be a movie with a blank entry

xml:


2023-03-14 14:26:25.712 T:9624    debug <general>: Kodi selective cleaner movie menu selection is: 

2023-03-14 14:26:25.712 T:9624    debug <general>: KC Cleaner Movies Menu selection: 

2023-03-14 14:26:25.714 T:9624    debug <general>: KS Cleaner parse: mysql 127.0.0.1 3306 xxx xxx myvideos121

2023-03-14 14:26:25.879 T:9624    error <general>: KS Cleaner Movies error. 

2023-03-14 14:26:25.879 T:9624    error <general>: EXCEPTION IN (C:\Kodi\portable_data\addons\plugin.program.kscleaner\resources\lib\movies.py, LINE 78 "if len(movie[2]) < 1:                               # Handle blank movie names"): object of type 'int' has no len()

As far as I know, that is "The Doom that came to Gotham" which was scraped with UMS. Would it be possible to log which line in the affected table triggered the error?

[jbinkley60]

All videodatabase tables cleaned out nicely with 0.0.9f. However, I triggered another error while browsing movies (obfuscating login data):

xml:
2023-03-14 14:26:29.211 T:9624 debug <general>: Kodi selective cleaner movie menu selection is: 1

2023-03-14 14:26:29.211 T:9624 debug <general>: KC Cleaner Movies Menu selection: 1

2023-03-14 14:26:29.213 T:9624 debug <general>: KS Cleaner parse: mysql 127.0.0.1 3306 xxx xxx myvideos121

2023-03-14 14:26:29.295 T:9624 error <general>: KS Cleaner Movies error.

2023-03-14 14:26:29.296 T:9624 error <general>: EXCEPTION IN (C:\Kodi\portable_data\addons\plugin.program.kscleaner\resources\lib\movies.py, LINE 78 "if len(movie[2]) < 1: # Handle blank movie names"): object of type 'int' has no len()

Also, strangely enough, there seems to be a movie with a blank entry

xml:


2023-03-14 14:26:25.712 T:9624 debug <general>: Kodi selective cleaner movie menu selection is:

2023-03-14 14:26:25.712 T:9624 debug <general>: KC Cleaner Movies Menu selection:

2023-03-14 14:26:25.714 T:9624 debug <general>: KS Cleaner parse: mysql 127.0.0.1 3306 xxx xxx myvideos121

2023-03-14 14:26:25.879 T:9624 error <general>: KS Cleaner Movies error.

2023-03-14 14:26:25.879 T:9624 error <general>: EXCEPTION IN (C:\Kodi\portable_data\addons\plugin.program.kscleaner\resources\lib\movies.py, LINE 78 "if len(movie[2]) < 1: # Handle blank movie names"): object of type 'int' has no len()

As far as I know, that is "The Doom that came to Gotham" which was scraped with UMS. Would it be possible to log which line in the affected table triggered the error?

Sure.  I can also check for blanks better to avoid these exceptions.  You should be able to see the specifics by doing a CSV export of the movie table.


Thanks,

Jeff

[jbinkley60]

As far as I know, that is "The Doom that came to Gotham" which was scraped with UMS. Would it be possible to log which line in the affected table triggered the error?

Do me a favor and don't fix the error in your database.  let me code a fix and then we can test it.  I'll probably write a message to the logs so you can see the error message and maybe a popup dialog box to for immediate visibility.


Jeff

[HeresJohnny]

Also, strangely enough, there seems to be a movie with a blank entry

Turns out, I forgot to change "Get title" from IMDB to themoviedb for one source...

[HeresJohnny]

Do me a favor and don't fix the error in your database.  let me code a fix and then we can test it.  I'll probably write a message to the logs so you can see the error message and maybe a popup dialog box to for immediate visibility.

Sorry, already fixed it. I can reproduce it again by emptying the culprit field in the db, though.

[jbinkley60]

Also, strangely enough, there seems to be a movie with a blank entry

Turns out, I forgot to change "Get title" from IMDB to themoviedb for one source...

So it sounds like the tool is handy at helping keep things cleaner.  I presume the cleaning process was much faster with the shared cursor ?


Jeff

[jbinkley60]

Do me a favor and don't fix the error in your database.  let me code a fix and then we can test it.  I'll probably write a message to the logs so you can see the error message and maybe a popup dialog box to for immediate visibility.

Sorry, already fixed it. I can reproduce it again by emptying the culprit field in the db, though.

If you'd like to try again.  I added a check for a blank title field.  Version 0.0.9g has the update.  It will show in the listing that the title is invalid and write it to the addon logs.



Jeff

[HeresJohnny]

Hm, I'm getting "Can't connect to Kodi databases" again for Movies and TV shows.

[jbinkley60]

Hm, I'm getting "Can't connect to Kodi databases" again for Movies and TV shows.

That is odd.  Can you posted the exception message out of the kodi.log file so I an see what is happening ?  I double checked that I uploaded the correct code and it is working for me.  also please make sure it says version 0.0.9g and you didn't accidentally load an older version.


Jeff

[HeresJohnny]

The error message is the same as in post #82. I'm guessing it's either the movie title length or a special character that triggers the error in the case of all TV shows and Movies that start with 1,2 or 9. For example, there's a TV show with the title

'Allo 'Allo!

and movies with titles like

1900
2012
23
9

If you want, I can get you access to my server for testing or provide you with a database dump - please PM me for that.