[jbinkley60]

The error message is the same as in post #82. I'm guessing it's either the movie title length or a special character that triggers the error in the case of all TV shows and Movies that start with 1,2 or 9. For example, there's a TV show with the title

'Allo 'Allo!

and movies with titles like

1900
2012
23
9

If you want, I can get you access to my server for testing or provide you with a database dump - please PM me for that.

Was it giving the same error with 0.0.9f ?

Jeff

[HeresJohnny]

I didn't try browsing with the previous version, just cleaning.

[jbinkley60]

I didn't try browsing with the previous version, just cleaning.

Thanks for your database dump.  It was helpful in resolving the issue.  What was happening is that for movies that began with 1,2,3 and 9 MySQL was returning the movie names which began with those numbers as integers but for the rest it returned them as strings.  In looking closer 1,2,3 and 9 only had movies with numbers in them, whereas the others had characters (i.e. 47 Ronin etc..) so for those they had to be returned as strings.  Anyway, the easy fix was just to convert everything received from MySQL to a string. 

I also found some interesting things in TV Shows.  You might want to look at the name field in the seasons table.  The Kodi scraper generally leaves it blank and sometimes places some odd characters there.  For my Mezzmo sync I insert Season 1, Season 2 etc..  I modified cleaner addon to ignore the name field and I just create the season listing by appending the season number to "Season - ".  Otherwise you would get a screen full of invalid season name errors.  

I've posted version 0.0.9h if you want to give it a test.


Thanks,

Jeff

[HeresJohnny]

All issues seem to have been resolved and the Video Database Cleaner appears to be fully functioning. The project is coming along nicely! I noticed, though, that after scraping new items into the movie database that KSC immediately finds new items to clean, especially in the actor and *-link tables. I wonder if that's scraper related.

[jbinkley60]

All issues seem to have been resolved and the Video Database Cleaner appears to be fully functioning. The project is coming along nicely! I noticed, though, that after scraping new items into the movie database that KSC immediately finds new items to clean, especially in the actor and *-link tables. I wonder if that's scraper related.

It is likely scraper related but is this scraping something new or rescraping an existing media file ?  I wouldn't expect something new to do this but I could easily see a rescrape where the actor information may have gotten updated and some actors were removed.  If it is happening on new items I'd compare what it wants to clean to whether those actors are really aligned to the newly scraped media.  It is possible there is an SQL query issue in my code but I am not seeing it with my Mezzmo scraper which syncs to Kodi. 

If you get something specific, I can look at it.  I'd need a CSV export of the actor table and the actor_link table (which I need to add the link tables to the KSC tool) before and after a scrape along with a CSV output of the analyzer table analysis when it sees new stuff to clean.  From that I can dig into what is happening in detail.  I'll add some of the link tables to the CSV exporter in the next day or so. 
 

I appreciate the feedback on the tool.  While I will rarely use it, I hope it helps folks maintain their databases better and provide more visibility.  A few things I have as other potential features are:

- Complete the media browser section where you can selectively delete things in your library and the tool will clean out all relevant entries vs. leaving orphaned entries (potentially including deleting the media file itself).
- A media comparison routine where it will scan your library looking for missing media files.  This would likely be slow and I'd need a progress bar.  
- Finish the music database analyzer

I am open to suggestions and preferences, as time permits.  I presume the analyzer / cleaner is running pretty quick now with the shared cursor code change ?


Thanks,

Jeff

[jbinkley60]

I've posted version 0.0.9i which now has support for the actor_link, director_link and writer_link tables since they all point to the actor table.  At some point I may just add all the Kodi tables to the CSV exporter, since I have all the code in the Mezzmo addon.   Note that the actor_link table can take a few seconds to generate the CSV since folks may have a few hundred thousand or more entries in that table.  My Kodi instance has just over 400k. 


Jeff

[gibxxi]

I tried the latest version 0.0.9i, but KS Cleaner pops-up the message "There is a problem with your MySQL Configuration" 4 times (when pressing the OK button for the error pop-up) and the following entry is written to the log:
 

Code:

2023-03-25 02:55:04.895 T:640     error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\Kodi-Selective-Cleaner\resources\lib\common.py, LINE 220 "db = mysql.connector.connect(**config)"): connect() argument after ** must be a mapping, not NoneType

My advancedsettings.xml file is valid, and Kodi can see the databases, but KS Cleaner can't, evidently. Probably something i've done wrong here (It usually is), but the error message above means nothing to me. Any ideas?

Dan / Gib.

[jbinkley60]

I tried the latest version 0.0.9i, but KS Cleaner pops-up the message "There is a problem with your MySQL Configuration" 4 times (when pressing the OK button for the error pop-up) and the following entry is written to the log:
 

Code:

2023-03-25 02:55:04.895 T:640     error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\Kodi-Selective-Cleaner\resources\lib\common.py, LINE 220 "db = mysql.connector.connect(**config)"): connect() argument after ** must be a mapping, not NoneType

My advancedsettings.xml file is valid, and Kodi can see the databases, but KS Cleaner can't, evidently. Probably something i've done wrong here (It usually is), but the error message above means nothing to me. Any ideas?

I'll need to see your advanced settings MySQL settings.  This is all new code so it is possible there are setups I am not checking for.  Are you maybe only running a video database off of MySQL and don't have a music database section ?  Line 220 is associated with the music database.

Here's my file:

python:
<advancedsettings>

    <videodatabase>

        <type>mysql</type>

        <host>192.168.0.35</host>

        <port>3306</port>

        <user>kodi</user>

        <pass>password</pass>

        <compression>true</compression>

    </videodatabase>

    <musicdatabase>

        <type>mysql</type>

        <host>192.168.0.35</host>

        <port>3306</port>

        <user>kodi</user>

        <pass>password</pass>

        <compression>true</compression>

    </musicdatabase>

</advancedsettings>

Thanks,

Jeff

[gibxxi]

Here's mine:
 

Code:

    <videodatabase>
        <type>mysql</type>
        <host>192.168.0.10</host>
        <port>3306</port>
        <user>REMOVED</user>
        <pass>REMOVED</pass>
        <name>kodi_video_v</name>
        <compression>true</compression>
    </videodatabase>

<musicdatabase>
        <type>mysql</type>
        <host>192.168.0.10</host>
        <port>3306</port>
        <user>REMOVED</user>
        <pass>REMOVED</pass>
        <name>kodi_music_v</name>
        <compression>true</compression>
    </musicdatabase>

My xml file also has a lot of annotation in the file in the way of "<!-- / -->" tags (I'm a bit OCD like that). I've stripped those out for purposes of posting here. As well as obfuscated the usernames & passwords of course.

I have TWO NASes running both the MySQL databases. If one fails, I can quickly switch to the other, as there have been issues with the MySQL servers in the past, and other issues with my general network setup that I won't bore you with here. Only one setup (NAS MySQL server) is in use at any one time though, just to be clear.

Dan / Gib.

[HeresJohnny]

I would like to throw some ideas around what I would love to see checked in the music database. Really, just a brainstorming and some ideas might be outside the scope of this add-on.

1. Table "album artist"
This table ties an album artist to an artist ID and an album ID
a) Apparently due to some manual tagging errors on my part some album artists appear with their musicbrainz ID. It would be great if the Analyzer could tell me which artist and album this concerns. Of course, corrections need then to be applied at file tag level.
b) Theoretically there could be cases where the same album artist name is linked to different artist IDs. It would be great if the Analyzer could point out these occurences.

2. Table "artist"
Most importantly, this table ties a local artist name to an artist ID entry in the musicbrainz database.
Even though I have meticulously tagged my files there are literally thousands of entries without musicbrainz IDs in this table and I'm unsure why this is. I suspect that additional tags other than "album artist" and "artist" are evaluated for this table, like composer and performer.
My OCD compels me to want to consolidate matching names and I wonder if and how a music database analyzer could support me in this. A listing of matching names with and without musicbrainz IDs might be within the scope of this add-on. Ideally an (additional) add-on would be able to (semi-automatically after revision by the user) apply corrections and write musicbrainz ID to green-lighted entries.

3. Table "role"
Don't even get me started... in my case there are over thousand entries, many containing names instead of functions or instruments... I suspect this is a consequence of a mixture of bad tags and bad musicbrainz data quality. I'm open for suggestions but I'm not optimistic to be able to fix this... ever :-)

[gibxxi]

Prior to the earlier error message I posted, I'm also seeing this one (must of overlooked it last time):

Code:

2023-03-25 13:33:12.436 T:26632   error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 85 "tree = ET.parse(config_file)"): not well-formed (invalid token): line 48, column 15

Not sure whether this is referring to a line in your code, or my advancedsettings.xml. I tried removing all my annotations out of the file, but it made no difference.

Dan / Gib.

[jbinkley60]

Prior to the earlier error message I posted, I'm also seeing this one (must of overlooked it last time):
 

Code:

2023-03-25 13:33:12.436 T:26632   error <general>: EXCEPTION IN (D:\Kodi Media Center\portable_data\addons\plugin.program.kscleaner\resources\lib\common.py, LINE 85 "tree = ET.parse(config_file)"): not well-formed (invalid token): line 48, column 15

Not sure whether this is referring to a line in your code, or my advancedsettings.xml. I tried removing all my annotations out of the file, but it made no difference.

Dan / Gib.

It is having a problem parsing your advancedsettings.xml file.  The other tags shouldn't be a problem.  Do me a favor and drop into debug mode, restart the addon and  it will log what it finds in your config file and that should give me more insight as to what is happening.  Also, can you post line in your advancedsettings.xml file which corresponds to this invalid token error (not sure if it is still line 48) ?  This is a general XML parsing error.


Thanks,

Jeff

[jbinkley60]

I would like to throw some ideas around what I would love to see checked in the music database. Really, just a brainstorming and some ideas might be outside the scope of this add-on.

1. Table "album artist"
This table ties an album artist to an artist ID and an album ID
a) Apparently due to some manual tagging errors on my part some album artists appear with their musicbrainz ID. It would be great if the Analyzer could tell me which artist and album this concerns. Of course, corrections need then to be applied at file tag level.
b) Theoretically there could be cases where the same album artist name is linked to different artist IDs. It would be great if the Analyzer could point out these occurences.

2. Table "artist"
Most importantly, this table ties a local artist name to an artist ID entry in the musicbrainz database.
Even though I have meticulously tagged my files there are literally thousands of entries without musicbrainz IDs in this table and I'm unsure why this is. I suspect that additional tags other than "album artist" and "artist" are evaluated for this table, like composer and performer.
My OCD compels me to want to consolidate matching names and I wonder if and how a music database analyzer could support me in this. A listing of matching names with and without musicbrainz IDs might be within the scope of this add-on. Ideally an (additional) add-on would be able to (semi-automatically after revision by the user) apply corrections and write musicbrainz ID to green-lighted entries.

3. Table "role"
Don't even get me started... in my case there are over thousand entries, many containing names instead of functions or instruments... I suspect this is a consequence of a mixture of bad tags and bad musicbrainz data quality. I'm open for suggestions but I'm not optimistic to be able to fix this... ever :-)

Thanks for the suggestions.  I somewhat answered my thoughts on this near term in the other thread on artist table uniqueness.  If I go down this road with the music table analyzer the Mezzmo Kodi addon syncing music will take priority and then I would use the output of that effort and adapt to the Kodi Selective Cleaner.  Ironically there has been a bit of interest in the Kodi music database analysis, with even some of the Kodi developers interested in this feature.  I suspect others may be seeing what you are seeing.  

Did you ever figure out anything with the actor_link table entries ?  Also I was amazed at the size of your path table, 2727 entries.  Mine has 4 with about the same number of entries 


Jeff

[gibxxi]

I'll PM you a copy of it. I edit it with Notepad++. I'm suspecting at this point some of my 'music seperators' might be causing a problem here as they do contain special characters and the file has Notepad++ TAB spacing utilized which the parser might not like.

I'll also do a debug run, and post you the output. Thanks for the help.



Dan / Gib.

[jbinkley60]

I'll PM you a copy of it. I edit it with Notepad++. I'm suspecting at this point some of my 'music seperators' might be causing a problem here as they do contain special characters and the file has Notepad++ TAB spacing utilized which the parser might not like.

I'll also do a debug run, and post you the output. Thanks for the help.

OK, I think I have it fixed.   You are correct on the cause.  It was an XML parsing error on your advancedsettings.xml file caused by this line in your musiclibrary section:

        <separator> & </separator>

I uploaded a fix to version 0.0.9i vs. a new release because I am working on being able to set the database type (i.e. local or MySQL) separately for the music and video databases and that code is underway.  It was a 2 line code fix for this issue.

A more proper fix would be to change your advancedsettings file to:

        <separator> \u0026 </separator>            

but you'd need to ensure Kodi parses that correctly.  That worked with the current version of Kodi Selective Cleaner prior to the fix.  Now both options should work.


Thanks,

Jeff