[HeresJohnny]

Hi there, reporting back for the path clean function. I see several problems in my debug log, of which I'm posting a sample:

Code:

20:14:32.231 T:9388   DEBUG: Thread LanguageInvoker start, auto delete: false
20:14:32.232 T:9388    INFO: initializing python engine.
20:14:32.232 T:9388   DEBUG: CPythonInvoker(61, C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs\standalone.py): start processing
20:14:32.298 T:9388   DEBUG: -->Python Interpreter Initialized<--
20:14:32.298 T:9388   DEBUG: CPythonInvoker(61, C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs\standalone.py): the source file to load is "C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs\standalone.py"
20:14:32.298 T:9388   DEBUG: CPythonInvoker(61, C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs\standalone.py): setting the Python path to C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs;C:\Program Files (x86)\Kodi\addons\script.module.pil\lib;C:\Users\JoScha\AppData\Roaming\Kodi\addons\script.module.beautifulsoup\lib;C:\Users\JoScha\AppData\Roaming\Kodi\addons\script.module.myconnpy\lib;C:\Program Files (x86)\Kodi\system\python\DLLs;C:\Program Files (x86)\Kodi\system\python\Lib;C:\Program Files (x86)\Kodi\python27.zip;C:\Program Files (x86)\Kodi\system\python\lib\plat-win;C:\Program Files (x86)\Kodi\system\python\lib\lib-tk;C:\Program Files (x86)\Kodi;C:\Program Files (x86)\Kodi\system\python;C:\Program Files (x86)\Kodi\system\python\lib\site-packages
20:14:32.298 T:9388   DEBUG: CPythonInvoker(61, C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs\standalone.py): entering source directory C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs
20:14:32.298 T:9388   DEBUG: CPythonInvoker(61, C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs\standalone.py): instantiating addon using automatically obtained id of "service.cleansubs" dependent on version 2.1.0 of the xbmc.python api
20:14:32.634 T:9388   DEBUG: CLEANSUBS >> DEFINITIONS >> NO NEW DEFINITIONS (L:21244 == R:21244)
20:14:32.693 T:9388   DEBUG: CLEANSUBS >> DELETED AND CREATED NEW DEF DB
20:14:43.619 T:9388   DEBUG: CLEANSUBS >> READ TOTAL DEFINITIONS: 0 elements
20:14:43.619 T:9388   DEBUG: CLEANSUBS STANDALONE >> STARTED VERSION 5.0
20:14:43.620 T:9388   DEBUG: JSONRPC: Incoming request: {
                                                "jsonrpc": "2.0",
                                                "id": 1,
                                                "method": "Files.GetSources",
                                                "params": {
                                                    "media": "video"
                                                    }
                                                }
20:14:43.620 T:9388   DEBUG: CLEANSUBS >> VIDEO PATHS >> multipath://nfs%3a%2f%2f192.168.1.185%2fd%2fTV-Movie%2f_Movie%2fAction-Adventure-Western%2f/nfs%3a%2f%2f192.168.1.185%2fd%2fTV-Movie%2f_Movie%2fAsian%2f/nfs%3a%2f%2f192.168.1.185%2fd%2fTV-Movie%2f_Movie%2fComedy-Family-Romance%2f/nfs%3a%2f%2f192.168.1.185%2fd%2fTV-Movie%2f_Movie%2fCrime-Suspense-Mystery%2f/nfs%3a%2f%2f192.168.1.185%2fd%2fTV-Movie%2f_Movie%2fDrama-War%2f/nfs%3a%2f%2f192.168.1.185%2fd%2fTV-Movie%2f_Movie%2fHorror%2f/nfs%3a%2f%2f192.168.1.185%2fd%2fTV-Movie%2f_Movie%2fSf-Fantasy%2f/nfs%3a%2f%2f192.168.1.185%2ft%2fTV-Movie2%2f_Doku%2f/nfs%3a%2f%2f192.168.1.185%2ft%2fTV-Movie2%2f_Anime%2f_Movies%2f/
20:14:43.620 T:9388   DEBUG: CLEANSUBS >> VIDEO PATHS >> nfs://192.168.1.185/t/TV-Movie2/_Anime/_Series/
20:14:43.620 T:9388   DEBUG: CLEANSUBS >> VIDEO PATHS >> nfs://192.168.1.185/t/TV-Movie2/_tv/
20:14:43.620 T:9388   DEBUG: CLEANSUBS >> VIDEO PATHS >> nfs://192.168.1.185/q/Music/_dvd-V/
20:14:43.620 T:9388   DEBUG: CLEANSUBS >> VIDEO PATHS >> nfs://192.168.1.185/d/TV-Movie/_Movie/Animation/
20:14:48.486 T:9388   DEBUG: DialogProgress::Open called
20:14:48.486 T:9388   DEBUG: ------ Window Init (DialogConfirm.xml) ------
20:16:11.077 T:9388   DEBUG: CLEANSUBS STANDALONE >> BEGIN PATH: >>\\POSTMAN\TV-Movie2\_tv\<< FOLDERS IN PATH: >>362<<
...
20:16:51.647 T:9388   DEBUG: CLEANSUBS STANDALONE >> FILE: >>American.Horror.Story.S02E12.en.srt<<
20:16:51.677 T:9388   DEBUG: CLEANSUBS >> SQL ERROR IN CheckDatabase
20:16:51.677 T:9388   DEBUG: CLEANSUBS >> SUB STATS WILL BE ADDED TO LOCAL DATABASE
20:16:51.692 T:9388   DEBUG: CLEANSUBS >> ENC >> OPENED WITH ENCODING: utf-8
20:16:51.713 T:9388   DEBUG: CLEANSUBS >> SQL ERROR IN AddtoDatabase : no such table: stats
20:16:51.736 T:9388   DEBUG: Previous line repeats 1 times.
20:16:51.736 T:9388   DEBUG: CLEANSUBS >> PROCESSED IN 0.09 SECONDS, NO LINES REMOVED
20:16:51.736 T:9388   DEBUG: CLEANSUBS STANDALONE >> FILE: >>American.Horror.Story.S02E12.ja.srt<<
20:16:51.766 T:9388   DEBUG: CLEANSUBS >> SQL ERROR IN CheckDatabase
20:16:51.767 T:9388   DEBUG: CLEANSUBS >> SUB STATS WILL BE ADDED TO LOCAL DATABASE
20:16:51.777 T:9388   DEBUG: CLEANSUBS >> ENC >> TRYING ENCODING utf-8
20:16:51.782 T:9388   DEBUG: CLEANSUBS >> ENC >> TRYING ENCODING cp1250
20:16:51.792 T:9388   DEBUG: CLEANSUBS >> ENC >> TRYING ENCODING cp1251
20:16:51.797 T:9388   DEBUG: CLEANSUBS >> ENC >> TRYING ENCODING cp1252
20:16:51.807 T:9388   DEBUG: CLEANSUBS >> ENC >> TRYING ENCODING cp1253
20:16:51.817 T:9388   DEBUG: CLEANSUBS >> ENC >> TRYING ENCODING cp1254
20:16:51.827 T:9388   DEBUG: CLEANSUBS >> ENC >> TRYING ENCODING cp1257
20:16:51.827 T:9388   DEBUG: CLEANSUBS >> ENC >> OPENED WITH KODI ENCODING:
20:16:51.848 T:9388   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.LookupError'>
                                            Error Contents: unknown encoding:
                                            Traceback (most recent call last):
                                              File "C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs\standalone.py", line 290, in <module>
                                                intCancel = scanPaths(manFolder, 1, 1, 3)
                                              File "C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs\standalone.py", line 176, in scanPaths
                                                process_subs(os.path.join(path, basePath, name), 1)
                                              File "C:\Users\JoScha\AppData\Roaming\Kodi\addons\service.cleansubs\default.py", line 300, in process_subs
                                                file_input = codecs.open(fileName, 'r', SubCharset, errors='ignore')
                                              File "C:\Program Files (x86)\Kodi\system\python\Lib\codecs.py", line 899, in open
                                                info = lookup(encoding)
                                            LookupError: unknown encoding:
                                            -->End of Python script error report<--

1. It seems like there's problem with a non-existing table "stats".
2. It seems like there's a problem with double-byte encoded subs, in my case codepage 932 ANSI/OEM Japanese (Shift JIS). That's the part where Cleansubs tries a few different codepages and finally fails with erroring out. Maybe this could be made more resilient by skipping.

3. The cleaning itself seems to fail partially. Example:
American.Horror.Story.S02E12.en.srt vs. American.Horror.Story.S02E12.en.srt_ORIGINAL

Cleansubs manages to clean the last lines of the sub which are

Code:

728
00:42:42,598 --> 00:42:52,817
<font color="#ec14bd">Sync & corrections by honeybunny</font>
<font color="#ec14bd">www.addic7ed.com</font>

However, it fails to clean stuff from the top which still has

Code:

1
00:00:48,917 --> 00:00:51,152
Daddy?

2
00:00:51,220 --> 00:00:53,788
Daddy'll be there in a minute.

3
00:01:48,608 --> 00:01:58,632
<font color="#ec14bd">Sync & corrections by honeybunny</font>
<font color="#ec14bd">www.addic7ed.com</font>

I'll be back with some more tests about NFS.