Login at Kodi Home

dziobak · (This post was last modified: 2017-11-01, 10:52 by dziobak.)

Quote:Edworld
What does the thread option do? default is 8

Thanks for fixing the tvshow progress, now shows progress at an episode level.

From 1st post:

Quote:For Movies and TV Shows libraries update, in the add-on settings it is possible to set the number of parallel threads (default value is 8). Please use this option carefully: a high number of threads may lead to an inefficent behaviour as well as increase too much the number of queries to the IMDb site (the requests will be rejected). In case you experience any issues, use 4 as a balanced number of threads for any platform.

In processing - addon does 2 things: waiting for IMDB answer and compute this answer later. So we made some parallelism, to minimise idle time in first thing. This gives gain in addon speed, but only when not causes overload (this gives slowdown). We made some tests, and 8 threads is safe on RPi platform (and on PC also).
About timeouts - we belive that IMDb site has some limiting on queries, so addon pause in IMDB requests when timeout occurs. If this will be too frequent (in effect of increasing threads number), then gives slowdown also.
We add additional settings about threads number (12 and 16) for testing purpose mostly, but be aware of above.

Glottis · 2017-11-01, 16:07

I get the following several times when the update process runs (with different movies):

Code:
50:32.217 T:548185571552   DEBUG: [Light IMDb Ratings Update] - "Sucker Punch": Update started (IMDb ID: tt0978764, TVDB ID: None)

50:32.312 T:548185571552   ERROR: Unhandled exception in thread started by

50:32.312 T:548185571552   ERROR: <function thread_parse_IMDb_page at 0x7f2e195938>

50:32.312 T:548185571552   ERROR: Traceback (most recent call last):

50:32.312 T:548185571552   ERROR:   File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/update_main.py", line 50, in thread_parse_IMDb_page

50:32.312 T:548185571552   ERROR: (updatedRating, updatedVotes, updatedTop250, statusInfo) = parse_IMDb_page(IMDb)

50:32.312 T:548185571552   ERROR:   File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/imdb_scraper.py", line 49, in parse_IMDb_page

50:32.313 T:548185571552   ERROR: (html, status, statusInfo) = get_IMDb_page(imdb_id)

50:32.313 T:548185571552   ERROR:   File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/imdb_scraper.py", line 27, in get_IMDb_page

50:32.313 T:548185571552   ERROR: response = urllib2.urlopen(req)

50:32.313 T:548185571552   ERROR:   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen

50:32.313 T:548185571552   ERROR:   File "/usr/lib/python2.7/urllib2.py", line 429, in open

50:32.313 T:548185571552   ERROR:   File "/usr/lib/python2.7/urllib2.py", line 447, in _open

50:32.313 T:548185571552   ERROR:   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain

50:32.314 T:548185571552   ERROR:   File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open

50:32.314 T:548185571552   ERROR:   File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open

50:32.314 T:548185571552   ERROR: urllib2

50:32.314 T:548185571552   ERROR: .

50:32.314 T:548185571552   ERROR: URLError

50:32.314 T:548185571552   ERROR: :

50:32.314 T:548185571552   ERROR: <urlopen error [Errno 113] No route to host>

The update kinda freezes. Once at 80% of the movies and now at 63%.
I'm using 3.2.0 on Odroid C2 (64-bit ARM) with Libreelec 8.2 (Kodi 17.5).

PS: I cannot start the process from the add-ons menu (the buttons are disabled). Is there another problem? I deleted everything (deinstalled the add-on and removed the addon-data folders) and reinstalled Light IMDb Ratings Update, but it's still the same as before.

Thank you all.

dziobak · (This post was last modified: 2017-11-01, 18:09 by dziobak.)

(2017-11-01, 16:07)Glottis Wrote: I get the following several times when the update process runs (with different movies):

Code:
14:50:32.217 T:548185571552 DEBUG: [Light IMDb Ratings Update] - "Sucker Punch": Update started (IMDb ID: tt0978764, TVDB ID: None) 14:50:32.312 T:548185571552 ERROR: Unhandled exception in thread started by 14:50:32.312 T:548185571552 ERROR: <function thread_parse_IMDb_page at 0x7f2e195938> 14:50:32.312 T:548185571552 ERROR: Traceback (most recent call last): 14:50:32.312 T:548185571552 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/update_main.py", line 50, in thread_parse_IMDb_page 14:50:32.312 T:548185571552 ERROR: (updatedRating, updatedVotes, updatedTop250, statusInfo) = parse_IMDb_page(IMDb) 14:50:32.312 T:548185571552 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/imdb_scraper.py", line 49, in parse_IMDb_page 14:50:32.313 T:548185571552 ERROR: (html, status, statusInfo) = get_IMDb_page(imdb_id) 14:50:32.313 T:548185571552 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/imdb_scraper.py", line 27, in get_IMDb_page 14:50:32.313 T:548185571552 ERROR: response = urllib2.urlopen(req) 14:50:32.313 T:548185571552 ERROR: File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen 14:50:32.313 T:548185571552 ERROR: File "/usr/lib/python2.7/urllib2.py", line 429, in open 14:50:32.313 T:548185571552 ERROR: File "/usr/lib/python2.7/urllib2.py", line 447, in _open 14:50:32.313 T:548185571552 ERROR: File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain 14:50:32.314 T:548185571552 ERROR: File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open 14:50:32.314 T:548185571552 ERROR: File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open 14:50:32.314 T:548185571552 ERROR: urllib2 14:50:32.314 T:548185571552 ERROR: . 14:50:32.314 T:548185571552 ERROR: URLError 14:50:32.314 T:548185571552 ERROR: : 14:50:32.314 T:548185571552 ERROR: <urlopen error [Errno 113] No route to host>

The update kinda freezes. Once at 80% of the movies and now at 63%.
I'm using 3.2.0 on Odroid C2 (64-bit ARM) with Libreelec 8.2 (Kodi 17.5).

PS: I cannot start the process from the add-ons menu (the buttons are disabled). Is there another problem? I deleted everything (deinstalled the add-on and removed the addon-data folders) and reinstalled Light IMDb Ratings Update, but it's still the same as before.

Thank you all.

I think that this error is specifical timeout on IMDb request. We will add handler for this error.

Update:
@Glottis, I sent you patched version for testing

After crash like this Kodi must be restarted (something holds add-on status as working).

Glottis · 2017-11-01, 20:59

(2017-11-01, 17:04)dziobak Wrote: Update:
@Glottis, I sent you patched version for testing

Thank you, but I'm unable to find your message. (Just updated my e-mail address so hopefully it will work next time)

Furthermore: Should I be able to start a scan from within the add-on's menu? (the place where I can configure and deinstall it)

axlt2002 · 2017-11-01, 21:53

(2017-11-01, 20:59)Glottis Wrote:
(2017-11-01, 17:04)dziobak Wrote: Update:
@Glottis, I sent you patched version for testing

Thank you, but I'm unable to find your message. (Just updated my e-mail address so hopefully it will work next time)

Furthermore: Should I be able to start a scan from within the add-on's menu? (the place where I can configure and deinstall it)

Hi Glottis,

I guess dziobak sent you a private message and not an email.

I have checked other programs add-on and it seems is not possible to run them from their menu. Btw, it was something possible in one of the first releases (launch it from the settings), but it was considered quite unconvenient by almost all the users so that is why it was modified to be launched from the Kodi Home-menu.

dziobak · (This post was last modified: 2017-11-01, 23:26 by dziobak.)

I sent email (and try another one few seconds ago), because: "Glottis has private messaging disabled. You cannot send private messages to this user."

About add-on starting, I added it to favorites and this gives easy way to run.

Edworld · 2017-11-02, 01:31

Is there a way of knowing when was the last time you updated the database?

IMDB update addon displays the date which I find very helpful

axlt2002 · (This post was last modified: 2017-11-02, 01:44 by axlt2002.)

(2017-11-02, 01:31)Edworld Wrote: Is there a way of knowing when was the last time you updated the database?

IMDB update addon displays the date which I find very helpful

Where that information was available? Pop-up window? In the add-on settings? Other?

Glottis · 2017-11-02, 02:40

(2017-11-01, 23:21)dziobak Wrote: I sent email (and try another one few seconds ago), because: "Glottis has private messaging disabled. You cannot send private messages to this user."

About add-on starting, I added it to favorites and this gives easy way to run.

Thanks again. This time I received it Wink

(I also enabled PMs now)

Will report back when I tested the patched version (probably tomorrow).

Edworld · (This post was last modified: 2017-11-02, 05:07 by Edworld.)

(2017-11-02, 01:44)axlt2002 Wrote:
(2017-11-02, 01:31)Edworld Wrote: Is there a way of knowing when was the last time you updated the database?

IMDB update addon displays the date which I find very helpful

Where that information was available? Pop-up window? In the add-on settings? Other?

You click on the addon and it presents 3 options:

Glottis · 2017-11-02, 15:12

I tested the script.light.imdb.ratings.update-3.2.0patch1 three times now, but it did not finish. There's a lot of the following error in the log file, but it still goes on and updates a few episodes until it finally freezes completely. (Again with the following message Smile

Code:
54:09.960 T:545688908000   ERROR: Unhandled exception in thread started by

54:09.960 T:545688908000   ERROR: <function thread_parse_IMDb_page at 0x7eeac4e938>

54:09.960 T:545688908000   ERROR: Traceback (most recent call last):

54:09.960 T:545688908000   ERROR:   File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/update_main.py", line 36, in thread_parse_IMDb_page

54:09.960 T:545688908000   ERROR: (IMDb, statusInfo) = get_IMDb_ID(dType, TVDB)

54:09.960 T:545688908000   ERROR:   File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/tvdb_scraper.py", line 30, in get_IMDb_ID

54:09.960 T:545688908000   ERROR: response = episode.info()

54:09.960 T:545688908000   ERROR:   File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/support/tvdbsimple/episode.py", line 53, in info

54:09.960 T:545688908000   ERROR: response = self._GET(path)

54:09.960 T:545688908000   ERROR:   File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/support/tvdbsimple/base.py", line 150, in _GET

54:09.960 T:545688908000   ERROR: return self._request('GET', path, params=params, cleanJson=cleanJson)

54:09.960 T:545688908000   ERROR:   File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/support/tvdbsimple/base.py", line 133, in _request

54:09.960 T:545688908000   ERROR: headers=self._headers)

54:09.960 T:545688908000   ERROR:   File "/storage/.kodi/addons/script.module.requests/lib/requests/api.py", line 56, in request

54:09.961 T:545688908000   ERROR: return session.request(method=method, url=url, **kwargs)

54:09.961 T:545688908000   ERROR:   File "/storage/.kodi/addons/script.module.requests/lib/requests/sessions.py", line 488, in request

54:09.961 T:545688908000   ERROR: resp = self.send(prep, **send_kwargs)

54:09.961 T:545688908000   ERROR:   File "/storage/.kodi/addons/script.module.requests/lib/requests/sessions.py", line 609, in send

54:09.961 T:545688908000   ERROR: r = adapter.send(request, **kwargs)

54:09.961 T:545688908000   ERROR:   File "/storage/.kodi/addons/script.module.requests/lib/requests/adapters.py", line 487, in send

54:09.962 T:545688908000   ERROR: raise ConnectionError(e, request=request)

54:09.962 T:545688908000   ERROR: requests.exceptions

54:09.962 T:545688908000   ERROR: .

54:09.962 T:545688908000   ERROR: ConnectionError

54:09.962 T:545688908000   ERROR: :

54:09.962 T:545688908000   ERROR: HTTPSConnectionPool(host='api.thetvdb.com', port=443): Max retries exceeded with url: /episodes/4845979 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7ee8618890>: Failed to establish a new connection: [Errno 113] No route to host',))

I can see why it says "Max retries exceeded", but I don't think the add-on should freeze forever. I waited several hours twice without success.

axlt2002 · 2017-11-02, 16:08

(2017-11-02, 15:12)Glottis Wrote: I tested the script.light.imdb.ratings.update-3.2.0patch1 three times now, but it did not finish. There's a lot of the following error in the log file, but it still goes on and updates a few episodes until it finally freezes completely. (Again with the following message

Code:
13:54:09.960 T:545688908000 ERROR: Unhandled exception in thread started by 13:54:09.960 T:545688908000 ERROR: <function thread_parse_IMDb_page at 0x7eeac4e938> 13:54:09.960 T:545688908000 ERROR: Traceback (most recent call last): 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/update_main.py", line 36, in thread_parse_IMDb_page 13:54:09.960 T:545688908000 ERROR: (IMDb, statusInfo) = get_IMDb_ID(dType, TVDB) 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/tvdb_scraper.py", line 30, in get_IMDb_ID 13:54:09.960 T:545688908000 ERROR: response = episode.info() 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/support/tvdbsimple/episode.py", line 53, in info 13:54:09.960 T:545688908000 ERROR: response = self._GET(path) 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/support/tvdbsimple/base.py", line 150, in _GET 13:54:09.960 T:545688908000 ERROR: return self._request('GET', path, params=params, cleanJson=cleanJson) 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/support/tvdbsimple/base.py", line 133, in _request 13:54:09.960 T:545688908000 ERROR: headers=self._headers) 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.module.requests/lib/requests/api.py", line 56, in request 13:54:09.961 T:545688908000 ERROR: return session.request(method=method, url=url, **kwargs) 13:54:09.961 T:545688908000 ERROR: File "/storage/.kodi/addons/script.module.requests/lib/requests/sessions.py", line 488, in request 13:54:09.961 T:545688908000 ERROR: resp = self.send(prep, **send_kwargs) 13:54:09.961 T:545688908000 ERROR: File "/storage/.kodi/addons/script.module.requests/lib/requests/sessions.py", line 609, in send 13:54:09.961 T:545688908000 ERROR: r = adapter.send(request, **kwargs) 13:54:09.961 T:545688908000 ERROR: File "/storage/.kodi/addons/script.module.requests/lib/requests/adapters.py", line 487, in send 13:54:09.962 T:545688908000 ERROR: raise ConnectionError(e, request=request) 13:54:09.962 T:545688908000 ERROR: requests.exceptions 13:54:09.962 T:545688908000 ERROR: . 13:54:09.962 T:545688908000 ERROR: ConnectionError 13:54:09.962 T:545688908000 ERROR: : 13:54:09.962 T:545688908000 ERROR: HTTPSConnectionPool(host='api.thetvdb.com', port=443): Max retries exceeded with url: /episodes/4845979 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7ee8618890>: Failed to establish a new connection: [Errno 113] No route to host',))

I can see why it says "Max retries exceeded", but I don't think the add-on should freeze forever. I waited several hours twice without success.

Thanks a lot Glottis to report back. Pity that the error is still present...

I'm actually busy with some other improvements/modifications so I leave the floor to dziobak for this issue (especially because he is definitely more expert than me on such stuffs!).

Hope we will be able to solve the issue soon.

Alex

dziobak · 2017-11-02, 16:43

(2017-11-02, 15:12)Glottis Wrote: I tested the script.light.imdb.ratings.update-3.2.0patch1 three times now, but it did not finish. There's a lot of the following error in the log file, but it still goes on and updates a few episodes until it finally freezes completely. (Again with the following message

Code:
13:54:09.960 T:545688908000 ERROR: Unhandled exception in thread started by 13:54:09.960 T:545688908000 ERROR: <function thread_parse_IMDb_page at 0x7eeac4e938> 13:54:09.960 T:545688908000 ERROR: Traceback (most recent call last): 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/update_main.py", line 36, in thread_parse_IMDb_page 13:54:09.960 T:545688908000 ERROR: (IMDb, statusInfo) = get_IMDb_ID(dType, TVDB) 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/core/tvdb_scraper.py", line 30, in get_IMDb_ID 13:54:09.960 T:545688908000 ERROR: response = episode.info() 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/support/tvdbsimple/episode.py", line 53, in info 13:54:09.960 T:545688908000 ERROR: response = self._GET(path) 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/support/tvdbsimple/base.py", line 150, in _GET 13:54:09.960 T:545688908000 ERROR: return self._request('GET', path, params=params, cleanJson=cleanJson) 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.light.imdb.ratings.update/resources/support/tvdbsimple/base.py", line 133, in _request 13:54:09.960 T:545688908000 ERROR: headers=self._headers) 13:54:09.960 T:545688908000 ERROR: File "/storage/.kodi/addons/script.module.requests/lib/requests/api.py", line 56, in request 13:54:09.961 T:545688908000 ERROR: return session.request(method=method, url=url, **kwargs) 13:54:09.961 T:545688908000 ERROR: File "/storage/.kodi/addons/script.module.requests/lib/requests/sessions.py", line 488, in request 13:54:09.961 T:545688908000 ERROR: resp = self.send(prep, **send_kwargs) 13:54:09.961 T:545688908000 ERROR: File "/storage/.kodi/addons/script.module.requests/lib/requests/sessions.py", line 609, in send 13:54:09.961 T:545688908000 ERROR: r = adapter.send(request, **kwargs) 13:54:09.961 T:545688908000 ERROR: File "/storage/.kodi/addons/script.module.requests/lib/requests/adapters.py", line 487, in send 13:54:09.962 T:545688908000 ERROR: raise ConnectionError(e, request=request) 13:54:09.962 T:545688908000 ERROR: requests.exceptions 13:54:09.962 T:545688908000 ERROR: . 13:54:09.962 T:545688908000 ERROR: ConnectionError 13:54:09.962 T:545688908000 ERROR: : 13:54:09.962 T:545688908000 ERROR: HTTPSConnectionPool(host='api.thetvdb.com', port=443): Max retries exceeded with url: /episodes/4845979 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7ee8618890>: Failed to establish a new connection: [Errno 113] No route to host',))

I can see why it says "Max retries exceeded", but I don't think the add-on should freeze forever. I waited several hours twice without success.

When one of threads is crashing on error main program don't know this and not start new thread in place, add-on stuck in endless loop when all threads are crashed.
After my last fix is OK with connections to IMDb, but now problem is with TVDb querrying (same nature of curious "No route to host" error - that means "no routing" or "no connection to net where destination is"). Do you have stable Internet connection?

Glottis · 2017-11-02, 17:01

(2017-11-02, 16:43)dziobak Wrote: Do you have stable Internet connection?

The connection should definitely be stable. It's all wired and 400 MBit/s via cable. I guess TheTVDB temporarily blocks my IP address after that many requests, but I've got no proof.

dziobak · (This post was last modified: 2017-11-02, 17:36 by dziobak.)

(2017-11-02, 17:01)Glottis Wrote:
(2017-11-02, 16:43)dziobak Wrote: Do you have stable Internet connection?

The connection should definitely be stable. It's all wired and 400 MBit/s via cable. I guess TheTVDB temporarily blocks my IP address after that many requests, but I've got no proof.

I sent you next fix for that.

If this is problem with temporary block of TVDb, then you are first person with this. Second - in that case it should be something like socket timeout or HTTP reject, but error is "no route to host".
Try simple thing: ping -t thetvdb.com