(2020-01-14, 09:19)axlt2002 Wrote: (2020-01-13, 22:58)zerocool_ie Wrote: hi @axlt2002
is there a way that I can restrict concurrent threads to 1? Or add a X second pause between scrapes?
I'm still trying to get it to run for more than a few hundred movies at a time.... still the same error messages accessing IMDb after a few minutes.
Maybe slowing down my requests will allow me to go further...?
Thanks!
Hi @zerocool_ie, thanks for your post.
I didn't understand if the issue is the same you experienced some time ago (see this post). Would you be so kind to post the update.log just to let me check the error messages? How big is the library you are trying to update? Is the error happening almost at the same time? I'm just asking because I would like to understand if this issue does affect also other users; it is something that was not reported before. so I take the occasion to ask users to report about the same issue in case (e.g. @Karellen all ok on your side?).
After understanding the error, I will evaluate the reintroduce the number of threads in the settings (that by the way is present in version 3.5.2...@zerocool_ie, may be you can give it a try and let me know).
Let's stay updated!
hi,
I did a movies-only (3439 movies) run using v3.5.2 with threads set as 4.
It did finish the job, v4 would always get stuck and I'd have to reboot my pi.
'Error accessing IMDb site' occurs multiple times per minute, but only starts showing in the logs 12 minutes after starting the job.
Minimal logs below... I'll keep checking things on my end.
thanks,
-Z
PS. Can you please allow reducing threads down to 1 as you integrate that setting into 4.x? My use case is that I schedule for 1AM and let it run while I sleep, so duration of job is not important.
kodi.log
sql:
2020-01-16 17:33:41.713 T:1807721344 DEBUG: [Light IMDb Ratings Update] - Ratings update for movies started
.....
2020-01-16 17:42:21.470 T:1298113408 DEBUG: [Light IMDb Ratings Update] - Error accessing IMDb site (<urlopen error [Errno 113] No route to host>)
....
2020-01-16 17:42:21.505 T:1323291520 DEBUG: [Light IMDb Ratings Update] - Error accessing IMDb site (<urlopen error [Errno 113] No route to host>)
....
2020-01-16 18:13:40.462 T:1807721344 DEBUG: [Light IMDb Ratings Update] - Ratings update for movies finished
sql:
pi:~ # grep 'Starting updates for movie' /storage/.kodi/temp/kodi.log | wc -l = 3252
pi:~ # grep 'Finished updates for movie' /storage/.kodi/temp/kodi.log | wc -l = 3252
pi:~ # grep 'Update finished (no update)' /storage/.kodi/temp/kodi.log | wc -l = 81
pi:~ # grep 'urlopen error' /storage/.kodi/temp/kodi.log | wc -l = 595
update.log
sql:
pi:~ # grep 'Error accessing' update.log | wc -l = 106
sql:
Light IMDb Ratings Update log file - 2020-01-16 17:30:31
--> Ratings update for movies
13 Sins:
Method get_IMDb_page: http://akas.imdb.com/title/tt2059171/ -> Error accessing IMDb site (<urlopen error [Errno 113] No route to host>)
A Good Day to Die Hard:
Method get_IMDb_page: http://akas.imdb.com/title/tt1606378/ -> Error accessing IMDb site (<urlopen error [Errno 113] No route to host>)
A Lot Like Love:
Method get_IMDb_page: http://akas.imdb.com/title/tt0391304/ -> Error accessing IMDb site (<urlopen error [Errno 113] No route to host>)
..................
Her Alibi:
Method get_IMDb_page: http://akas.imdb.com/title/tt0097500/ -> Error accessing IMDb site (<urlopen error [Errno 113] No route to host>)
The Big Bus:
Method get_IMDb_page: http://akas.imdb.com/title/tt0074205/ -> Error accessing IMDb site (<urlopen error [Errno 113] No route to host>)
Whitney Cummings: I Love You:
Method get_IMDb_page: http://akas.imdb.com/title/tt3794552/ -> Error accessing IMDb site (<urlopen error [Errno 113] No route to host>)
And please remember to post the update.log in case.
