v18 memory leak on scraper 404.
#1
Hi,

My kodi installation currently DoSes the machine it runs on by consuming all available memory. I tracked it down to scraping (not sure which scraper yet).the log is full of
Quote:2019-11-23 14:56:10.971 T:140078949037824   ERROR: Run: Unable to parse web site
2019-11-23 14:56:11.246 T:140078949037824   ERROR: CCurlFile::Open failed with code 404 for http://akas.imdb.com/title//:
Knowing that scrapers are a bunch of regexes run in order, I'd say this is more a Kodi bug than a scraper bug anyway.

edit: forgot the version.  Starting Kodi (18.5 Git:20191122)
Reply
#2
We always prefer full debug-enabled kodi.log files.
Also our crystal ball is currently in service.
Reply
#3
(2019-11-23, 16:56)Klojum Wrote: We always prefer full debug-enabled kodi.log files.
Also our crystal ball is currently in service.

https://hastebin.com/ifepojetaj.sql

It gets stuck in an endless loop after the 404.
Reply
#4
Looks like your weather service is having hiccups at timestamp 2019-11-23 16:02:00.203 T:139907395327744
If you have multiple 404 problems you may have a DNS issue.
Reply
#5
(2019-11-23, 18:33)Klojum Wrote: Looks like your weather service is having hiccups at timestamp 2019-11-23 16:02:00.203 T:139907395327744
If you have multiple 404 problems you may have a DNS issue.

Yes it is, and no, it isn't the source of the issue. It was disabled, and now is uninstalled, and the issue remains.
It really is linked to library scanning: I can trigger it by force scanning a specific file. Namely "Antropocén Epocha člověka", using the csfd scraper (from https://kodi-czsk.github.io/repository/ )
Scraping starts at line 860 in the log

And last: A DNS issue shouldn't make the memory usage of Kodi run amok...
Reply
#6
(2019-11-23, 18:59)Serafean Wrote: And last: A DNS issue shouldn't make the memory usage of Kodi run amok...

Yep, but there are two more problems now. Normally our kodi paste site show banned repositories, your hastebin log file does not.
As it stands, both your installed repositories are on Kodi's banned list. So as per forum rules (wiki) support on this forum to you will be halted here for now.

Plenty of 3rd party repos/add-ons cause havoc on Kodi core functionality.
Start with a fresh installation, I think that leak will stop pretty soon, unless that weather addon really has bug.
Reply
#7
(2019-11-23, 19:19)Klojum Wrote:
(2019-11-23, 18:59)Serafean Wrote: And last: A DNS issue shouldn't make the memory usage of Kodi run amok...

Yep, but there are two more problems now. Normally our kodi paste site show banned repositories, your hastebin log file does not. 
xbmclogs.com keeps greeting me with a "Whoops, looks like something went wrong.", so no way for me to post there. Maybe you should update this thread to list the actual official paste site...

Your forum, your rules... keep your scraper bug in place.
Reply
#8
(2019-11-23, 19:50)Serafean Wrote: Your forum, your rules... keep your scraper bug in place.

Hmm... You just had to find a thread from 2015? Smile
That page has now been updated.

You also already posted 27 times now. This is above every Post Reply button in bold red text: "Do not post logs in the forum. Use https://paste.kodi.tv/ instead.".

A debug-enabled clean log file will be appreciated.
Reply

Logout Mark Read Team Forum Stats Members Help
memory leak on scraper 404.0