[izprtxqkft]

Added (lengthy) modules for Trakt (trending, popular, watchlist, custom lists, collection) and TMDB scraping for Movie content, speed tests are great taking less than 3 seconds to scrape and display 100 movies on initial listing.
Employing an sqlite3 cache the subsequent listings of already scraped content taking on average 0.5-0.8 seconds for the same 100 movies.

But, I wrote the modules on my Linux development machine using pyCurl for the speed increase only to realize this is not part of Kodi and is not available on Android platforms (potentially not Windows either, untested).
So that will need to be re-written using requests, hoping it will be comparable in speed to pyCurl but I have my doubts.
Potentially I will include both and on the unavailability of pyCurl on a platform just fallback to requests - if I cannot get requests to the same performance.

Are you using pyCurl to pull images or metadata ?  Both can be pulled via normal urllib commands.  Here's an example of image and xml fetching / parsing that I use for TMDB and it is lightening fast.


Thanks,

Jeff

just metadata, currently setting images with list item SetArt but i intend to utilize addAvailableArtwork in the future

urllib3 might be a good alternative, thanks for the suggestion


here's the chart i found, my own tests seem to indicate it is accurate

[jbinkley60]

just metadata, currently setting images with list item SetArt but i intend to utilize addAvailableArtwork in the fu
urllib3 might be a good alternative, thanks for the suggestion


here's the chart i found, my own tests seem to indicate it is accurate

I presume this chart is on a local LAN and not across the WAN.  If so, the WAN will be your slowdown.  I just ran a check of fetching 500 images / image checks from TMDB across the WAN.  I averaged 248 checks /min or about 4 a second.  I suspect curl would be slower but it also depends upon whether you are using HTTP 1.1 and higher and things like connection pooling and similar.  One downside of urllib, regardless of version, is that you end up doing more error handling.   For your local use, urllib3 is likely a good alternative.


Jeff

[izprtxqkft]

urllib3 with caching seems fast enough

i'll start making calls more universal so i can include both as a module and use pycurl if available otherwise urllib3

the caching is just a decorator that catches and returns cache if it exists so it can be universal

[izprtxqkft]

with that i should be on track to continue adding features

i created another decorator function on top of my request function to fallback to urllib3 and behave the same way with the same function parameters, with the same return values

the functions need an additional couple lines to determine which is being used but otherwise function as they did

using except ImportError on pycurl so it can fail gracefully and instead import urllib3 HTTPSConnectionPool


here's the wrapper if anyone is curious

the original function is request at the bottom, cachereq sits on top of both to continue caching as it did

the major differences were headers to pycurl use a list of "header: value" where urllib3 uses a dict, urllib3 also uses a dict on response headers where pycurl is raw so i adopted the dict format to both libraries as i feel its better anyhow

the http status was also different, pycurl has a function to retrieve it where liburl3 does not so now both just set it on the passed class of either Curl() or HTTPSConnectionPool depending on what's in use

also, pycurl uses the full http address "https://host/path?params" where HTTPSConnectionPool just uses the path and params '/path?params' so i wrote a quick function to extract the path for urllib3 when used

View Spoiler

python:
def getpath(url):

    url = url.replace("https://","")

    url = url.split("/",1)[1]

    return "/"+url



def urllibrequest(func):

    def wrapper(*args, **kwargs):

        global defaultheaders

        if usePycurl:

            if 'headers' in kwargs:

                kwargs['headers'] = ["%s: %s" % (key,value) for key, value in kwargs['headers'].items()]

            else:

                kwargs['headers'] = ["%s: %s" % (key,value) for key, value in defaultheaders.items()]

            return func(*args, **kwargs)

        else:

            f = getattr(args[0],'urlopen')

            if 'headers' in kwargs:

                headers = kwargs['headers']

            else:

                headers = defaultheaders

            if 'postdata' in kwargs:

                method = "POST"

                if type(kwargs['postdata'])==dict:

                    postdata = json.dumps(kwargs['postdata'])

                else:

                    postdata = kwargs['postdata']

            else:

                method = "GET"

                postdata = ''

            response = f(method, getpath(args[1]), headers=headers, body=postdata)

            setattr(args[0],'status',response.status)

            setattr(args[0],'headers',response.getheaders())

            if 'raw' in kwargs and kwargs['raw']:

                return response.data

            else:

                try:

                    return json.loads(response.data)

                except:

                    return response.data

    return wrapper



#@cachereq

@urllibrequest

def request(c, url, headers='', postdata=False, verbose=0, raw=False):

[jbinkley60]

urllib3 with caching seems fast enough

i'll start making calls more universal so i can include both as a module and use pycurl if available otherwise urllib3

the caching is just a decorator that catches and returns cache if it exists so it can be universal

You've sufficiently motivated me to look at moving the Mezzmo Kodi Addon for Kodi 19-21 from urllib to urllib3, now that I see there is a Kodi wrapper for it.  I'll begin some testing to look at performance.  I have some good historical performance stats to compare.


Jeff

[izprtxqkft]

right on, happy to know i motivated someone

i'm also using jurialmunkey's infotagger from https://github.com/jurialmunkey/script.m...infotagger

but not as-is, using this part of it with some expansion on the keys it accepts so that it can accept data directly received from tmdb - https://github.com/jurialmunkey/script.m...tem.py#L90

so before i release any more code i need to find out what the etiquette for such a thing would be, obviously at minimum i will include a credit and a link back

[izprtxqkft]

thinking on this i decided the best etiquette would be to fork the code and make the changes so they can be public

https://github.com/wlincpdjhbxz/script.m...infotagger

[Fuzzard]

Out of curiosity, why not PR to the upstream repo? Im sure jurialmunkey would be open to accepting improvements

[izprtxqkft]

Out of curiosity, why not PR to the upstream repo? Im sure jurialmunkey would be open to accepting improvements

mainly, it only improves the code when used in conjunction with mine, original use of the code the data is transformed to meet the requirements of the infotagger which is what they have decided was best for them

the primary purpose is to ensure all proper lineage and credit is maintained for the code, i could not have written better if i tried and do not want anyone thinking i am trying to steal it (or any other code)

[Fuzzard]

Well it is a script plugin that people use, so maybe your changes might be useful for others who use the addon.

All good either way, was just wondering.

[izprtxqkft]

Well it is a script plugin that people use, so maybe your changes might be useful for others who use the addon.

All good either way, was just wondering.

thank you

i am not trying to be standoffish i just fail see the value of the addition to others, the nudge to contribute a PR is appreciated

[izprtxqkft]

new testing version available, lots of changes made

see the first post for the link and the changelog - https://forum.kodi.tv/showthread.php?tid=374003

[jbinkley60]

- Automated bookmark management through a Library Monitor service, when removing imported items from Kodi natively through "Manage" the monitoring service will remove the corresponding bookmark.

FYI, I am not sure this is necessary with newer versions of the Kodi video database.  There is a delete trigger that fires and removes bookmarks when the underlying file is removed from the files table:

Delete File Trigger:
CREATE TRIGGER delete_file AFTER DELETE ON files

FOR EACH ROW BEGIN DELETE FROM bookmark

WHERE idFile=old.idFile; DELETE FROM settings WHERE idFile=old.idFile;

DELETE FROM stacktimes WHERE idFile=old.idFile;

DELETE FROM streamdetails WHERE idFile=old.idFile; END

Since you aren't doing any direct inserts and such this addon should work with SQLite or MySQL, correct ?


Thanks,

Jeff

[izprtxqkft]

FYI, I am not sure this is necessary with newer versions of the Kodi video database.  There is a delete trigger that fires and removes bookmarks when the underlying file is removed from the files table:

let me clarify that, "bookmarks" refers to it's own bookmarks.json file - when importing something to kodi it creates an entry into bookmarks.json in addon_data

these bookmarks are what's presented as "content" to kodi, if you remove a movie from kodi but the bookmark remains it will get rescanned into kodi the next library update

which is annoying to say the least, "i just removed that movie why is it back?", so there was a method in the addon to remove the bookmark and subsequently the kodi entry

this reverses that action and instead of when removing bookmark also remove kodi library entry - it will remove the bookmark on kodi library entry removal, and i dont have to be removing things from kodi via jsonrpc calls

 

Since you aren't doing any direct inserts and such this addon should work with SQLite or MySQL, correct ?

yes it should work fine with mysql since there is no direct database interaction other than with it's own cache database which is of no consequence to kodi

[jbinkley60]

let me clarify that, "bookmarks" refers to it's own bookmarks.json file - when importing something to kodi it creates an entry into bookmarks.json in addon_data

these bookmarks are what's presented as "content" to kodi, if you remove a movie from kodi but the bookmark remains it will get rescanned into kodi the next library update

which is annoying to say the least, "i just removed that movie why is it back?", so there was a method in the addon to remove the bookmark and subsequently the kodi entry

this reverses that action and instead of when removing bookmark also remove kodi library entry - it will remove the bookmark on kodi library entry removal, and i dont have to be removing things from kodi via jsonrpc calls

Got it.  Makes sense.  It doesn't help that folks sometimes call them bookmarks and other times resume pointers.  That adds to the confusion.


Thanks,

Jeff