Kodi Community Forum
script.module.urlresolver development - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=26)
+--- Thread: script.module.urlresolver development (/showthread.php?tid=105707)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28


- t0mm0 - 2011-09-17

k_zeon Wrote:Hi t0mm0.
just tried the new version and

addon.add_video_item({'url':'http://www.megavideo.com/?v=LYWNYM1J'},
{'title': 'megavideo'})

fails . cannot resolve url

what else is required

i assume you mean the videoid branch? i am currently porting the plugins to this version, at the moment in github there is only putlocker as mentioned earlier.

t0mm0


- k_zeon - 2011-09-17

t0mm0 Wrote:i assume you mean the videoid branch? i am currently porting the plugins to this version, at the moment in github there is only putlocker as mentioned earlier.

t0mm0

sorry, did not notice that. look forward to your update


- t0mm0 - 2011-09-17

k_zeon Wrote:sorry, did not notice that. look forward to your update

just for you i just pushed a version of the megaupload plugin that works in the videoid branch with an updated test plugin showing that it works Wink

working my way through the rest of it slowly but surely!

t0mm0


- k_zeon - 2011-09-17

t0mm0 Wrote:just for you i just pushed a version of the megaupload plugin that works in the videoid branch with an updated test plugin showing that it works Wink

working my way through the rest of it slowly but surely!

t0mm0

Excellent work. I wish i had your skill. I am just making my first addon and it works , but not what you would call good.

Many thanks


- k_zeon - 2011-09-17

t0mm0
in your example links , if i click a link that has been removed i get a script error.

is there an easy solution to pop up a dialog box and let user know video has been taken down and then return to previous menu.

answer when you have time.

tks


- t0mm0 - 2011-09-17

k_zeon Wrote:t0mm0
in your example links , if i click a link that has been removed i get a script error.

is there an easy solution to pop up a dialog box and let user know video has been taken down and then return to previous menu.

answer when you have time.

tks

can you give me an example? in theory at least you should never get a script error but i probably missed some cases Wink

edit: ah is this still megaupload? if so there isn't much you can do at the moment, the megaupload code really needs rewriting to make that possible, at the moment it is just completely stolen from anarchintosh and wasn't really designed t be used this way Wink

all plugins are now ported to videoid branch and seem to be working (except for the 2gbhosting one which was already broken - i think the javascript needs to be unpacked properly as the regex in use was a bit too precarious. maybe my unpacker from vidxden can be used/modified and put somewhere where any plugin can use it?)

i think i'll leave the docs till tomorrow as it is looking fairly daunting (i have to substantially edit the plugin tutorial too) Eek

t0mm0


- k_zeon - 2011-09-17

yes it was a megaupload.
I did see a little snippet from plugin.video.fpt that has code that checks.
Code below from One def

def PUTLOC(url,name):
#login()
link1 = geturl(url)
link2 = redirect(url)
try:
hash=re.compile('type="hidden" value="(.+?)" name="hash"').findall(link1)[0]
except:
dialog = xbmcgui.Dialog()
ok = dialog.ok("FastPassTv",'The file has been removed due to copyright.')
return

values = {'hash': hash, 'confirm':'Continue as Free User'}
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
headers = { 'User-Agent' : user_agent}
cookiejar = cookielib.LWPCookieJar()
cookiejar = urllib2.HTTPCookieProcessor(cookiejar)
opener = urllib2.build_opener(cookiejar)
urllib2.install_opener(opener)
data = urllib.urlencode(values)
req = urllib2.Request(link2, data, headers)
response = urllib2.urlopen(req)
link = response.read()
code = re.compile("stream=(.+?)'").findall(link)
req = urllib2.Request('http://www.putlocker.com/get_file.php?stream='+code[0])
req.add_header('User-Agent', user_agent)
response = urllib2.urlopen(req)
link=response.read()
finalurl = re.compile('<media:content url="(.+?)"').findall(link)[0]
Play(finalurl)


Also just wanted to know if you had time to update the 45 sec for megaupload that broke last time i asked


- t0mm0 - 2011-09-17

k_zeon Wrote:yes it was a megaupload.
I did see a little snippet from plugin.video.fpt that has code that checks.
yeah i know how to do it (i have error handling stuff everywhere i can think of in my code just take a look!), just haven't had time to rewrite anarchintosh's code as it is basically working and there is still lots of other stuff to do and i think it will be quite a big change to that code for a relatively small gain. i will get to it eventually if nobody else does.
k_zeon Wrote:Also will you/have you updated the 45 sec for megaupload that broke last time i asked
it is all working for me in the videoid branch so if it's not working for you let me know what the problem is with a log or something. i haven't moved it yet, and there is also a small patch from anarchintosh (you can see in the pull requests on github) to be applied but that is just for 'perfectionism' as he puts it Wink so doesn't change it much.

t0mm0


- k_zeon - 2011-09-18

is it possible to have 2 sets of your modules and addons

ie script.module.urlresolver_old & script.module.urlresolver

then in other programs that have not updated to the new codes ads the _old in the imports etc.

that way if an addon has not yet updated to use the new codes then they would still work ( with a little help)

just my little work around...
tks


- t0mm0 - 2011-09-18

k_zeon Wrote:is it possible to have 2 sets of your modules and addons

ie script.module.urlresolver_old & script.module.urlresolver

then in other programs that have not updated to the new codes ads the _old in the imports etc.

that way if an addon has not yet updated to use the new codes then they would still work ( with a little help)

just my little work around...
tks

the short answer is no.

this stuff isn't released yet. once it is, i will maintain backwards compatibility as much as possible so it shouldn't be an issue. i am trying to make sure we are happy with the basics before it is released in order to avoid that problem.

t0mm0.


- Eldorado - 2011-09-20

Hey t0mm0, any ideas on how to handle a situation where you only have a host name and no .com or .net etc.

eg. http://www.quicksilverscreen.im/details?c=86736&fc=1&e=1

For each source they only list the name of the host, so I would need to figure out the host and apply the correct suffix to it

I was looking to see if resolver could be changed to handle it and I think it would be ok, but wanted to talk it out

For putlocker maybe we could just do this - remove the '.com' for hosted_media_file

Code:
def valid_url(self, hosted_media_file):
        return (re.match('http://(www.)?(putlocker|sockshare).com/(file|embed)' +
                         '/[0-9A-Z]+', hosted_media_file.get_url()) or
                hosted_media_file.get_host().find('putlocker') > -1 or
                hosted_media_file.get_host().find('sockshare') > -1)

get_url() would need a change as well, might be a bit trickier to keep it clean, just an idea:

Code:
def get_url(self, host, media_id):
        if not host.startswith('www.'):
            host = 'www.' + host
        if not host.endswith('.com'):
            host = host + '.com'
        return 'http://%s/file/%s' % (host, media_id)



- t0mm0 - 2011-09-20

Eldorado Wrote:Hey t0mm0, any ideas on how to handle a situation where you only have a host name and no .com or .net etc.

eg. http://www.quicksilverscreen.im/details?c=86736&fc=1&e=1

For each source they only list the name of the host, so I would need to figure out the host and apply the correct suffix to it

I was looking to see if resolver could be changed to handle it and I think it would be ok, but wanted to talk it out

For putlocker maybe we could just do this - remove the '.com' for hosted_media_file

Code:
def valid_url(self, hosted_media_file):
        return (re.match('http://(www.)?(putlocker|sockshare).com/(file|embed)' +
                         '/[0-9A-Z]+', hosted_media_file.get_url()) or
                hosted_media_file.get_host().find('putlocker') > -1 or
                hosted_media_file.get_host().find('sockshare') > -1)

get_url() would need a change as well, might be a bit trickier to keep it clean, just an idea:

Code:
def get_url(self, host, media_id):
        if not host.startswith('www.'):
            host = 'www.' + host
        if not host.endswith('.com'):
            host = host + '.com'
        return 'http://%s/file/%s' % (host, media_id)

yes, seems reasonable to me. i'd actually thought about doing that, then thought it was too ambiguous (in case there were hosters with the same name but different TLD) but i now think you are right and i was being overly worried and this sort of thing will be fine. i think we just need to make sure that the acceptable options for the host argument are documented for each plugin to make it clear what is going on.

i think for many plugins, the host part of get_url() can actually be fixed anyway (eg, for megaupload, the url is always 'http://www.megaupload.com/?d=' + media_id) which actually makes it much easier than the putlocker one.

i'm going to try and finish tidying up and documenting the videoid branch tomorrow. thanks for continuing to suggest stuff - i'm sure we'll end up with a more useful module the more input there is from more devs. also i'm sorry to everyone who still has outstanding pull requests, i'll start merging them again once i've done the videoid merge and i haven't forgotten them, honest Wink it has just taken longer than i thought to sort this big change out.

t0mm0


- Eldorado - 2011-09-20

Sounds good!

Found one bug using filter_urls()

Code:
14:01:41 T:2276   ERROR: Error Type: <type 'exceptions.NameError'>
14:01:41 T:2276   ERROR: Error Contents: global name 'UrlResolver' is not defined
14:01:41 T:2276   ERROR: Traceback (most recent call last):
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\plugin.video.projectfreetv\default.py", line 101, in <module>
                                                validsources = urlresolver.filter_urls(links.keys())
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.urlresolver\lib\urlresolver\__init__.py", line 89, in filter_urls
                                                imp = find_resolver(url)
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.urlresolver\lib\urlresolver\__init__.py", line 136, in find_resolver
                                                for imp in UrlResolver.implementors():
                                            NameError: global name 'UrlResolver' is not defined

Should this be just lower case?

Code:
for imp in UrlResolver.implementors():
        if imp.valid_url(web_url):
            return imp
    return False



- t0mm0 - 2011-09-20

Eldorado Wrote:Sounds good!
just doing it now, should be committed soon....
Eldorado Wrote:Found one bug using filter_urls()

Code:
14:01:41 T:2276   ERROR: Error Type: <type 'exceptions.NameError'>
14:01:41 T:2276   ERROR: Error Contents: global name 'UrlResolver' is not defined
14:01:41 T:2276   ERROR: Traceback (most recent call last):
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\plugin.video.projectfreetv\default.py", line 101, in <module>
                                                validsources = urlresolver.filter_urls(links.keys())
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.urlresolver\lib\urlresolver\__init__.py", line 89, in filter_urls
                                                imp = find_resolver(url)
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.urlresolver\lib\urlresolver\__init__.py", line 136, in find_resolver
                                                for imp in UrlResolver.implementors():
                                            NameError: global name 'UrlResolver' is not defined

Should this be just lower case?

Code:
for imp in UrlResolver.implementors():
        if imp.valid_url(web_url):
            return imp
    return False

no this is correct, the problem is all the plugins are actually implementing NewUrlResolver now as that helped me test while only some of the plugins were ported. they will be switced back soon. i also haven't looked at the urlresolver functions at all since the change and some of them may need to be moved around or may no longer even be needed. i'm working on it now.... (well actually i'm just trying to figure out how i managed to break putlocker at the moment!)

t0mm0

edit:
just pushed commits which do the host name thing as discussed earlier, change back to using UrlResolver rather than NewUrlResolver and fix putlocker which broke due to a change in their site.


- Eldorado - 2011-09-21

Very good so far! The new HostedMediaFile and the functionality around it work well, and I like the changes to choose_source()

Found an issue with filter_urls()

I'm assuming it can also be used to filter a list of hosts, as in not a url but simply a host name?

Code:
23:00:30 T:6548   ERROR: Error Type: <type 'exceptions.TypeError'>
23:00:30 T:6548   ERROR: Error Contents: valid_url() takes exactly 3 arguments (2 given)
23:00:30 T:6548   ERROR: Traceback (most recent call last):
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\plugin.video.quicksilverscreen\default.py", line 78, in <module>
                                                test = urlresolver.filter_urls(host)
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.urlresolver\lib\urlresolver\__init__.py", line 89, in filter_urls
                                                imp = find_resolver(url)
                                              File "C:\Users\M33282\AppData\Roaming\XBMC\addons\script.module.urlresolver\lib\urlresolver\__init__.py", line 137, in find_resolver
                                                if imp.valid_url(web_url):
                                            TypeError: valid_url() takes exactly 3 arguments (2 given)
23:00:30 T:6548    INFO: -->End of Python script error report<--

I've updated my addons in the separate branch, all working well so far!