[t0mm0]

hi,

i have just pushed an initial version of urlresolver to github. it is NOT ready to be used and not in a repo, but just for developers at the moment.

it is inspired by the work previously done by anarchintosh and unbehagan and includes anarchintosh's megaupload code.

the api is currently much simpler than their version - i don't know if all the rest of the stuff is actually needed - comments welcome.

look in test.py to find out how to use it. it is currently not tested from within xbmc.

included are plugins for

• megaupload
  
• novamov
  
• putlocker
  
• seeon.tv (to show it can be used for live stuff too)
  

assorted stuff to do includes

• handle delays for megaupload
  
• find a way of managing settings/login details for plugins so that each addon doesn't have to do that
  
• lots more error handling
  
• extend api (what would be useful)
  
• write lots more plugins (best to hold off until api stabilised?)
  
• lots of nice documentation
  
• probably lots of other stuff i haven't thought of!
  

please let me know what you think, as if nobody is going to actually find this useful there is little point in continuing. all comments, ideas and pull requests are most welcome!

t0mm0

[t0mm0]

in case anyone is interested, work is progressing in git.

i shuffled the interfaces a bit, so there are now separate interfaces to implement if the plugin requires login (SiteAuth) or settings (PluginSettings)

i have added a method for resolver plugins to have settings (such as megaupload username and password for example) which will be global and so users will only have to set them once for all addons that use the resolver. i don't know if the method i've used is the best way of doing this or if it is sensible, but it seems to work. please take a look and let me know what you think.

delay code (from anarchintosh) has been added so that megaupload works properly.

it is now tested in xbmc and working nicely. i have written a basic addon to test it including some basic tubeplus.me support just to have something visible to test with but this is not yet committed to git as it is in a bit of a mess. will get that sorted soon so that people can easily see it in action.

there is now an extra plugin to resolve videos from tubeplus.me episode pages. if there is more than one playable link it offers a choice. it is interesting because the tubeplus plugin uses the other plugins to resolve links it finds which shows the flexability of the system.

the novamov plugin was fixed as it broke due to site changes.

developers please let me know what you think of the progress so far and don't be shy about telling me i've done it all wrong (preferably if you can tell me how to do it better or send a patch!)

t0mm0

ps. i promise i will add lots of comments and write some docs once it gets more stable

[mrdally204]

I'm surprised there is little interest on this. Seems like it could make everything a bit easier going down the road of future add ons. Keep up the work my friend, maybe this bump will find someone (:

[Temhil]

Yes, that is a great job you are doing and I am pretty sure it will be reuse in a lot of addons in the future.
Thank you for developing this module!

[anarchintosh]

hey t0mm0 this looks great! if you have any queries or questions you can PM me.

i'm currently sitting on an updated megaup resolver (i wrote a few months ago) that i'd like to commit to your repo, could you add me with write access? cheers.

EDIT: don't worry, i will submit a pull request

[t0mm0]

hi anarchintosh!

hey t0mm0 this looks great! if you have any queries or questions you can PM me.

i'm currently sitting on an updated megaup resolver (i wrote a few months ago) that i'd like to commit to your repo, could you add me with write access? cheers.

EDIT: don't worry, i will submit a pull request

glad you like it so far! i simplified the interface quite a bit from your previous stuff as i wasn't sure it was really needed to expose all that stuff to people - my plan is to make it as simple to use as possible to encourage more people to have a go at writing stuff.

i also added the settings stuff which i think might come in useful, and the priority stuff is half done (the idea is that there will be a setting that says 'use the resolver i've given the highest priority rather than ask me which link to play')

i'm also planning to change the login stuff so that the addons can call a 'log_in_to_everything()' type function when they first load rather than the resolvers logging in every time they are run as it does at the moment.

a pull request would be great - it's all still in a bit of flux at the moment. i have a few changes i've made locally and not pushed to github yet but nothing major at the moment. having your megaupload and delay popup code was really great to get it going quickly so i could focus on the structure and plugin stuff. (can i just confirm that code you've just put 'copyleft' on without a specific licence is ok to use as GPLv3?)

hopefully once i release a test addon using it then it will be easier to try out and there will be more feedback from people. and once the interfaces are stable it should be really easy for people to add loads of resolver plugins very quickly.

thanks,

t0mm0.

[cloom]

Hello t0mm0,

I have a bunch of ideas for plugins ongoing and I can't wait to have your module working to publish my plugins.
I can't find the file test.py you are talking about, is it in GIT?
Can you post it here please? I would like to give it a try.

Thank you.

[anarchintosh]

@t0mm0

yeah GPLv3 is great. (better than v2 IMO). my later code i began to put GPLv3 in.
i think you were right about a simple structure. me and unbehagan went with a slighty overkill interface previously.

'log in to everything' sounds good. could maybe even execute this on XBMC start up using autoexec.py (Dharma) / Services (Eden)

Bear in mind that some things (ie. hotfile premium) have an API which just asks for the user + pass in the request url.
For those type resolvers it would still be good to rig together a simple html form filling (see megaup resolver) login routine (just as a way of verifying whether the user + pass are vaild)


once it is basically up and running i reckon it will be very popular and quickly attract a lot of resolver contributers. it's also possible to manually translate the jdownloader java plugins' source code for things like fileserve/sonic etc

there were some problems with the updated megaup resolver... so i'll have to hsve another look at it.

[t0mm0]

Hello t0mm0,

I have a bunch of ideas for plugins ongoing and I can't wait to have your module working to publish my plugins.
I can't find the file test.py you are talking about, is it in GIT?
Can you post it here please? I would like to give it a try.

Thank you.

it is no longer in git as it is out of date.

at the moment, assuming you're using isPlayable on your listitems, you just need to do something like the following to resolve urls:

Code:

import urlresolver

stream_url = urlresolver.resolve(web_url)

if stream_url:
    xbmcplugin.setResolvedUrl(plugin_handle, True,
                              xbmcgui.ListItem(path=stream_url))
else:
    xbmcplugin.setResolvedUrl(plugin_handle, False,
                              xbmcgui.ListItem())

be aware that this module is currently in flux and stuff may change without warning! once it is more stable there will be proper docs available.

@t0mm0

yeah GPLv3 is great. (better than v2 IMO). my later code i began to put GPLv3 in.

cool, i prefer v3 too

i think you were right about a simple structure. me and unbehagan went with a slighty overkill interface previously.

'log in to everything' sounds good. could maybe even execute this on XBMC start up using autoexec.py (Dharma) / Services (Eden)

interesting idea, although we would have to think about what would happen when people leave xbmc running (how often do these sites need new login cookies?)

Bear in mind that some things (ie. hotfile premium) have an API which just asks for the user + pass in the request url.
For those type resolvers it would still be good to rig together a simple html form filling (see megaup resolver) login routine (just as a way of verifying whether the user + pass are vaild)

good plan

once it is basically up and running i reckon it will be very popular and quickly attract a lot of resolver contributers. it's also possible to manually translate the jdownloader java plugins' source code for things like fileserve/sonic etc

yes i hope to do some solid work on this over the next few days as i have some time off work, and i hope once it is stable people will contribute resolver plugins as you say (it is pretty easy and i've tried to make it so that you need to write as little code as possible). like you say there is loads of code for various sites floating around that can be adapted, like jdownloader, pyload, or even get-flash-videos if anyone can understand that perl stuff

it should make some of the requests for help with addons that have been popping up on the forums really easy too.

there were some problems with the updated megaup resolver... so i'll have to hsve another look at it.

cool, no problem, the existing stuff seems to work well. the only thing i changed was the method names (you added __ in front of the public methods which meant they are 'private' and so not (easily) accessible from outside the module)

t0mm0

[rogerthis]

Would it be possible to add a flag for working as each module could stop working at any time and need to get updated?

So in my addon I could have something like this

Code:

if urlresolver.novamov.working == true and urlresolver.novamov.exists == true
    novamov=re.compile('href="http://www.novamov.com/video/(.+?)"').findall(html)

I could have all links ready to be scrapped, but it would only show working links.

[t0mm0]

hi,

Would it be possible to add a flag for working as each module could stop working at any time and need to get updated?

So in my addon I could have something like this

Code:

if urlresolver.novamov.working == true and urlresolver.novamov.exists == true
    novamov=re.compile('href="http://www.novamov.com/video/(.+?)"').findall(html)

This would enable my addon to only scrap working links.

i don't think there is any need for that. the module would have to be updated to say it wasn't working, and in that case it may as well just get fixed, or if it was too hard to fix quickly then valid_url() could be changed to just return False. that way the addon doesn't have to care (or add the extra checking code).

as it stands, addons don't directly access the resolver plugins, they just send a url to resolve() and it automatically picks the correct resolver to use (soon using user changeable priorities which will define the order that resolver plugins are tried). i think this makes it more flexible and means there is less code required in the addons themselves (i'm trying to make both the addons and the resolver plugins as simple as possible so they are easier to write, and any complicated stuff should be in urlresolver)

there is already filter_urls() that takes a list of urls and only returns those that can be resolved, and i'm thinking of adding another method that simply takes raw html and finds all the resolvable urls (so you wouldn't have to do any scraping at all). of course there is no way to tell if the files actually exist until you visit each hoster site....

thoughts anyone? (assuming any of that made sense!)

t0mm0

[rogerthis]

So, a page like this http://www1.zmovie.tv/movies/view/captai...enger-2011 that has multiple links, it will find all the links. It wouldn't really want to resolve the final playable url until you select which one you wanted. That would be too much work (too slow), wouldn't it?

Also there are sites like this http://www.fastpasstv.ms/tv/flashpoint/s...episode-4/ that have redirects to itself before you get to the video site url. Would it be able to handle this?

[t0mm0]

So, a page like this http://www1.zmovie.tv/movies/view/captai...enger-2011 that has multiple links, it will find all the links. It wouldn't really want to resolve the final playable url until you select which one you wanted. That would be too much work (too slow), wouldn't it?

filter_urls() just takes a list of urls scraped from the page and checks them against the valid_url() method of the resolver plugins. all this does is checks to see if any of the resolver plugins say they can resolve those urls. it doesn't actually do any resolving until you call resolve() so this is fast. obviously it doesn't know whether the files actually exist or not, just whether there is a resolver plugin that says it can resolve urls from that particular host site.

Also there are sites like this http://www.fastpasstv.ms/tv/flashpoint/s...episode-4/ that have redirects to itself before you get to the video site url. Would it be able to handle this?

those links look suspiciously base64-ish...

anyway, you could either write a simple resolver plugin that accepted fastpasstv redirect links (like the tubeplus.me one in git), or work out the real url in the addon, or possibly you could do a HEAD request and see if there is a redirect in place (which would be faster than grabbing the whole page)

well, in case anyone thinks i've been slacking i've been busy splitting out a load of generic stuff from the test addon and the urlresolver module into another module which provides a bunch of handy functions for addons (like adding items to the directory list) and some lower level stuff like handling html entities, and another module that wraps up urllib2 to make it easy to make http requests with gzip support and character encoding decoded properly (maybe caching and proxy stuff too). should end up in git sometime tomorrow. then i can release the test addon and it might be more obvious how the urlresolver works.

t0mm0

[t0mm0]

hi all,

head on over to the github page and you'll noticed i just added a load of stuff.

script.module.t0mm0.common - has a bunch of useful code that is common to lots of addons. currently there is a class (Addon) that wraps around various xbmc addon functionality, and another one (Net) that wraps urllib2 to make it easy to deal with cookies, proxies etc and tries it's best to return proper unicode no matter what character set is used on the web page)

thoughts (or patches!) on more useful stuff to add are welcomed (already got a few ideas...)

plugin.video.t0mm0.test - demos the functionality of both urlresolver and t0mm0.common. includes a test section which has a test link for each resolver plugin currently available, and an example of scraping tubeplus.me (i don't like this site as it is slow, has lots of incorrect links, and lists every show ever made even if there are no links available, but it is ok to test stuff with - anyone know of a better candidate for testing?)

hopefully this will make it a little clearer how everything works, but please ask if it's not. there are still no docs as i'm not done changing stuff around yet. there is not much error handling in some places either, so expect some 'script errors' (eg when there is no resolvable link or if the file has been removed from the hoster site)

please give it a test and look at the code (especially the test addon, which shows you how to use stuff, and the plugins directory of urlresolver which shows how to make a plugin) and let me know what you think. my goal is to make writing an addon as simple as possible requiring as little code as possible.

thanks,

t0mm0.

[rogerthis]

Big THANK YOU for this.

Had a quick look and it looking really good.

You don't have this in your repository.
Is the plan to get the two scripts script.module.t0mm0.common and script.module.urlresolver into the official xbmc repository?