Kodi Community Forum
script.module.urlresolver development - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=26)
+--- Thread: script.module.urlresolver development (/showthread.php?tid=105707)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28


- rogerthis - 2011-08-14

I having a problem with HttpResponse. I need it for the get_url function for movshare. Here's my code http://pastebin.com/1Kw5zAt6 And the log http://pastebin.com/sdEXjTyR

This is driving me mad, I can't figure it out.


- t0mm0 - 2011-08-14

rogerthis Wrote:I having a problem with HttpResponse. I need it for the get_url function for movshare. Here's my code http://pastebin.com/1Kw5zAt6 And the log http://pastebin.com/sdEXjTyR

This is driving me mad, I can't figure it out.

you never make a HttpResponse directly, it is what is returned by Net.http_GET() or Net.http_POST()

what exactly are you trying to do?

edit:

ah, i think i see what you are trying to do, is it this?
Code:
link0 = self.net.http_GET(web_url).get_url()

if you want the html as well you can do something like:

Code:
response = self.net.http_GET(web_url)
html = response.content
link0 = response.get_url()

does that make sense?


- icky - 2011-08-14

hi t0mm0, just a quick heads up

linksnappy.com or other multihosters might be the cleverest resolver to make as this will then resolve links to everything else automaticallyHuh?


- rogerthis - 2011-08-15

t0mm0 Wrote:you never make a HttpResponse directly, it is what is returned by Net.http_GET() or Net.http_POST()

what exactly are you trying to do?

edit:

ah, i think i see what you are trying to do, is it this?
Code:
link0 = self.net.http_GET(web_url).get_url()

if you want the html as well you can do something like:

Code:
response = self.net.http_GET(web_url)
html = response.content
link0 = response.get_url()

does that make sense?

Thanks t0mm0, That's exactly what I wanted, sending a pull request for movshare now.


- t0mm0 - 2011-08-15

icky Wrote:hi t0mm0, just a quick heads up

linksnappy.com or other multihosters might be the cleverest resolver to make as this will then resolve links to everything else automaticallyHuh?

interesting idea. will take a proper look later.

rogerthis Wrote:Thanks t0mm0, That's exactly what I wanted, sending a pull request for movshare now.

thanks rogerthis! it is now committed. you are officially the first contributor (other than me) Wink who will be next......

i've also just committed the first pass at a proper addon - it's for letmewatchthis.ch and is very basic at the moment (only has support for TV, and you can only browse by letter) but it seems to work fairly ok.

my two weeks off work is over now so i'll probably slow down a bit for a while Sad but i will continue to work on it as much as possible!

t0mm0.


- rogerthis - 2011-08-16

t0mm0, sent pull request for vidxden/vidbux. Not 100% happy with it, see what you think.


- t0mm0 - 2011-08-16

rogerthis Wrote:t0mm0, sent pull request for vidxden/vidbux. Not 100% happy with it, see what you think.

thanks - you're on fire Wink

i might not get to do too much with it tonight, but i will take a look as soon as i can.

t0mm0


urlresolver and cookies - DragonWin - 2011-08-17

Hi,

I have just started (2 days ago) to learn to code python and xbmc addons, so please be gentle if my questions warrants a slap or two Shocked

I saw a request to make an addon for http://www.solarmovies.eu on the forum and the site makes use of a lot of links, so I figured it would be a good place to start learning how to code addons, specially after I found urlresolver.

I have written 3 plugins for the resolver (divexstage, 2gb-hosting, nolimitvideo), which all still needs extensive testing, but I plan to release them once I'm confident that they work properly.

So here is my question about urlresolver:

How do I make use of cookies, so I can grap the cookies sent to me through a get, and then do a post and at the same time "return" the cookies I just received ?

I have tried fixing / changing the putlocker plugin as the "post" no longer work due to missing cookies.

Would it be possible to show an example of how to generate the cookie path, set the cookie_file, how to "catch" the cookie, and how to "return" the cookie, by using the self.net.set_cookies(self.cookie_file) etc?

ps. Great work on the resolver, it makes creating addons a breeze.

Thanks,
DragonWin


- t0mm0 - 2011-08-17

DragonWin Wrote:Hi,

I have just started (2 days ago) to learn to code python and xbmc addons, so please be gentle if my questions warrants a slap or two Shocked
no problem! happy to help if i can Wink
DragonWin Wrote:I saw a request to make an addon for http://www.solarmovies.eu on the forum and the site makes use of a lot of links, so I figured it would be a good place to start learning how to code addons, specially after I found urlresolver.
nice - looks like a good site!
DragonWin Wrote:I have written 3 plugins for the resolver (divexstage, 2gb-hosting, nolimitvideo), which all still needs extensive testing, but I plan to release them once I'm confident that they work properly.
great - the more people who contribute resolvers the more useful the project will be - please feel free to send them when you're happy - if possible a pull request on github is the best method as it's easier for me to keep track of all the patches.
DragonWin Wrote:So here is my question about urlresolver:

How do I make use of cookies, so I can grap the cookies sent to me through a get, and then do a post and at the same time "return" the cookies I just received ?

I have tried fixing / changing the putlocker plugin as the "post" no longer work due to missing cookies.
now you've lost me - putlocker seems to be working fine for me? (it already sends back cookies)
DragonWin Wrote:Would it be possible to show an example of how to generate the cookie path, set the cookie_file, how to "catch" the cookie, and how to "return" the cookie, by using the self.net.set_cookies(self.cookie_file) etc?
well, cookies are automatically handled already, assuming you use the same instance of Net. for example if you do
Code:
self.net = Net()
response1 = self.net.http_GET('http://example.com/page1')
response2 = self.net.http_GET('http://example.com/page2')
then if the first GET sets cookies, the second one will automatically send them back.

you only need to use a cookie file if you want to store cookies between different executions of the plugin.

to do that (which you probably don't ever need to do at the moment - it might be useful in login code if i add a function to auto log in to all sites that require it), when you create the Net instance, set the cookie_file and it will load in any previously saved cookies, and before your program quits call save_cookies()

eg.
Code:
#obviously you would generate the cookie file using
self.net = Net(cookie_file='/my/cookie/file')
response1 = self.net.http_GET('http://example.com/page1')
self.net.save_cookies('/my/cookie/file')
but to reiterate - you probably don't want to do that (unless this is code in your addon rather than a plugin and you need to store login cookies), cookies are already handled automatically for all http calls within one run of the plugin (as long as you only use one instance of Net).

hope that makes sense?
DragonWin Wrote:ps. Great work on the resolver, it makes creating addons a breeze.

that's the plan Wink

just bear in mind that as the module is not yet released there may be changes that mess up your code until the first real release is done!

any suggestions to make it easier are always welcome too...

t0mm0


- DragonWin - 2011-08-17

Hey T0mm0

I'm unfamiliar with Git, but once I get closer to being satisfied I'll get in contact with you and figure out how it works.

Thanks for the extensive explanation, I feel like a lot of the fog has just lifted from my eyes.

I am aware that the urlresolver is alpha/beta software and not released yet, so I expect some changes to affect the plugins I have / will write. Hopefully it will be fairly easy to adapt to.

You know, after I read your post this morning I suddenly realized that it might as well be my code that supplies an incorrect link (regex not matching all links 100% and getting to much of the line).


Just some thoughts I had...

As I'm a big fan of the icefilms plugin I was wondering if the urlresolver could include a feature to bookmark links for later viewing like icefilms can, or would that be out of scope?

Another idea I had was to build a scraper functions for anonymous proxies to use in plugins, some sites will only allow you to stream from them if your within a specific region. I was thinking that it could be an xml option to "enable anonymous proxy" and "select country" and the feature would scrape the anon proxy sites and do a "head" to eg. google, to detect a working proxy. This would be nice to have in the t0mm0.common.net class to be available to all plugins.


I'm going to have to stop writing now as I have to order plane tickets and book a hotel as I'm leaving tomorrow morning for a business trip Rolleyes

Thanks for the help so far I hope I can put it to good use Big Grin


- rogerthis - 2011-08-17

DragonWin Wrote:As I'm a big fan of the icefilms plugin I was wondering if the urlresolver could include a feature to bookmark links for later viewing like icefilms can, or would that be out of scope?

Yeah, I really like the Favorites in IceFilms. It the first section I go to when I'm using it. I had a look at the icefilms code, and anarchintosh has put it into his addDir:
Code:
def addDir(name, url, mode, iconimage, metainfo=False, imdb=False, delfromfav=False, total=False, disablefav=False, searchMode=False):

Whether it would come under the scope of this project, I don't know.


- DragonWin - 2011-08-17

Hey T0mm0

I think I found out what's wrong with the putlocker plugin

The submit on the page is different from the one that's in the plugin. It looks like they have multiple ways they present the streaming pages.

Here is the url taken from the debug output when it was added by the plugin (removed the url encoding)

http://www.putlocker.com/embed/B28AE94A046F58F1

Web page source
Code:
<input type="hidden" value="1fb066322ff81229" name="hash">
<input name="confirm" type="submit" value="Close Ad and Watch as Free User" class="confirm_button" style="width:250px;">

Plugin code
Code:
try:
            self.net.http_POST(web_url, form_data={'hash': session_hash,
                                          'confirm': 'Continue as Free User'})
        except urllib2.URLError, e:

The submit value is different

I'm off again need some sleep before catching the plane tomorrow morning, I'll check here before I make changes to the putlocker plugin to see if you have changed it.

@rogerthis
Interesting would have to talk to anarchintosh before ripping of any code from the plugin, but def. worth investigating (if it's within scope)

--
DragonWin


- t0mm0 - 2011-08-17

hi!

DragonWin Wrote:I'm unfamiliar with Git, but once I get closer to being satisfied I'll get in contact with you and figure out how it works.
i'm not a git expert either but there's plenty of good docs on github and i'm finding the tools a nice way to collaborate.

DragonWin Wrote:Thanks for the extensive explanation, I feel like a lot of the fog has just lifted from my eyes.

I am aware that the urlresolver is alpha/beta software and not released yet, so I expect some changes to affect the plugins I have / will write. Hopefully it will be fairly easy to adapt to.
absolutely (and of course the docs will be updated). it's good to have people looking at it before it becomes 'stable' and released because you will probably find stuff that is silly and should be changed. since starting to write the letmewatchthis addon i've already started thinking about ways to make it easier.....
DragonWin Wrote:Just some thoughts I had...

As I'm a big fan of the icefilms plugin I was wondering if the urlresolver could include a feature to bookmark links for later viewing like icefilms can, or would that be out of scope?
i don't know if you have read anarchintosh's document that inspired this effort, but basically the original idea was to build a bunch of specialised modules to do different things that could be reused across addons. i started off just saying i would try to implement the resolving bit, though i have also added the generic stuff in t0mm0.common in the process

i'm afraid i am not much of a power user when it comes to xbmc (i spend much more time playing with the code than actually using it!) so i am not really familiar with all these features so not sure i could comment on whether that stuff should be added to t0mm0.common at the moment (it's definitely not urlresolver territory). maybe if someone could define exactly what the feature is and roughly how it works we would be able to make a decision on where to put it.
DragonWin Wrote:Another idea I had was to build a scraper functions for anonymous proxies to use in plugins, some sites will only allow you to stream from them if your within a specific region. I was thinking that it could be an xml option to "enable anonymous proxy" and "select country" and the feature would scrape the anon proxy sites and do a "head" to eg. google, to detect a working proxy. This would be nice to have in the t0mm0.common.net class to be available to all plugins.
i think i like this idea Wink
maybe this could be done in urlresolver, the settings interface could be expanded to add a call to set the preferred country for your plugin? then users could enable the proxy and/or set a custom one instead of an auto-detected one (if they know of a stable proxy in the right country) in each plugins settings. the actual code could probably go in t0mm0.common.net so it is available to addons that don't use urlresolver.

what do you think?

i also want to add decent caching support to t0mm0.common.net as most of the time when browsing show or movie listings the listings remain unchanged so fetching a new copy is pointless.

(also maybe we should come up with a better naming scheme for these modules - any ideas on that front are welcome!)
DragonWin Wrote:I'm going to have to stop writing now as I have to order plane tickets and book a hotel as I'm leaving tomorrow morning for a business trip Rolleyes

Thanks for the help so far I hope I can put it to good use Big Grin

no problem! i hope you find this stuff useful and end up contributing code back to the project Cool

t0mm0


- t0mm0 - 2011-08-17

DragonWin Wrote:Hey T0mm0

I think I found out what's wrong with the putlocker plugin

The submit on the page is different from the one that's in the plugin. It looks like they have multiple ways they present the streaming pages.

Here is the url taken from the debug output when it was added by the plugin (removed the url encoding)

http://www.putlocker.com/embed/B28AE94A046F58F1

Web page source
Code:
<input type="hidden" value="1fb066322ff81229" name="hash">
<input name="confirm" type="submit" value="Close Ad and Watch as Free User" class="confirm_button" style="width:250px;">

Plugin code
Code:
try:
            self.net.http_POST(web_url, form_data={'hash': session_hash,
                                          'confirm': 'Continue as Free User'})
        except urllib2.URLError, e:

The submit value is different

I'm off again need some sleep before catching the plane tomorrow morning, I'll check here before I make changes to the putlocker plugin to see if you have changed it.

ah, looks like we just need to scrape the value of the submit button too then? i'll take a quick look now and see if that sorts it out.

edit: actually just committed the fix - the problem was that to get the url for the xml file to grab i was searching and replacing on 'file', but that didn't do anything when the url was an 'embed' url instead. changed now to look for 'file' or 'embed'. i also changed valid_url() to return True for both putlocker and sockshare urls and removed the sockshare plugin.

thanks,

t0mm0


- rogerthis - 2011-08-18

t0mm0 Wrote:i don't know if you have read anarchintosh's document that inspired this effort, but basically the original idea was to build a bunch of specialised modules to do different things that could be reused across addons. i started off just saying i would try to implement the resolving bit, though i have also added the generic stuff in t0mm0.common in the process

i'm afraid i am not much of a power user when it comes to xbmc (i spend much more time playing with the code than actually using it!) so i am not really familiar with all these features so not sure i could comment on whether that stuff should be added to t0mm0.common at the moment (it's definitely not urlresolver territory). maybe if someone could define exactly what the feature is and roughly how it works we would be able to make a decision on where to put it.
t0mm0

Favorites in IceFilms:
From with the main menu of Icefilms, one of the options is Favourites
To add a TV show you like, right click to bring up the menu. Select "Add to Favorites"
Then when ever you go into Favourites all your favourite shows will be listed. So no more search through long lists in A - Z to bring up your favourite tv shows. It works the same for movies. Very handy indeed. All the favourites are stored locally in a cache file on your xbmc device.

In the IceFilms code, here are the list of Favourite related functions.
def favRead(string):
def addFavourites(enablemetadata,directory,dircontents):
def FAVOURITES(url):
def MOVIE_FAVOURITES(url):
def TV_FAVOURITES(url):
def ADD_TO_FAVOURITES(name,url,imdbnum):
def DELETE_FROM_FAVOURITES(name,url):
def CLEAR_FAVOURITES(url):

There is actually a fair chuck of icefilms addon code related to Favourites.
It could be a project of it's self. I can't see how you could separate it into a completely different module though, it would be better within the script.module.t0mm0.common. t0mm0, if you could have a look it would be great!

t0mm0 Wrote:i think i like this idea Wink
maybe this could be done in urlresolver, the settings interface could be expanded to add a call to set the preferred country for your plugin? then users could enable the proxy and/or set a custom one instead of an auto-detected one (if they know of a stable proxy in the right country) in each plugins settings. the actual code could probably go in t0mm0.common.net so it is available to addons that don't use urlresolver.

what do you think?
This could be another interest setting: X-Forwarded-For
http://forum.xbmc.org/showpost.php?p=691501&postcount=260
It's used to bypass geolock, It seems to work better for the smaller hosters. The likes of the BBC iPlayer and Hulu, it doesn't work with.


t0mm0 Wrote:i also want to add decent caching support to t0mm0.common.net as most of the time when browsing show or movie listings the listings remain unchanged so fetching a new copy is pointless.

Would adding a caching support, enable watched status support? This would be a very useful feature.