[spoyser]

Addon looking good.

However, just tried a search for Eastenders and got the dreaded infinite spinning "working".

I'm guessing a regex somewhere is getting stuck.

[CaptainT]

However, just tried a search for Eastenders and got the dreaded infinite spinning "working".

I'm guessing a regex somewhere is getting stuck.

I can't verify this right now. However, how long did you wait for a result? Search is known to take very long, usually at least 20-30 seconds even on high-end computers. If you are using something like Raspi 1 it may take more than a minute to complete.

[speedwell68]

However, just tried a search for Eastenders and got the dreaded infinite spinning "working".

I'm guessing a regex somewhere is getting stuck.

I can't verify this right now. However, how long did you wait for a result? Search is known to take very long, usually at least 20-30 seconds even on high-end computers. If you are using something like Raspi 1 it may take more than a minute to complete.

Surely it is also based on the speed on the BBC's servers?

[spoyser]

However, just tried a search for Eastenders and got the dreaded infinite spinning "working".

I'm guessing a regex somewhere is getting stuck.

I can't verify this right now. However, how long did you wait for a result? Search is known to take very long, usually at least 20-30 seconds even on high-end computers. If you are using something like Raspi 1 it may take more than a minute to complete.

45 minutes and still spinning

The fetching of the html on my machine is less than 1 second so it isn't that, it is definitely the regex that is at fault.

A poorly constructed regex can take an age to perform, and just because a regex works doesn't mean it is efficient. I well crafted scrape of a page can be orders of magnitude faster than a poorly designed one, and I mean maybe 90 seconds down to 5 seconds (funnily enough that I observed that on one of the other iPlayer addons).

I would suggest looking at the use of the split function, this can be used to break the html down into small manageable chunks of text that a regex can parse much more efficiently.

To be honest the whole scrape should only take a few seconds once the html has been fetched (assuming only one page needs to be fetched).

Also I had a very quick look at your code, and a

Code:

while True:
    try:
        #do stuff
    except:
        break

Isn't the best, 'while True' is virtually always a bad idea, especially when you are relying on the throwing on an exception to break out of it.

(It isn't that that which is causing the failing search though)

[CaptainT]

45 minutes and still spinning

OK. That's way too long. So Search seems to be broken for now.

The fetching of the html on my machine is less than 1 second so it isn't that, it is definitely the regex that is at fault.

A poorly constructed regex can take an age to perform, and just because a regex works doesn't mean it is efficient. I well crafted scrape of a page can be orders of magnitude faster than a poorly designed one, and I mean maybe 90 seconds down to 5 seconds (funnily enough that I observed that on one of the other iPlayer addons).

I certainly believe that and I am not a master of regexps. The main task is to keep the add-on working from my point of view.

I would suggest looking at the use of the split function, this can be used to break the html down into small manageable chunks of text that a regex can parse much more efficiently.

To be honest the whole scrape should only take a few seconds once the html has been fetched (assuming only one page needs to be fetched).

The thing with search is, that it needs to scrape multiple pages. For Eastenders it should be about 15 pages, for other searches it may be up to 20 pages. The reason is that the Beeb lists lots of unavailable episodes on the first pages of the search results and then some available episodes later. This is a weird concept and unfortunately means that all pages need to be scraped to get all available episodes.

Also I had a very quick look at your code, and a

Code:

while True:
    try:
        #do stuff
    except:
        break

Isn't the best, 'while True' is virtually always a bad idea, especially when you are relying on the throwing on an exception to break out of it.

(It isn't that that which is causing the failing search though)

Agreed that this is not the best way of doing it. And I also agree that it needs fixing. I currently have no access to a working Kodi installation and won't for the next week at least. Before that, I won't be able to locate the problem in the code.

Your post indicates that you already found out where it fails. If you did, please drop me a message and I may be able to provide something for you to test and work around the issue and in case of success may even release a new version.

[spoyser]

45 minutes and still spinning

OK. That's way too long. So Search seems to be broken for now.

The fetching of the html on my machine is less than 1 second so it isn't that, it is definitely the regex that is at fault.

A poorly constructed regex can take an age to perform, and just because a regex works doesn't mean it is efficient. I well crafted scrape of a page can be orders of magnitude faster than a poorly designed one, and I mean maybe 90 seconds down to 5 seconds (funnily enough that I observed that on one of the other iPlayer addons).

I certainly believe that and I am not a master of regexps. The main task is to keep the add-on working from my point of view.

I would suggest looking at the use of the split function, this can be used to break the html down into small manageable chunks of text that a regex can parse much more efficiently.

To be honest the whole scrape should only take a few seconds once the html has been fetched (assuming only one page needs to be fetched).

The thing with search is, that it needs to scrape multiple pages. For Eastenders it should be about 15 pages, for other searches it may be up to 20 pages. The reason is that the Beeb lists lots of unavailable episodes on the first pages of the search results and then some available episodes later. This is a weird concept and unfortunately means that all pages need to be scraped to get all available episodes.

Also I had a very quick look at your code, and a

Code:

while True:
    try:
        #do stuff
    except:
        break

Isn't the best, 'while True' is virtually always a bad idea, especially when you are relying on the throwing on an exception to break out of it.

(It isn't that that which is causing the failing search though)

Agreed that this is not the best way of doing it. And I also agree that it needs fixing. I currently have no access to a working Kodi installation and won't for the next week at least. Before that, I won't be able to locate the problem in the code.

Your post indicates that you already found out where it fails. If you did, please drop me a message and I may be able to provide something for you to test and work around the issue and in case of success may even release a new version.

Rather than scrape all pages, might be worth just listing the program, then if the user wants to they can drill down.

Looking at the code some more I can see where it's inspiration came from which would suggest it suffers from the same problem as the other iPlayer addon.

I'll find my fix and post it when I get a chance (probably not till weekend now though)

[CaptainT]

Rather than scrape all pages, might be worth just listing the program, then if the user wants to they can drill down.

I thought about that, too, but decided against it. There is this strange situation, where you would get no results at all, say, on page 8 and then again results on page 9. It really seems to me like the search on the iPlayer website was kind of broken on purpose. I would expect them to offer an option to show only available programmes, for instance. So in order to make it easier to use, I chose to scrape all pages and then display the results.

I'll find my fix and post it when I get a chance (probably not till weekend now though)

Please do so. I will open a ticket on Github in parallel, perhaps some of the other contributors will pick it up.

Also, if you know any good tutorial on efficient regexps in Python, please drop me a link. I found the Python documentation itself very comprehensive, but not very helpful.

[thatguy_za]

@CaptainT, thank you very much for all of the work you are doing.

I have tried to install iPlayer WWW from the Kodi/OSMC video add-ons store and it is not listed. Is this because I am outside of the UK?

Thanks

[primaeval]

I've got a working Search in my Github version but it is a major change.

[scallywagger]

Although I'm having trouble using the add on with Kodi 15.2 Isengard on Amazon Fire TV in the UK.

I can log in using my BBC ID, retrieve my favourite shows and search program listings but whenever I try to play a stream I get the following error:

('Connection aborted.',error(104, 'Connection reset by peer'))

Any ideas please?

Sorry, I don't have a clue at the moment.

Could you please try to disable the autoplay function in settings and then manually try to play several different streams if you always get the same error?

Thanks in advance!

Thanks for getting back to me CaptainT.

If I disable automatic streaming I don't get an option to manually choose a stream when trying to play non-live content. I do get this option with live content and it appears that this works with Limelight (any quality) but not Akamai.

Whether automatic stream selection is enabled or not, non-live content fails with the error message previously mentioned.

I've tried uninstalling and reinstalling to no avail and other Iplayer clients on the same network function perfectly.

Baffled!

[spoyser]

Rather than scrape all pages, might be worth just listing the program, then if the user wants to they can drill down.

I thought about that, too, but decided against it. There is this strange situation, where you would get no results at all, say, on page 8 and then again results on page 9. It really seems to me like the search on the iPlayer website was kind of broken on purpose. I would expect them to offer an option to show only available programmes, for instance. So in order to make it easier to use, I chose to scrape all pages and then display the results.

I'll find my fix and post it when I get a chance (probably not till weekend now though)

Please do so. I will open a ticket on Github in parallel, perhaps some of the other contributors will pick it up.

Also, if you know any good tutorial on efficient regexps in Python, please drop me a link. I found the Python documentation itself very comprehensive, but not very helpful.

I have uploaded a file here in which I applied the same fix as I did to the other iPlayer addon:

https://dl.dropboxusercontent.com/u/1155...default.py

Seems to work okay, although I couldn't see what some of your code was doing so I return before it is called I'm sure it did something important though.

I have also added an optional keyword to the search feature, this is so a user can call your addon externally with a pre-populated keyword (I would like this so I can add your addon to the iSearch feature of my Super Favourites addon)

[CaptainT]

Seems to work okay, although I couldn't see what some of your code was doing so I return before it is called I'm sure it did something important though.

OK, thanks. I can see what you are trying to do and it does make sense to break the HTML down before hitting it with regexps. However, because your code tries to parse all types of content with the same regexp. I don't think that it returns all correct results, especially because there are unavailable programmes, that still have an ID. Plus, the part that is really broken (search groups) is completely ignored, just like in 1.2.8, where it is disabled for now.

Nevertheless, your approach shows some new ideas how to optimize the whole add-on. This would be a possible alternative to primaeval's approach to replace regexps with beautiful soup. Let's see which one wins.

I have also added an optional keyword to the search feature, this is so a user can call your addon externally with a pre-populated keyword (I would like this so I can add your addon to the iSearch feature of my Super Favourites addon)

That sounds like an interesting addition. I will keep it in mind for a future release. Thanks.

[CaptainT]

If I disable automatic streaming I don't get an option to manually choose a stream when trying to play non-live content. I do get this option with live content and it appears that this works with Limelight (any quality) but not Akamai.

This really looks like geoblocking, but a kind of geoblocking that I haven't seen before. So for some reason, the Beeb thinks you are outside of the UK, but treats you different than the usual crowd. Are you able to edit the plugin and look at the Kodi.log on the Amazon Fire TV? If so, I can send you some instructions for debugging by PM. Otherwise, I would need to prepare a special version for.

I've tried uninstalling and reinstalling to no avail and other Iplayer clients on the same network function perfectly.

Now that's an interesting bit of information. Do I understand you correctly, that you tried other clients on the same network, but not on the same Amazon Fire TV? If so, my assumption would be that Amazon Fire TV uses internally different network settings, i.e. "special Amazon DNS" which leads to this strange geoblocking behaviour.

[spoyser]

Seems to work okay, although I couldn't see what some of your code was doing so I return before it is called I'm sure it did something important though.

OK, thanks. I can see what you are trying to do and it does make sense to break the HTML down before hitting it with regexps. However, because your code tries to parse all types of content with the same regexp. I don't think that it returns all correct results, especially because there are unavailable programmes, that still have an ID. Plus, the part that is really broken (search groups) is completely ignored, just like in 1.2.8, where it is disabled for now.

Nevertheless, your approach shows some new ideas how to optimize the whole add-on. This would be a possible alternative to primaeval's approach to replace regexps with beautiful soup. Let's see which one wins.

Exactly, it was more to show how regex can be made more efficient by a simple bit of pre-processing. I've seen hideously complex inefficient regexes in the past, which one simple replace would have made orders of magnitude more efficient. I haven't really looked at all the possible pages that the Beeb return hence my comments about the code I removed still being important Possibly have 3 different parsing methods which all receive a slightly different versions of the processed html? At the moment what is there works for what I tend to search for though

I have also added an optional keyword to the search feature, this is so a user can call your addon externally with a pre-populated keyword (I would like this so I can add your addon to the iSearch feature of my Super Favourites addon)

That sounds like an interesting addition. I will keep it in mind for a future release. Thanks.

If you can keep it in that would be great, it doesn't make any difference to the behaviour of the addon but with that extra code I can call it from Super Favourites like this:

Code:

ActivateWindow(10025,"plugin://plugin.video.iplayerwww/?mode=104&url=url&keyword=[%SF%]&sf_options=fanart%3Dspecial%3A%2F%2Fhome%2Faddons%5Cplugin.video.iplayerwww%5Cfanart.jpg%26_options_sf",return)

Where Super Favourites injects a keyword into the [%SF%]

(Which people are already asking for LOL)

[scallywagger]

If I disable automatic streaming I don't get an option to manually choose a stream when trying to play non-live content. I do get this option with live content and it appears that this works with Limelight (any quality) but not Akamai.

This really looks like geoblocking, but a kind of geoblocking that I haven't seen before. So for some reason, the Beeb thinks you are outside of the UK, but treats you different than the usual crowd. Are you able to edit the plugin and look at the Kodi.log on the Amazon Fire TV? If so, I can send you some instructions for debugging by PM. Otherwise, I would need to prepare a special version for.

I've tried uninstalling and reinstalling to no avail and other Iplayer clients on the same network function perfectly.

Now that's an interesting bit of information. Do I understand you correctly, that you tried other clients on the same network, but not on the same Amazon Fire TV? If so, my assumption would be that Amazon Fire TV uses internally different network settings, i.e. "special Amazon DNS" which leads to this strange geoblocking behaviour.

Thanks for your thoughts on this CaptainT, they've lead me in an interesting direction!

Not wanting to root my FireTV, I use URL filtering on my router (as per this post here) to prevent it from automatically updating the software and potentially borking the Kodi install.

Unfortunately text entry length limitations in my router's UI prevents me from specifically blocking "amzdigitaldownloads.edgesuite.net" and so I was blocking the whole edgesuite.net domain instead and it appears that this was preventing the iPlayer WWW plugin from working.

I've now removed this entry and all is working as it should but I'm not sure if just blocking the other two Amazon sites will be enough to prevent updates from being pushed to my FireTV... may have to root after all..