Javascript is required

  Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
antrrax Offline
Junior Member
Posts: 30
Joined: Nov 2016
Reputation: 0
Post: #1
Reformulating the topic not to violate the rules of the forum:

How to get html source code on sites that require Javascript without using Selenium WebDriver? Is there any other way?

Skipmode A1 Wrote:You can run javascript with nodejs: see here

This however requires nodejs present on the computer you running your kodi-add on.

I use Windows, and I installed cfscrape in python 2.7:
python -m pip install -U pip cfscrape

And installed Node.JS

I'm trying to get the source code of the site where javascript is required, eg sites hosted on Cloudflare
first test in Python IDLE (Windows):
import cfscrape

scraper = cfscrape.create_scraper()
print scraper.get('').content

But it still did not open Javascript. Could you give me an example of how to use cfscrape with nodejs?

(This post was last modified: 2017-04-19 06:49 by antrrax.)
find quote
Skipmode A1 Offline
Skilled Python Coder
Posts: 311
Joined: Mar 2013
Reputation: 25
Post: #2
Here some code that seemed to work when i was messing with it a while back:

# Make a session
        sess = requests.session()

        # Set cookies for cookie-firewall and nsfw-switch
        if SETTINGS.getSetting('nsfw') == 'true':
            cookies = {"Cookie": "cpc=10", "nsfw": "1"}
            cookies = {"Cookie": "cpc=10"}

        # Determine if cloudflare protection is active or not
        html_source = sess.get(self.video_list_page_url, cookies=cookies).text
        if str(html_source).find("cloudflare") >= 0:
            cloudflare_active = True
            cloudflare_active = False

        # Get the page
        if cloudflare_active == True:
                import cfscrape
                xbmcgui.Dialog().ok(LANGUAGE(30000), LANGUAGE(30513))
                # returns a CloudflareScraper instance
                scraper = cfscrape.create_scraper(sess)
                xbmcgui.Dialog().ok(LANGUAGE(30000), LANGUAGE(30514))
                html_source = scraper.get(self.video_list_page_url).content
                xbmcgui.Dialog().ok(LANGUAGE(30000), LANGUAGE(30515))

        # Parse response
        soup = BeautifulSoup(html_source)
find quote
pinoytracker Offline
Junior Member
Posts: 13
Joined: Apr 2017
Reputation: 0
Post: #3
any update on this, Thanks
find quote