How to scrape an "infinite" scrolling website like this?
#1
So I saw some methods about how to scrape "infinite" scrolling websites, but they don't apply to the site I'm trying to scrape.

Maybe someone with more experience with scrappers can take a look and help me.

PS: It's an adult site.

https://www.myfreecams.com/#Homepage

I'm not sure on how to scrape a website like that in "pages"

Once again thanks for any help.
Reply
#2
Self-quote from another topic:
(2018-11-05, 14:37)Roman_V_M Wrote: with modern JavaScript-based websites the situation gets more complicated. You need to use either a headless browser engine, which is not a trivial task for a Kodi addon, or to reverse-engineer the front-end JavaScript application to find REST endpoints it gets its data from and use those endpoints in your addon to fetch data.
Raspberry PI 2 + LibreELEC 8 (Kodi 17)
Samsung Galaxy Tab A 10.1 + Kodi 17 for Android
Reply
#3
(2018-11-15, 22:40)Roman_V_M Wrote: Self-quote from another topic:
(2018-11-05, 14:37)Roman_V_M Wrote: with modern JavaScript-based websites the situation gets more complicated. You need to use either a headless browser engine, which is not a trivial task for a Kodi addon, or to reverse-engineer the front-end JavaScript application to find REST endpoints it gets its data from and use those endpoints in your addon to fetch data.
Thanks, will take a look into things like that I see what I can find about it Wink
Reply
 
Thread Rating:
  • 0 Vote(s) - 0 Average



Logout Mark Read Team Forum Stats Members Help
How to scrape an "infinite" scrolling website like this?00