Kodi Community Forum

Full Version: [Linux] script.module.chromium: Chromium service for Kodi
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Before I start, this does not give users a browser gui or any kind, but is an addon for developers to use Chromium browser to web scrape.

What is it?:
This addon uses system docker to spawn a Chromium process with gui enabled. Even though gui is enabled inside the docker, your host OS does not need to have a working X11 or any kind of redirection, because the GUI of chromium is running inside Xvfb (Virtual X framebuffer). The spawned Chromium process is enabled with the CDP (Chromium Developer Protocol? or sth like that), so that, and third party client can conenct with websockets and control the browser according to your needs.

How can i use it?:

Depend the addon in addon xml and use following syntax:
 

from chromium import Browser  # import the library

useragent = None  # String value to spoof User-Agent, dont spoof it unless you know what you are doing
loadtimeout = 5  # Maximum time to wait in seconds for Document to load dynamically (time needed for JS or client side scripts to finish manipulating the DOM)
maxtimeout = 15  # Maximum time to wait in seconds, if this times out, tab is closed and return value is what is already received
port = 9222  # the cdp port to conenct to, by default the service is running on 127.0.0.1:9222, so changing his value will not take affect since the service is hardcoded run on 9222, u can change it to connect another instance on your PC for debugging.

# The Browser arguments are default to above

with Browser(useragent, loadtimeout, maxtimeout,port) as browser:
    page = browser.navigate("https://www.google.com")
    #the return value is unicode HTML data

# so simply use:
with Browser() as browser:
    page = browser.navigate("https://www.google.com")


Why would i use it?:
I started this to overcome Cloudflare IUAM mode when using TOR. Unlike normal browsing, TOR browsing is close to impossible thanks to Cloudflare is being a faschist against the requests come from TOR network, and always utilize the Javascript bypass page, and this bypass mechanism is extremely dynamic, complicated, and frequently updated. Therefore i gave up on pure python bypass technics and use this instead, and so far works like charm. And besides, today web sites heavily depend on client side scripting therefore this is kind of a must to scrape stuff from the web.

Which platforms are supported?:
Linux, only linux. I have tested on my Mint Desktop and Corelec STB and everthing is smooth there. It is written PY2 + PY3 compatible so works on Kodi 18 & 19 boh from the same repo. Architecturally, i have infrastructure that can support even from XBMC 14 gotham but dont go that far, i have never test lover than 18. 

This system is using Xvfb which is a linux component, so it wont scale to windows or Android. May be another method can considered for Windows but there is no chance in Androids. At least i dont really care at.

PS: on linux make sure the user that spawns the kodi process has rights to execute docker commands, the service basically automates Docker, so there must a a working docker to sue this. On corelec/libreelec it is based on service.system.docker which comes with the distro, so you dont need to do much.

Advanced usage:

The browser instance has already implemented a websocket connection management, and primitives to execute the CDP commands, so instead of directly using .navigate method you can interact with CDP according to your wish.

Sources:
The code lies below:
https://github.com/hbiyik/repository.biy...e.chromium

The docker files:
https://github.com/hbiyik/docker-xvfb-chromium

The repo itself:
https://github.com/hbiyik/repository.biyik/releases

I highly suggest to use the repo, if you want to do development, there are dependencies for this script which is hard to manage manually.

Todo:
  • Inroduce a 'validation' callback method as an input to navigate method, so that the client will wait until the DOM response is expected, instead of listening DOM events with timer   done in 0.0.2
  • Make the script as program and utilize the settings.xml
  • Make docker settings customizable
  • Make the port customizable? Who needs that? Dunno

Known Issues:
On Libre/Corelec A dialog keeps popping up sayin "xvfb-chromium:kill" however does not really kill, This does not impact the operation but gets kind of boring when it keeps spamming the beeps. I ll look into it. Seems like there is another service monitoring all Libreelec Dockers.
After several improvements and tests, i can say this works pretty good.

On Windows the same thing can be applied with native Chrome in the background. Did a dirty test and it works.

But when it comes to Android, i could not find a way to utilize the Chrome listning on a port for "devtools protocol" websocket requests. May be there are people here who has an idea how to achieve simikar aplication in Android?
If feels kinda speaking with myself Smile but i think i have an idea for android:

The termux, terminal emulator, has a repo of linux appllications compiled/ported against android SDK and so there is docker there.
https://packages.termux.org/apt/termux-r.../d/docker/

So theoretically, it is possible to sideload all the necessary user space binaries and libraries to profile folder under linux. And after necessary chmod settings and environment manipulations (ie:LD_LIBRARY_PATH) it should be possible to the execute Android Docker Service from Kodi. Then rest is the same process for installing the chromium container inside.

One concern would be if, Android kernel really supports the Docker virtualization things, which i have no clue about. The container will be in the same architecture as the host ABI so there will be no emulation and so forth, therefore speed should not be  a concern.

If that works, this system would provide cross-platform browser based scraping which is kind of good stuff.

Again all of it is a fun theory to experiment when i have the time.
Can we work this solution under Xbox Series? What do you think?
which OS does it run, or does it have an OS Smile? i have no clue how to kodi runs on XBOX, the most important question is is there any Chromium that can run on XBOX where we can connect as CDP client.
Does it run on LibreELEC? Which is Linux, but maybe amputated as far as your add-on's demands go?
(2022-04-27, 16:52)DiMag Wrote: [ -> ]Does it run on LibreELEC? Which is Linux, but maybe amputated as far as your add-on's demands go?

Yep, i am using it daily on Coreelec, so should run on Libreelec as well. You just need to install Docker Service addon.