2012-07-07, 12:43
You can now follow the work in https://github.com/topfs2/heimdall
The base design of the engine is that the process of scraping is split into tasks, which can run in parallell. These tasks are triggered automatically by the engine, when the scraping item has certain properties etc. So for example we will do the task of searching on tmdb when an item is of type "movie". This work is still in the early stages and the triggering is extremely basic for now but its something to show.
So example of tmdb is https://raw.github.com/topfs2/heimdall/m...rc/tmdb.py and as you can see there is no need to use regexp and its possible to just use json instead to parse, the task can choose the tool fitting for the job on its own
The base design of the engine is that the process of scraping is split into tasks, which can run in parallell. These tasks are triggered automatically by the engine, when the scraping item has certain properties etc. So for example we will do the task of searching on tmdb when an item is of type "movie". This work is still in the early stages and the triggering is extremely basic for now but its something to show.
So example of tmdb is https://raw.github.com/topfs2/heimdall/m...rc/tmdb.py and as you can see there is no need to use regexp and its possible to just use json instead to parse, the task can choose the tool fitting for the job on its own