[RELEASE] Scraper Editor (Based on ScraperXML open source C# Library) - Help wanted!

  Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #121
olympia Wrote:Hi Nicezia,

are you still maintaining your vital applications?
In other words: do you accept bug reports and will you adopt the editor for the upcoming addon structure?

Yeah but i've been unfortunately offline lately due to financial issues.
as soon as i get the version i'm working on now of all programs stable i'm going to upload the new source

as far as add-on structure, release versions won't be supporting it until a release version of XBMC includes it.
but the svn of scraperxml, gcm, scrapercontrols, and scraperxml editor will see support for add-on structure.

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2010-06-03 23:04 by Nicezia.)
find quote
olympia Offline
Team-Kodi Member
Posts: 2,523
Joined: May 2008
Reputation: 32
Post: #122
Nicezia Wrote:Yeah but i've been unfortunately offline lately due to financial issues.
as soon as i get the version i'm working on now of all programs stable i'm going to upload the new source

as far as add-on structure, release versions won't be supporting it until a release version of XBMC includes it.
but the svn of scraperxml, gcm, scrapercontrols, and scraperxml editor will see support for add-on structure.

So may I report some issues to you? Where do you prefer to have the reports? Here or somewhere else?

Thanks!
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #123
Here or e-mail is the best place

niceziavincent@gmail.com

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #124
Just a note.

ScraperXML Editor is on hold while i update it to support creating scrapers as add-ons... I may be uploading the final version of the 3.x version soon, depending on how much work remains to be done on it, and whether XBMC 10.x comes out before that work is done

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #125
Need someone to explain just how the add-on management works in XBMC for scrapers.

So far i have it so scraperxml will create the Scraper, The settings, The language strings(needing to be defined by user), and zipping up the the entire package.

I just need some help in creating an addon.xml to drop beside it so it can easily be submitted to repositories, and being able to import easily from repositories. So i need to understand the addon.xml

for instance, 1lting.com scraper has three lines defining <extension>
Code:
<extension language="zh" library="1ting.xml" point="xbmc.metadata.scraper.albums"/>
<extension language="zh" library="1ting.xml" point="xbmc.metadata.scraper.artists"/>
<extension point="xbmc.addon.metadata">
   <summary lang="en">1ting Scraper</summary>
   <summary lang="se">1ting Musikskrapa</summary>
   <summary lang="kr">1ting 음악 스크래퍼</summary>
   <description lang="en">Download Music information from www.1ting.com</description>
   <description lang="se">Ladda ner musikinformation från www.1ting.com</description>
   <description lang="nl">Download muziek informatie van www.1ting.com</description>
   <description lang="kr">www.1ting.com 에서 음악 정보 다운로드</description>
   <platform>all</platform>
</extension>

three of these defining "point"

two out of the three points define the type of info it scrapes, the third defines that it is a metadata scraper. From this i gather that there are three different categories this would show up in... metadata, metadata.artist, metadata.albums... the part that confuses me, is that under the one that defines it as a metadata the format of info is completely different (and the languae of the scraper itself is left unused) I don't undersand why its descrption and summary is defined inside extension rather than as a child of the <addon> itself as seems logical to me, so i need this explained. because this makes it difficult to create a class to handle it not understanding why the info is available only under extension, if extension defines where this item is to appear in the add-on management, then the summary and description would appear with it ONLY where it is accessible in the add-on management, right?

So basically the question is why the above and not this?

Code:
<addon id="metadata.1ting.com" name="1ting" provider-name="Taxigps" version="1.0.1">
   <requires>
      <import addon="xbmc.metadata" version="1.0"/>
   </requires>
   <extension language="zh" library="1ting.xml" point="xbmc.metadata.scraper.albums"/>
   <extension language="zh" library="1ting.xml" point="xbmc.metadata.scraper.artists"/>
   <extension language="zh" library="1ting.xml" point="xbmc.addon.metadata"/>
   <summary lang="en">1ting Scraper</summary>
   <summary lang="se">1ting Musikskrapa</summary>
   <summary lang="kr">1ting 음악 스크래퍼</summary>
   <description lang="en">Download Music information from www.1ting.com</description>
   <description lang="se">Ladda ner musikinformation från www.1ting.com</description>
   <description lang="nl">Download muziek informatie van www.1ting.com</description>
   <description lang="kr">www.1ting.com 에서 음악 정보 다운로드</description>
   <platform>all</platform>
</addon>

I just need to undestand how it gets processed so i can handle it in the same manner.

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,221
Joined: Oct 2003
Reputation: 178
Post: #126
This might be of use:

http://wiki.xbmc.org/index.php?title=Add..._Extension

Regarding the way it's processed, technically the xbmc.addon.metadata extension isn't required for an addon to do something, it just gives the description information to the user. Note that <summary> etc. are inside the xbmc.addon.metadata <extension> as they're properties of that extension.

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
(This post was last modified: 2010-06-14 04:57 by jmarshall.)
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #127
jmarshall Wrote:This might be of use:

http://wiki.xbmc.org/index.php?title=Add..._Extension

Regarding the way it's processed, technically the xbmc.addon.metadata extension isn't required for an addon to do something, it just gives the description information to the user. Note that <summary> etc. are inside the xbmc.addon.metadata <extension> as they're properties of that extension.

Cheers,
Jonathan


Thanks that cleared up everything quite nicely, i believe...

Just to be clear, extension points can provide information about how they modify the point they are insterted into, for example an extention point intefacing the gui, can give information about resolutions supported, and such...

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2010-06-14 06:17 by Nicezia.)
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,221
Joined: Oct 2003
Reputation: 178
Post: #128
Correct. Each <extension> tag describes an extension to some part of XBMC (or some part of another addon) that this particular addon extends. The "point" attribute is the specific part that it extends, and there may be more attributes or elements associated with that (such as the library attribute).

Music scrapers can do both artists and album scraping, thus extend 2 parts of XBMC, whereas TV or Movie scrapers extend only 1 part of XBMC.

In theory, one addon could be written that could provide data for all 4 types, though the support for that internally is not yet fully resolved - essentially it would be a bundle of individual scrapers.

In addition, all addons that users will see extend xbmc by providing information about themselves by extending the xbmc.addon.metadata point. You'll notice that addons that provide scraper libraries don't need this extension as they don't show up in the UI anywhere - they're only a dependency of other addons.

If you want further info on extensions and extension points then the c-pluff docs are useful.

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #129
Allright almost done with the new version

I have edited process so it will work with either add-on or old method (which you need to specify upon opening the program, also can be set to stay in add-on mode, or old method). If in add-on mode any older method scraper loaded will lose any include references upon opening, but will be converted to add-on specifications.

Also if in addon mode, a new tab will appear amongst Editor & Tester, the "Info" tab, which will allow you to specify the add-on info, add imports (formerly includes)... etc

Internet connection is required at least the first time running ScraperXml Editor, as the program will download all available imports, and cache them for use with scrapers (you can manually update these at anytime after that).

Only have a few more aspects of the add-on spec to account for...

Sorry for the time its taking to get this ready, initially i thought i'd have to rethink the whole process, but i was overthinking thee problem, and it turned out to be a simple process to account for both methods

I will still continue supporting old method after 10.5 is released for those of us still using XBMC for XBox, at least till XBMC for XBox moves to the add-on system.

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
hojel Offline
Junior Member
Posts: 11
Joined: Dec 2009
Reputation: 0
Post: #130
GetEpisodeList behavior was changed in new svn trunk.
I can not find any related post on it, but it works differently for multiple episode pages.

The function was called for each page in the list with two parameters: $$1 that holds page contents and $$2 for URL.
In the current version, however, it is called once with parameters:
$$1 for the first page, $$2 for the second page...
Number of buffers passed to the function is one of urls defined in <episodelist> + 1. The last buffer holds the first url.

In my addon(metadata.tv.daum.net), number of episode pages are determined in parsing a main page, so the new method lets the scrapper code ugly. Sad
find quote
UsagiYojimbo Offline
Member
Posts: 85
Joined: Feb 2010
Reputation: 1
Location: Debrecen, Hungary
Question   
Post: #131
I have a question:

Is it possible with the ScraperXML Editor to create a multi-format scraper?
I mean a scraper that can scrape EG movies as well as tv-shows or even music videos?
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #132
UsagiYojimbo Wrote:I have a question:

Is it possible with the ScraperXML Editor to create a multi-format scraper?
I mean a scraper that can scrape EG movies as well as tv-shows or even music videos?

that's a request better made in XBMC development, Scraperxml and all projects i work on as derivitives of it are based on XBMC scraper process.

in a way it is possible once the add-on structure is supported in ScraperXML Editor, but i can see that there would be all kinds of problems with a scraper of that sort...
for example... how would you know which scraper type to use when its running?
say you had this scraper with multi-format....

since TV Show scrapers, Movie Scrapers, and Music Video Scrapers all use the same function (CreateSearchUrl) to create a url... how would you tell it to create a different url for whatever type you are using?
i.e
Code:
[b][url]http://foo.com/tv/search.php?q=Family%20Guy[/url][/b]
- or -
[b][url]http://foo.com/musicvideos/search.php?artist=Eminem&video=The%20Way%20I%20Am[/url][/b]

if the site being used makes no distinction at all between types during a search, then i suppose it could be safe to do something like you suggest... but until XBMC officially supports something like this i doubt i'll be incorporating it into ScraperXML Editor

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
(This post was last modified: 2010-07-17 00:53 by Nicezia.)
find quote
UsagiYojimbo Offline
Member
Posts: 85
Joined: Feb 2010
Reputation: 1
Location: Debrecen, Hungary
Post: #133
Nicezia Wrote:say you had this scraper with multi-format....
What I was interested in, is multiple scrapers (for the same site) in a single add-on...
Could this be created with new version of the editor?
find quote
Nicezia Offline
Fan
Posts: 369
Joined: Nov 2006
Reputation: 0
Location: Montgomery, Alabama
Post: #134
Looking for someone who knows how to create GTK+(& possibly QT) GUIs to give me a crash tutorial so i can make the ScraperXml Editor cross-platform through mono (and eventually in pure C++).

ScraperXML Open Source Web Scraper Library compatible with XBMC XML Scrapers


I Suck, and if you act now by sending only $19.95 and a self addressed stamped envelop, so can you!

[Image: teamumx_sigline.png]
find quote
flobbes Offline
Senior Member
Posts: 131
Joined: Mar 2009
Reputation: 0
Post: #135
Nicezia Wrote:Looking for someone who knows how to create GTK+(& possibly QT) GUIs to give me a crash tutorial so i can make the ScraperXml Editor cross-platform through mono (and eventually in pure C++).

I send you a pm.
find quote
Post Reply