ScraperEdit for XBMC (Java)

  Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
Michaehm Offline
Junior Member
Posts: 6
Joined: Feb 2013
Reputation: 0
Post: #31
It had some bugs
[Image: 11.jpg]
[Image: 12.jpg]
find quote
UsagiYojimbo Offline
Member
Posts: 91
Joined: Feb 2010
Reputation: 2
Location: Debrecen, Hungary
Post: #32
(2013-02-09 09:02)Michaehm Wrote:  It had some bugs
[Image: 11.jpg]
[Image: 12.jpg]

Sorry, the images are not accessible. Could You repost them?
(This post was last modified: 2013-02-12 08:43 by UsagiYojimbo.)
find quote
Marx1 Offline
Fan
Posts: 367
Joined: Jan 2011
Reputation: 3
Post: #33
Today I've tried 4 scrapers: universal, tmdb, filmwebm filmweb-lite. I've searched for "avatar". None of them gave me any result:
- tmdb 3.7.2 - no results found
- universal 2.4.1 - no results found
- filmweb 2.0 - no results found
- filmweb-lite 1.0.11 - error parsing xml (discussed previously, so not important)

I properly imported your sources and executed xbmc.scraper.Main
Seems that while I get
Code:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<results>
    <entity>
        <title> Avatar </title>
        <url cache="filmweb-299113">http://www.filmweb.pl/Film?id=299113</url>
    </entity>
    <entity>
        <title> Avatar 2 </title>
        <year>2015</year>
        <url cache="filmweb-625427">http://www.filmweb.pl/Film?id=625427</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>2004</year>
        <url cache="filmweb-152652">http://www.filmweb.pl/Film?id=152652</url>
    </entity>
    <entity>
        <title> Awatar / Avatar </title>
        <year>2007</year>
        <url cache="filmweb-480960">http://www.filmweb.pl/Film?id=480960</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>2003</year>
        <url cache="filmweb-238814">http://www.filmweb.pl/Film?id=238814</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>1915</year>
        <url cache="filmweb-5994">http://www.filmweb.pl/Film?id=5994</url>
    </entity>
    <entity>
        <title> Avanak Apti </title>
        <year>1978</year>
        <url cache="filmweb-261715">http://www.filmweb.pl/Film?id=261715</url>
    </entity>
    <entity>
        <title> Avanak kuzenler </title>
        <year>2008</year>
        <url cache="filmweb-493936">http://www.filmweb.pl/Film?id=493936</url>
    </entity>
</results>
in this code:
Code:
final Results rr = (Results) um.unmarshal(new StringReader(res));
parsing results fails and I get "No results found!"


Also there is an small GUI error: if I leave open debug window and load another scraper, executing debug from this opened window gives duplicated logs. Opening another scraper without closing this windows causes logs x3 during debugging
(This post was last modified: 2013-02-26 23:26 by Marx1.)
find quote
UsagiYojimbo Offline
Member
Posts: 91
Joined: Feb 2010
Reputation: 2
Location: Debrecen, Hungary
Post: #34
(2013-02-26 23:23)Marx1 Wrote:  Today I've tried 4 scrapers: universal, tmdb, filmwebm filmweb-lite. I've searched for "avatar". None of them gave me any result:
- tmdb 3.7.2 - no results found
- universal 2.4.1 - no results found
- filmweb 2.0 - no results found
- filmweb-lite 1.0.11 - error parsing xml (discussed previously, so not important)
There was a typo in the sources. (For some reason Huh i used entry instead of entity...)

(2013-02-26 23:23)Marx1 Wrote:  I properly imported your sources and executed xbmc.scraper.Main
Seems that while I get
Code:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<results>
    <entity>
        <title> Avatar </title>
        <url cache="filmweb-299113">http://www.filmweb.pl/Film?id=299113</url>
    </entity>
    <entity>
        <title> Avatar 2 </title>
        <year>2015</year>
        <url cache="filmweb-625427">http://www.filmweb.pl/Film?id=625427</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>2004</year>
        <url cache="filmweb-152652">http://www.filmweb.pl/Film?id=152652</url>
    </entity>
    <entity>
        <title> Awatar / Avatar </title>
        <year>2007</year>
        <url cache="filmweb-480960">http://www.filmweb.pl/Film?id=480960</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>2003</year>
        <url cache="filmweb-238814">http://www.filmweb.pl/Film?id=238814</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>1915</year>
        <url cache="filmweb-5994">http://www.filmweb.pl/Film?id=5994</url>
    </entity>
    <entity>
        <title> Avanak Apti </title>
        <year>1978</year>
        <url cache="filmweb-261715">http://www.filmweb.pl/Film?id=261715</url>
    </entity>
    <entity>
        <title> Avanak kuzenler </title>
        <year>2008</year>
        <url cache="filmweb-493936">http://www.filmweb.pl/Film?id=493936</url>
    </entity>
</results>
in this code:
Code:
final Results rr = (Results) um.unmarshal(new StringReader(res));
parsing results fails and I get "No results found!"
See above...

(2013-02-26 23:23)Marx1 Wrote:  Also there is an small GUI error: if I leave open debug window and load another scraper, executing debug from this opened window gives duplicated logs. Opening another scraper without closing this windows causes logs x3 during debugging
Corrected this one, too.

New version of 0.1.2.65 is out.
(This post was last modified: 2013-03-04 22:52 by UsagiYojimbo.)
find quote
UsagiYojimbo Offline
Member
Posts: 91
Joined: Feb 2010
Reputation: 2
Location: Debrecen, Hungary
Post: #35
Version 0.1.2.66 is out, with simple XML highlighter for the output field.
find quote
UsagiYojimbo Offline
Member
Posts: 91
Joined: Feb 2010
Reputation: 2
Location: Debrecen, Hungary
Post: #36
Currently I am redesigning the handling of includes.
find quote
Marx1 Offline
Fan
Posts: 367
Joined: Jan 2011
Reputation: 3
Post: #37
I'm behind a proxy. How can I enable proxy settings in ScraperEdit?
find quote
UsagiYojimbo Offline
Member
Posts: 91
Joined: Feb 2010
Reputation: 2
Location: Debrecen, Hungary
Post: #38
(2013-03-20 09:57)Marx1 Wrote:  I'm behind a proxy. How can I enable proxy settings in ScraperEdit?
Currently, there are no proxy settings available. Maybe system wide proxy settings can be used.
Added this to my to-do list.

Edit:
If You are permanently behind that proxy, You may set up a system-wide proxy for Your Java. See Here.
(This post was last modified: 2013-03-22 07:33 by UsagiYojimbo.)
find quote
UsagiYojimbo Offline
Member
Posts: 91
Joined: Feb 2010
Reputation: 2
Location: Debrecen, Hungary
Post: #39
Something strange happened to me...
I wanted to test the new include system to see whether it is working the way I wanted to. Previously I used a scraper stub I created for testing. This time I wanted to use a working scraper, so I searched through the addons folder of my Raspbmc, and found no <include> tag .
That was strange!
I looked all around the scrapers, and did not found any use of the <include> tag. However scrapers are referencing to functions declared in the common libraries. EG: metadata.themoviedb.org/tmdb.xml uses this:
PHP Code:
<RegExp conditional="!keeporiginaltitle" input="$$2" output="<chain function=&quot;GetTMDBTitleByIdChain&quot;>$$2</chain>" dest="5+"
This GetTMDBTitleByIdChain function is declared in the metadata.common.themoviedb.org/tmdb.xml...
(This post was last modified: 2013-04-11 15:18 by UsagiYojimbo.)
find quote
scudlee Offline
Team-Kodi Member
Posts: 913
Joined: Jul 2011
Reputation: 58
Post: #40
The includes are defined in the addon.xml.
e.g.
PHP Code:
<requires>
    <
import addon="xbmc.metadata" version="2.1.0"/>
    <
import addon="metadata.common.themoviedb.org" version="2.9.0"/>
    <
import addon="metadata.common.movieposterdb.com" version="1.0.5"/>
    <
import addon="metadata.common.imdb.com" version="2.7.0"/>
    <
import addon="metadata.common.impa.com" version="1.0.3"/>
    <
import addon="metadata.common.hdtrailers.net" version="1.0.7"/>
    <
import addon="metadata.common.youtubetrailers" version="1.0.4"/>
    <
import addon="metadata.common.rt.com" version="1.4.0"/>
    <
import addon="metadata.common.ofdb.de" version="1.0.0"/>
    <
import addon="metadata.common.trakt.tv" version="1.0.0"/>
    <
import addon="metadata.common.port.hu" version="1.0.0"/>
  </
requires
find quote
UsagiYojimbo Offline
Member
Posts: 91
Joined: Feb 2010
Reputation: 2
Location: Debrecen, Hungary
Post: #41
(2013-03-23 11:09)scudlee Wrote:  The includes are defined in the addon.xml.
Yeah, I was afraid of this...
find quote
Eleazar Coding Offline
Donor
Posts: 187
Joined: Oct 2013
Reputation: 1
Location: Jerusalem, Israel
Post: #42
Going to give it a whack soon, this app looks nice though.

עם ישראל חי
Running XBMC on the M8 Amlogic S802, Amazon Fire TV, Apple TV 2, and Mac (in order of personal preference)
find quote
beamer145 Offline
Junior Member
Posts: 14
Joined: Jan 2013
Reputation: 0
Post: #43
I am trying to figure out why \AppData\Roaming\XBMC\addons\packages\metadata.universal\universal.xml is not able to detect the returned results, but when I run through the sequence with the debugger I get a failed to parse url error after the CreateSearchUrl phase.

Is that suppose to work already ?

Any other way to jump to start debugging the GetSearchResults part directly with my own test data as input (If I do debug it seems I need to start from a movie title) ?

Thanks

Edit: Ok, managed to locate the problem using the regexp tester and some copy pasting around, slightly less efficient than the real debug facilities but certainly doable (and provides more info wrt the regexp functionality itself, something that is currently not available in the debugger i think)
(This post was last modified: 2013-11-02 02:26 by beamer145.)
find quote
UsagiYojimbo Offline
Member
Posts: 91
Joined: Feb 2010
Reputation: 2
Location: Debrecen, Hungary
Post: #44
(2013-11-02 01:16)beamer145 Wrote:  I am trying to figure out why \AppData\Roaming\XBMC\addons\packages\metadata.universal\universal.xml is not able to detect the returned results, but when I run through the sequence with the debugger I get a failed to parse url error after the CreateSearchUrl phase.
XBMC handles XML in a somewhat fuzzy way, while ScraperEdit, Java and JAXB are really strict about rules.
Most of the current scrapers do not return correct XML, as they usually use character & instead of entity &amp;. (Which would be the correct behavior...)

(2013-11-02 01:16)beamer145 Wrote:  Is that suppose to work already ?
ScraperEdit handles XML files as described by the XML rules.

(2013-11-02 01:16)beamer145 Wrote:  Any other way to jump to start debugging the GetSearchResults part directly with my own test data as input (If I do debug it seems I need to start from a movie title) ?
No such way is currently available, and as the XBMC scraper handling is moved toward add-ons, I do not intend to develop further features.
find quote
hadee41 Offline
Junior Member
Posts: 1
Joined: Mar 2014
Reputation: 0
Post: #45
Here is really awesome and most exciting publish. I completely believe the fact with you and would like to say thanks to you that you discuss this publish here with us. Keep it proceed please.
find quote