Kodi Community Forum
ScraperEdit for XBMC (Java) - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=60)
+--- Thread: ScraperEdit for XBMC (Java) (/showthread.php?tid=145204)

Pages: 1 2 3 4 5


RE: ScraperEdit for XBMC (Java) - Michaehm - 2013-02-09

It had some bugs
Image
Image


RE: ScraperEdit for XBMC (Java) - UsagiYojimbo - 2013-02-10

(2013-02-09, 09:02)Michaehm Wrote: It had some bugs
Image
Image

Sorry, the images are not accessible. Could You repost them?


RE: ScraperEdit for XBMC (Java) - Marx1 - 2013-02-26

Today I've tried 4 scrapers: universal, tmdb, filmwebm filmweb-lite. I've searched for "avatar". None of them gave me any result:
- tmdb 3.7.2 - no results found
- universal 2.4.1 - no results found
- filmweb 2.0 - no results found
- filmweb-lite 1.0.11 - error parsing xml (discussed previously, so not important)

I properly imported your sources and executed xbmc.scraper.Main
Seems that while I get
Code:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<results>
    <entity>
        <title> Avatar </title>
        <url cache="filmweb-299113">http://www.filmweb.pl/Film?id=299113</url>
    </entity>
    <entity>
        <title> Avatar 2 </title>
        <year>2015</year>
        <url cache="filmweb-625427">http://www.filmweb.pl/Film?id=625427</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>2004</year>
        <url cache="filmweb-152652">http://www.filmweb.pl/Film?id=152652</url>
    </entity>
    <entity>
        <title> Awatar / Avatar </title>
        <year>2007</year>
        <url cache="filmweb-480960">http://www.filmweb.pl/Film?id=480960</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>2003</year>
        <url cache="filmweb-238814">http://www.filmweb.pl/Film?id=238814</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>1915</year>
        <url cache="filmweb-5994">http://www.filmweb.pl/Film?id=5994</url>
    </entity>
    <entity>
        <title> Avanak Apti </title>
        <year>1978</year>
        <url cache="filmweb-261715">http://www.filmweb.pl/Film?id=261715</url>
    </entity>
    <entity>
        <title> Avanak kuzenler </title>
        <year>2008</year>
        <url cache="filmweb-493936">http://www.filmweb.pl/Film?id=493936</url>
    </entity>
</results>
in this code:
Code:
final Results rr = (Results) um.unmarshal(new StringReader(res));
parsing results fails and I get "No results found!"


Also there is an small GUI error: if I leave open debug window and load another scraper, executing debug from this opened window gives duplicated logs. Opening another scraper without closing this windows causes logs x3 during debugging


RE: ScraperEdit for XBMC (Java) - UsagiYojimbo - 2013-03-03

(2013-02-26, 23:23)Marx1 Wrote: Today I've tried 4 scrapers: universal, tmdb, filmwebm filmweb-lite. I've searched for "avatar". None of them gave me any result:
- tmdb 3.7.2 - no results found
- universal 2.4.1 - no results found
- filmweb 2.0 - no results found
- filmweb-lite 1.0.11 - error parsing xml (discussed previously, so not important)
There was a typo in the sources. (For some reason Huh i used entry instead of entity...)

(2013-02-26, 23:23)Marx1 Wrote: I properly imported your sources and executed xbmc.scraper.Main
Seems that while I get
Code:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<results>
    <entity>
        <title> Avatar </title>
        <url cache="filmweb-299113">http://www.filmweb.pl/Film?id=299113</url>
    </entity>
    <entity>
        <title> Avatar 2 </title>
        <year>2015</year>
        <url cache="filmweb-625427">http://www.filmweb.pl/Film?id=625427</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>2004</year>
        <url cache="filmweb-152652">http://www.filmweb.pl/Film?id=152652</url>
    </entity>
    <entity>
        <title> Awatar / Avatar </title>
        <year>2007</year>
        <url cache="filmweb-480960">http://www.filmweb.pl/Film?id=480960</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>2003</year>
        <url cache="filmweb-238814">http://www.filmweb.pl/Film?id=238814</url>
    </entity>
    <entity>
        <title> Avatar </title>
        <year>1915</year>
        <url cache="filmweb-5994">http://www.filmweb.pl/Film?id=5994</url>
    </entity>
    <entity>
        <title> Avanak Apti </title>
        <year>1978</year>
        <url cache="filmweb-261715">http://www.filmweb.pl/Film?id=261715</url>
    </entity>
    <entity>
        <title> Avanak kuzenler </title>
        <year>2008</year>
        <url cache="filmweb-493936">http://www.filmweb.pl/Film?id=493936</url>
    </entity>
</results>
in this code:
Code:
final Results rr = (Results) um.unmarshal(new StringReader(res));
parsing results fails and I get "No results found!"
See above...

(2013-02-26, 23:23)Marx1 Wrote: Also there is an small GUI error: if I leave open debug window and load another scraper, executing debug from this opened window gives duplicated logs. Opening another scraper without closing this windows causes logs x3 during debugging
Corrected this one, too.

New version of 0.1.2.65 is out.


RE: ScraperEdit for XBMC (Java) - UsagiYojimbo - 2013-03-05

Version 0.1.2.66 is out, with simple XML highlighter for the output field.


RE: ScraperEdit for XBMC (Java) - UsagiYojimbo - 2013-03-16

Currently I am redesigning the handling of includes.


RE: ScraperEdit for XBMC (Java) - Marx1 - 2013-03-20

I'm behind a proxy. How can I enable proxy settings in ScraperEdit?


RE: ScraperEdit for XBMC (Java) - UsagiYojimbo - 2013-03-21

(2013-03-20, 09:57)Marx1 Wrote: I'm behind a proxy. How can I enable proxy settings in ScraperEdit?
Currently, there are no proxy settings available. Maybe system wide proxy settings can be used.
Added this to my to-do list.

Edit:
If You are permanently behind that proxy, You may set up a system-wide proxy for Your Java. See Here.


RE: ScraperEdit for XBMC (Java) - UsagiYojimbo - 2013-03-23

Something strange happened to me...
I wanted to test the new include system to see whether it is working the way I wanted to. Previously I used a scraper stub I created for testing. This time I wanted to use a working scraper, so I searched through the addons folder of my Raspbmc, and found no <include> tag .
That was strange!
I looked all around the scrapers, and did not found any use of the <include> tag. However scrapers are referencing to functions declared in the common libraries. EG: metadata.themoviedb.org/tmdb.xml uses this:
PHP Code:
<RegExp conditional="!keeporiginaltitle" input="$$2" output="&lt;chain function=&quot;GetTMDBTitleByIdChain&quot;&gt;$$2&lt;/chain&gt;" dest="5+"
This GetTMDBTitleByIdChain function is declared in the metadata.common.themoviedb.org/tmdb.xml...


RE: ScraperEdit for XBMC (Java) - scudlee - 2013-03-23

The includes are defined in the addon.xml.
e.g.
PHP Code:
<requires>
    <
import addon="xbmc.metadata" version="2.1.0"/>
    <
import addon="metadata.common.themoviedb.org" version="2.9.0"/>
    <
import addon="metadata.common.movieposterdb.com" version="1.0.5"/>
    <
import addon="metadata.common.imdb.com" version="2.7.0"/>
    <
import addon="metadata.common.impa.com" version="1.0.3"/>
    <
import addon="metadata.common.hdtrailers.net" version="1.0.7"/>
    <
import addon="metadata.common.youtubetrailers" version="1.0.4"/>
    <
import addon="metadata.common.rt.com" version="1.4.0"/>
    <
import addon="metadata.common.ofdb.de" version="1.0.0"/>
    <
import addon="metadata.common.trakt.tv" version="1.0.0"/>
    <
import addon="metadata.common.port.hu" version="1.0.0"/>
  </
requires



RE: ScraperEdit for XBMC (Java) - UsagiYojimbo - 2013-03-24

(2013-03-23, 11:09)scudlee Wrote: The includes are defined in the addon.xml.
Yeah, I was afraid of this...


RE: ScraperEdit for XBMC (Java) - Eleazar Coding - 2013-10-10

Going to give it a whack soon, this app looks nice though.


RE: ScraperEdit for XBMC (Java) - beamer145 - 2013-11-02

I am trying to figure out why \AppData\Roaming\XBMC\addons\packages\metadata.universal\universal.xml is not able to detect the returned results, but when I run through the sequence with the debugger I get a failed to parse url error after the CreateSearchUrl phase.

Is that suppose to work already ?

Any other way to jump to start debugging the GetSearchResults part directly with my own test data as input (If I do debug it seems I need to start from a movie title) ?

Thanks

Edit: Ok, managed to locate the problem using the regexp tester and some copy pasting around, slightly less efficient than the real debug facilities but certainly doable (and provides more info wrt the regexp functionality itself, something that is currently not available in the debugger i think)


RE: ScraperEdit for XBMC (Java) - UsagiYojimbo - 2014-01-23

(2013-11-02, 01:16)beamer145 Wrote: I am trying to figure out why \AppData\Roaming\XBMC\addons\packages\metadata.universal\universal.xml is not able to detect the returned results, but when I run through the sequence with the debugger I get a failed to parse url error after the CreateSearchUrl phase.
XBMC handles XML in a somewhat fuzzy way, while ScraperEdit, Java and JAXB are really strict about rules.
Most of the current scrapers do not return correct XML, as they usually use character & instead of entity &amp;. (Which would be the correct behavior...)

(2013-11-02, 01:16)beamer145 Wrote: Is that suppose to work already ?
ScraperEdit handles XML files as described by the XML rules.

(2013-11-02, 01:16)beamer145 Wrote: Any other way to jump to start debugging the GetSearchResults part directly with my own test data as input (If I do debug it seems I need to start from a movie title) ?
No such way is currently available, and as the XBMC scraper handling is moved toward add-ons, I do not intend to develop further features.


RE: ScraperEdit for XBMC (Java) - hadee41 - 2014-03-26

Here is really awesome and most exciting publish. I completely believe the fact with you and would like to say thanks to you that you discuss this publish here with us. Keep it proceed please.