[RELEASE] Scraper Editor (Based on ScraperXML open source C# Library) - Help wanted!

[RELEASE] Scraper Editor (Based on ScraperXML open source C# Library) - Help wanted! - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32)
+--- Forum: Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=60)
+--- Thread: [RELEASE] Scraper Editor (Based on ScraperXML open source C# Library) - Help wanted! (/showthread.php?tid=52929)

Pages: 1 2 3 4 5 6 7 8 9 10 11

- Nicezia - 2009-08-29

Hopefully (if testing goes well this weekend, i'll have the new version out on Wednesday - lots of GUI changes, and Linear Testing (Walk through from CreateSearchUrl to the end of the scraper process- including custom functions) External function loading (provided you specify your XBMC scraper folder in settings - External functions are read only - I'm working on include editor as a seperate mode)

Added a lot of error handling, but I'll admit i haven't thought of every likely error scenario - however there shouldn't be a case where error will crash program, any error I've missed should describe itself in a Message Box, counting on users to note and report errors, so i can add appropriate handling.

also there is a wizard that will assist with creating standard functions will walk you through creating Create*SearchUrl, Get*SearchResults and Get*Details (Later i want to add a web-browser assisted mode to this - but that's a HUGE undertaking from the standpoint that i am looking at it)

Mind you the wizards are not mandatory, its just something I've added to make scraper writing easier for inexperienced scraper writers.

I hope the new changes will be received well.

- jurr - 2009-09-07

Hello Nicezia,

I'm unable to download version 1.5 from mediafire, and version 1.0 crash with error System.IO.FileNotFoundException: No se puede cargar el archivo o ensamblado 'System.Xml.Linq, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' ni una de sus dependencias. El sistema no puede hallar el archivo especificado. on windows XP.

Thanks in advance

- Nicezia - 2009-09-07

jurr Wrote:Hello Nicezia,

I'm unable to download version 1.5 from mediafire, and version 1.0 crash with error System.IO.FileNotFoundException: No se puede cargar el archivo o ensamblado 'System.Xml.Linq, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' ni una de sus dependencias. El sistema no puede hallar el archivo especificado. on windows XP.

Thanks in advance

Firstly You need .NET 3.5 SP1 (or mono 2.01 if linux)

Secondly I suggest downloading from sourceforge (link is in my signature)

- Nicezia - 2009-09-07

By the way the newest version will be out soon i found a buttload of problems testing

so far its almost a complete internal redesign to support the new features

- Daniel Malmgren - 2009-09-27

What happened here Nicezia?

Problem is, my scraper doesn't even load in the old editor, and I'm too damn lazy to fix it up in a usual text editor :p

/Daniel

- Nicezia - 2009-10-06

Still working out bugs on new features.
just about got it nailed down though, right now its the error handling that's got me tied down. Isolating all possible errors, and working on what to do in the case of these errors. Also, there's one thing that I may not be able to support right away, it may take me some time to figure out this problem, and that's RegExp's that are without a destination or expression(used simply for conditional statements - which tends to appear in some of the more important scrapers... ) at least till i figure out a constructive way to handle it.

version 3.5 Release - Nicezia - 2009-10-11

Release Notes:

Code:
Added Features:

*Editor Configuration options (svn option not working yet, select your local scraper

          folder before selecting to use scraper folder else Includes will be disabled)

*Settings Editor: Easily managing of scraper settings

*Tester Redesigned: Now all buffers are available at all times, Results of each function

          run are saved so that you can look over them, can validate results of Function. 

          Tester will show you the destination buffer of the currently selected RegExp 

          (so as to know which buffer to monitor)

*Scraper Walkthrough: Can run scraper from beginning to end, even allows for 

          Artist/Album selection in Music scrapers

*New Expression Field Manipulators (Encode & FixAmp)

          Encode: url encodes the field before writing to buffer

          FixAmp: (scraperXML dll exclusive) changes stray ampersands to "&amp;"

*Expression Test Text can now be loaded from Buffer, Clipboard, or file

*Expression Test - new window to view the matches and all captured fields for matches

Bugs: 

*Not exactley a bug but in Expression if you are creating a scraper to be used 

        with XBMC do not depend on the FixAmp fields as it is currently  a 

        scraperXML exclusive

*Help Info not yet added

*Use Svn Folders not working yet

*Translation options not yet added

*Doesn't work with tvdb.com scraper - reason being that the tvdb scraper uses 

          RegExp's as control statements which is not supported (and probably never

          will be), there are settings that are downloaded from internet (Soon as i 

          figure a creative way to work with that ability that will be supported), and it

          omits input on certain RegExp (which also will never be supported)

*Validate XML of buffers not functioning yet

*Run Scraper option "TV Episodes" will not work due to "GetEpisodeList" function being

          handled in a different manner from every other Function (linear instead of 

          chained into buffers - this would make it difficult to keep track of the results

          as it would overwrite the function results each time it finishes processing a page,

          however i am working on a creative alternative way to process this)

Download From SourceForge

Please Report any Bugs found in this thread or through email

Side Note: if anyone would like to design a icon and logo for scraperXML Editor let me know.

- Daniel Malmgren - 2009-10-11

Nicezia Wrote:Please Report any Bugs found in this thread or through email

Looks promising! And now it actually loads my scraper without crashing Nod

When running the tester I get the following exception though. Hope It says more to you than to me

Code:
System.ArgumentOutOfRangeException: Det angivna argumentet hamnade utanför intervallet med giltiga värden.

Parameternamn: index

   vid System.Windows.Forms.TreeNodeCollection.get_Item(Int32 index)

   vid ScraperXMLEditor.MainForm.MoveToFunction(String FunctionName)

   vid ScraperXMLEditor.MainForm.FindNextFunction(String currentFunction)

   vid ScraperXMLEditor.MainForm.ScraperTimer_Tick(Object sender, EventArgs e)

   vid System.Windows.Forms.Timer.OnTick(EventArgs e)

   vid System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)

   vid System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Sorry for the error being in swedish. Seems everything important is understandable in english though Rolleyes

This is running Windows 7. Haven't tried this version on any other OS yet.

/Daniel

- Nicezia - 2009-10-12

Daniel Malmgren Wrote:Looks promising! And now it actually loads my scraper without crashing

When running the tester I get the following exception though. Hope It says more to you than to me

Code:
System.ArgumentOutOfRangeException: Det angivna argumentet hamnade utanför intervallet med giltiga värden. Parameternamn: index vid System.Windows.Forms.TreeNodeCollection.get_Item(Int32 index) vid ScraperXMLEditor.MainForm.MoveToFunction(String FunctionName) vid ScraperXMLEditor.MainForm.FindNextFunction(String currentFunction) vid ScraperXMLEditor.MainForm.ScraperTimer_Tick(Object sender, EventArgs e) vid System.Windows.Forms.Timer.OnTick(EventArgs e) vid System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m) vid System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Sorry for the error being in swedish. Seems everything important is understandable in english though

This is running Windows 7. Haven't tried this version on any other OS yet.

/Daniel

I haven't got any way to test with windows 7

however its a index out of range error, which shouldn't happen - probably just something i missed along the way of editing - the whole program has become alot more complex than origionally intended, and keeping up with my own changes can be a pain in the butt, i've tested your scraper from svn and it works beautifully..

if you could give me a little more info on when it happens(what function, regexp it occurs on) and the working copy of your scraper (email) i could look into it deeper

- Daniel Malmgren - 2009-10-12

Nicezia Wrote:if you could give me a little more info on when it happens(what function, regexp it occurs on) and the working copy of your scraper (email) i could look into it deeper

I wish I could. What I did was simply to open the scraper (as found in svn) and push the "Run Scraper" button and wait for a while. Today I can't seem to reproduce the crash though, everything works like a charm. Can't really decide if I think that is good or bad...

/Daniel

- Nicezia - 2009-10-13

Daniel Malmgren Wrote:I wish I could. What I did was simply to open the scraper (as found in svn) and push the "Run Scraper" button and wait for a while. Today I can't seem to reproduce the crash though, everything works like a charm. Can't really decide if I think that is good or bad...

/Daniel

Damn?
or Yippie!

You pick one...
but if you run into the problem again let me know.

working on the svn portion now...

- flobbes - 2009-10-16

The "Find Matches" and "View Matches" Buttons aren't working for me.
I tried several different scrapers and windows and linux.

Anybody else experienced this as well?

- seedzero - 2009-10-16

flobbes Wrote:The "Find Matches" and "View Matches" Buttons aren't working for me.
I tried several different scrapers and windows and linux.

Anybody else experienced this as well?

Yep, same here. Also has anyone managed to install this program on a mac? I've had to use it under boot camp as I can't get it to run under wine. If anyone has then could they please tell me what dependencies etc are required? Thanks.

- fekker - 2009-10-17

redtapemedia Wrote:Yep, same here. Also has anyone managed to install this program on a mac? I've had to use it under boot camp as I can't get it to run under wine. If anyone has then could they please tell me what dependencies etc are required? Thanks.

try running it with mono 2.4.2.3

- seedzero - 2009-10-17

fekker Wrote:try running it with mono 2.4.2.3

tried that... installed mono via macports, and get

Code:
Unhandled Exception: System.TypeInitializationException: An exception was thrown by the type initializer for System.Windows.Forms.WindowsFormsSynchronizationContext ---> System.TypeInitializationException: An exception was thrown by the type initializer for System.Windows.Forms.ThemeEngine ---> System.TypeInitializationException: An exception was thrown by the type initializer for System.Windows.Forms.ThemeWin32Classic ---> System.TypeInitializationException: An exception was thrown by the type initializer for System.Drawing.KnownColors ---> System.TypeInitializationException: An exception was thrown by the type initializer for System.Drawing.GDIPlus ---> System.DllNotFoundException: gdiplus.dll

  at (wrapper managed-to-native) System.Drawing.GDIPlus:GdiplusStartup (ulong&,System.Drawing.GdiplusStartupInput&,System.Drawing.GdiplusStartupOutput&)

  at System.Drawing.GDIPlus..cctor () [0x00000] 

  --- End of inner exception stack trace ---

  at System.Drawing.KnownColors..cctor () [0x00000] 

  --- End of inner exception stack trace ---

  at System.Drawing.Color.get_Black () [0x00000] 

  at System.Windows.Forms.ThemeWin32Classic..cctor () [0x00000] 

  --- End of inner exception stack trace ---

  at System.Windows.Forms.ThemeVisualStyles..ctor () [0x00000] 

  at System.Windows.Forms.ThemeEngine..cctor () [0x00000] 

  --- End of inner exception stack trace ---

  at System.Windows.Forms.SystemInformation.get_MenuAccessKeysUnderlined () [0x00000] 

  at System.Windows.Forms.Control..ctor () [0x00000] 

  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control:.ctor ()

  at System.Windows.Forms.WindowsFormsSynchronizationContext..cctor () [0x00000] 

  --- End of inner exception stack trace ---

  at System.Windows.Forms.Control..ctor () [0x00000] 

  at System.Windows.Forms.ScrollableControl..ctor () [0x00000] 

  at System.Windows.Forms.ContainerControl..ctor () [0x00000] 

  at System.Windows.Forms.Form..ctor () [0x00000] 

  at ScraperXMLEditor.MainForm..ctor () [0x00000] 

  at (wrapper remoting-invoke-with-check) ScraperXMLEditor.MainForm:.ctor ()

  at ScraperXMLEditor.Program.Main () [0x00000]

in terminal. I'm guessing there's another dependency I'm needing, possibly windows forms?