Kodi Community Forum

Full Version: JAV Movie Scraper
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3 4 5 6 7 8 9
Hi Salinus

Thanks - unfortunately i am a git hub noob and couldnt find your fork - could you point me in the right direction as would love to try it
Wow Salinus! I downloaded your fork to try it out and your additions are really cool & useful.

I had dreaded refactoring the GUI code (it was on my todo list, but I was mostly prioritizing new features and bug requests and procrastinating on refactoring) so I really appreciate the time you took to do this and add all the other features that you did such as IAFD support, renaming, etc.

What does that dropdown for "choose specific parser" do? I don't see any entries in the list.

A bug I noticed is that if you have a folder which has the movie and it didn't conform to the name used in the rename settings and you then scrape & write the movie using rename settings, the .actor files don't move over to the new rename settings named folder. They stay in .actor folder in the original folder.

Do you want me to add you as a collaborator on github so you can push your changes directly into the main repo?
Hey Chuck,

Here's the link to to Sansibar's fork https://github.com/sansibar/JAVMovieScraper. There's no precompiled build on that page that Salinus has provided, but you can click download ZIP on that page to get the code and compile it yourself (this might be difficult without any programming experience). If Salinus wants I can probably also compile his build and upload it somewhere, but it might be better to wait for the changes Salinus provided to be merged into the main branch, assuming that's what Salinus prefers. I'm going to wait until Salinus responds to this thread to see what he wants to do.
Hello Laodong

I can add an option to turn off google translate for JAV files, but if I do so, I'd probably just scrape from DMM.co.jp and call it a day instead of using the other sites like javlibrary. This would make every field entirely in Japanese. Is that what you're looking for?
Hello DoctorD

Yep!That's exactly what feature I'm looking for!
I haven't tried Salinus' fork because i cannot compile. It would be great to have a pre-compiled version somewhere to try.

Meanwhile, in the main branch, i think the biggest problem with the current implementation is the Scrape, then Save business. It is very impractical and i think it would greatly improve the user experience if things were saved automatically with the Scrape JAV or Scrape JAV (Automatic).

You can then make the Scrape JAV button overwrite whatever is already saved and the Scrape JAV (Automatic) overwrite function could be an option that you turn on and off. You can also have an extra button to just remove all scraped items from the folder.
While I agree with you that having to write after scraping is somewhat weird, the reason I've been doing it that way so far is that as I was initially working on creating the program I wanted an easy way to be able to test each function separately or make changes in the intermediary steps without having to write things out twice.

Writing the actual files takes a while too, especially if downloading trailers and extrafanart, and if I integrate that into the scrape action it makes an already slow (>20 seconds sometimes to scrape some types of movie) process even slower, which is especially annoying if the scraper ends up finding the wrong file and you've then wasted a bunch of time.

As with everything, maybe an optional preference or something to control this would be best.

I've been really busy recently with work & other things and haven't had a lot of time I've been mentally able to commit to programming, so I'm really grateful that another contributor has added some cool new features. I may try to push them into the main github page so everyone can try them out, but I've noticed a few bugs with it, so I should probably create a development branch on github instead. Hopefully I'll have some time soon to try that out. I'm actually somewhat new to using git and writing GUI programs in general (I usually write server code), so it's always a bit of a learning process for me to get these things working!

I would just compile it for everyone to try out & use, but I'm not sure if Salinus' is comfortable with me doing that, so I'm going to wait a little longer to see if he responds.
OK cool... I guess maybe if there was some sort of progress bar for scraping and saving it would make it less awkward...As you know, i have around 1300+ Titles and it would be nice to just automate it and then go and fix the wrong ones after like in all other scraper programs Wink
For sure I need to add a progress bar and maybe some kind of message log or something for the program to communicate what it's doing while you're sitting there waiting. I've been kinda cheating when running it myself to get this functionality by examining console output for how the scrape is going, but there's no reason for any end user to be doing that since most people just double click icons to start the program.

I actually added a batch scrape feature a while back. Have you tried it out?

Select multiple items in the file list (hold shift key, select first item, select last item), then click scrape. If you're doing more than 5 or 6, you should probably use an automatic mode. Then after all that scraping is finally done, click write file. It should write everything all at once.

That said, with no progress bar, you may want to test it out in batches of 25-50 or so and see how it goes.
OK, I've got Salinus' changes merged into the main scraper download, so you can download the new version from my mediafire link on github. I also fixed a few bugs from his code that I noticed (such as the specific parser drop down not populating when running from a precompiled jar and an issue with the new preferences he added not saving properly to the settings.xml).

He actually added some cool new stuff such as support for these sites:

AV Entertainment
Tokyo Hot

You'll see them under a new drop down called "Choose specific parser". If you want to use one of these scrapers, just click the scrape button under this area. He didn't offer any documentation for what the file name has to be, but from what I can tell it requires that the ID be in the name of your file. Each site has its own naming convention for their ID. I'm not familiar with each of these sites, so this is what I've gotten from reading the code and trying it out:

For Aventertainments, it follows the usual JAV id naming structure like ABC-123. It does a search on the site using this ID.
For Kin8tengoku, you'll see the ID in the URL. It is the numeric part before /pht/.
For tokyohot, it appears the ID follows the format of n123 or n1234. In other words, a lowercase n followed by a 3 or 4 digit number.
(2014-09-25, 13:55)laodong0112 Wrote: [ -> ]Hello DoctorD

Yep!That's exactly what feature I'm looking for!

OK, I've added a new preference in the menu which allows you to scrape in Japanese instead of English. It doesn't work with any of the site specific scrapers yet, but if the movie is on dmm.co.jp, it should work.
Hi DoctorD!
The new feature working great!Thank u very much!
I've added a progress bar while scraping so you can see what's going on. There's also a cancel button there - handy to use if a site times out or something.

I've also put in an improvement to scraping logic for JAV movies which should prevent sites which returned a movie not having the same ID as the DMM movie scraped from factoring into the final results. This should help to remove the odd title or actor list showing up that is for completely the wrong movie.
Hello DoctorD!
I tried your program, seems to be nice,but data18 scraper dies at me after choosing the tumbs for scenes. I checked with Eclipse,
the thumbnail causes a nullpointer exception. It seems can not get the full size picture from the web site.
Does anyone else have this problem?
(I tired to modify the source, but I've just started with Java, so I couldn't fix it Smile yet Smile)))
Similarly for me
Pages: 1 2 3 4 5 6 7 8 9