Solved Scraping: Number of Actors ...
#16
(2020-09-24, 07:40)mlaggner Wrote: just had a look at the code - the compacting is done when the database shuts down (tmm shuts down). So it should be "self cleaning/shrinking" when tmm finally shuts down

Well, that's not happening at least with my database.
I have a backup of the db (that needs to be compacted) and just tried restarting tmm 5 times (with some interval in between) but the db size remains exactly the same.
Shouldn't something have to trigger that cleaning/compacting process?
#17
If you remove items from the database (because you remove about 1/2 of the actors), the database _should_ shrink after compacting. I say _should_ here, because that are internals of the mvstore of the H2 database (we use that database engine, we did not write our own).

just read a bit in the source of the MVStore and I saw, that this has a versioning feature -> per default 5 versions of one object (movie) are kept in the database. Since I was not aware of this parameter, the default of 5 versions is active within tmm (although we do not make use of it). I will try to remove that
tinyMediaManager - THE media manager of your choice - available for Windows, macOS and Linux
Help us translate tinyMediaManager at Weblate | Translations at 66%
Found a bug or want to submit a feature request? Contact us at GitLab
#18
Great! Hopefully that fix could reduce the db size (and launching time) even further.
#19
I found out that this parameter is only used in in-memory-databases and thus does not apply to our use case.

I will observe this issue and make some tests with newer versions of the mvstore in the next weeks
tinyMediaManager - THE media manager of your choice - available for Windows, macOS and Linux
Help us translate tinyMediaManager at Weblate | Translations at 66%
Found a bug or want to submit a feature request? Contact us at GitLab
#20
v4 is already released?!
#21
inofficially Wink
you can already get it at https://release.tinymediamanager.org - it has just not been announced yet
tinyMediaManager - THE media manager of your choice - available for Windows, macOS and Linux
Help us translate tinyMediaManager at Weblate | Translations at 66%
Found a bug or want to submit a feature request? Contact us at GitLab
#22
Is this feature already released (the talk about shrinking the DB suggests yes). I am running Build 2020-09-28 05:22, but don't see an option to scrape only credited actors, or to limit actors to just those with images.

And if it is implemented, would rescraping the actors delete all images for non-credited actors, or would they have to be manually deleated somehow?

Thanks, Carl.
CoreELEC 9.03
Aeon Nox 5: SiLVO
Minix Neo U9-H, Logitech Harmony Elite
#23
now we only scrape credited actors from imdb (which is ~50% of the actors we've scraped before). If you use .actors you might have to clean them too (can't remember now whether we have a cleanup function for the .actors or not - too much code to remember Tongue). Just re-scrape actors only and you will get much less actors
tinyMediaManager - THE media manager of your choice - available for Windows, macOS and Linux
Help us translate tinyMediaManager at Weblate | Translations at 66%
Found a bug or want to submit a feature request? Contact us at GitLab
#24
(2020-09-29, 08:09)mlaggner Wrote: now we only scrape credited actors from imdb (which is ~50% of the actors we've scraped before). If you use .actors you might have to clean them too (can't remember now whether we have a cleanup function for the .actors or not - too much code to remember Tongue). Just re-scrape actors only and you will get much less actors
What about TMDB?
#25
there is no such workaround at tmdb now..

just split at a given amount of actors may split apart some "important" actors for movies which have a high amount of well known actors (just look at Sin City)... apart from that all actors look similar in the tmdb API (nothing to distinguish between "main" cast and "guests")

if we think a step further - how should that amount being entered by the user? just a "global" setting for all scrapers? in every scraper? I am no friend of cluttering the settings just for such a features (the settings are complex enough yet)
tinyMediaManager - THE media manager of your choice - available for Windows, macOS and Linux
Help us translate tinyMediaManager at Weblate | Translations at 66%
Found a bug or want to submit a feature request? Contact us at GitLab
#26
Wouldn't it be possible to just skip the actors that have "uncredited" in their role?
#27
My ideal would be tickboxes for "Credited Only" and "Actor Images Only", and then a separate field to scrape the first xxx actors only.

Thanks, Carl.
CoreELEC 9.03
Aeon Nox 5: SiLVO
Minix Neo U9-H, Logitech Harmony Elite
#28
(2020-09-24, 11:09)mlaggner Wrote: If you remove items from the database (because you remove about 1/2 of the actors), the database _should_ shrink after compacting. I say _should_ here, because that are internals of the mvstore of the H2 database (we use that database engine, we did not write our own).

just read a bit in the source of the MVStore and I saw, that this has a versioning feature -> per default 5 versions of one object (movie) are kept in the database. Since I was not aware of this parameter, the default of 5 versions is active within tmm (although we do not make use of it). I will try to remove that
 
(2020-09-24, 17:05)mlaggner Wrote: I found out that this parameter is only used in in-memory-databases and thus does not apply to our use case.

I will observe this issue and make some tests with newer versions of the mvstore in the next weeks

Yesterday I changed datasource for about 5% of my movie collection, and it ended up increasing the db size by 46%.
I haven't tried yet but I'm pretty sure if I re-create the library I will get the original smaller db size again.
Looks like the db is designed to keep old data as the name MVstore standing for multi-version store.

See some discussions about its size issue:
https://github.com/h2database/h2database/issues/301
https://github.com/h2database/h2database/issues/1820
#29
(2020-10-09, 20:01)tars Wrote: I haven't tried yet but I'm pretty sure if I re-create the library I will get the original smaller db size again.

Yep, it has now returned to the previous size after re-creating library.
How about adding "Compact Library" command to force compact/shrink library after a major update (changes only for hundreds of entries causes a significant size growth) since re-creating library takes a lot of time (including rebuilding cache)?
#30
Thanks for improving loading time with the v4 nightly!
Now it only takes about two-thirds of the previous loading time with my database. That's a great improvement!

Logout Mark Read Team Forum Stats Members Help
Scraping: Number of Actors ...0