Login at Kodi Home

jhoyos · (This post was last modified: 2021-02-23, 08:26 by mlaggner.)

Hi, using TMM v4.1 I think I might have a problem with movie database.

I used to have 25.000 movies with a DB sizing no more than 300mb, and right now, scanning my library again in v4.1, and have at this time 17.000 movies, and database size is higher than 1GB!

Besides, it goes really really slow...

Is this normal?

**mlaggner** · 2021-02-20, 14:22

I think the cause of your problem comes from: https://forum.kodi.tv/showthread.php?tid=360512 - the internal compacting (defragmentation) of the database engine we use is slightly broken resulting in a steady re-write/defragmentation of the whole database even if there are no more changes to it (causing massive file system I/O and CPU usage).

I've deactivated the internal defragmentation and implemented a lazy writing of the database along with manual defragmentation. This has been looking good for me and a few other users... In your case this approach might hit some limitation which was not the cause for us. Could you have a look at your CPU usage in this case? Maybe the internal write cycles are slower than the interval with that much movies

jhoyos · 2021-02-20, 15:17

CPU usage is normal (around 20%). Memory usage (with 16GB Ram) is however pretty high, around 80%. It is true that a I have a virtual machine with VMWare where I have qbittorrent running, but still it seems quite big.

Regarding the problem, which I really don't understand... Is there anything I should do?

It is a problem with v4.1 in itself? will I have the same problem in v4.07? I just need to know if it is better to cancel the process of scanning and do it again with v4.07, or once it finish, the database will be compacted and I will have a good database file...

**mlaggner** · 2021-02-20, 15:50

nothing you can do here - looks like I need to tweak the logic a bit more for really big databases. Unfortunately it looks like I have no testbed for this size of database and my laptop is too strong that I don't see any issue here Big Grin

maybe I get some time today to tweak that here - stay tuned

jhoyos · 2021-02-20, 17:31

Thanks for your response. I will stay tuned to see if the problem is solved, or at least gets better.

By the way... haven't you thought into a sqlite database so it can be accesed outside TMM and change some things directly? that would be really great, in fact.

Right now, I guess there is no way to access tmm database but using tmm, right?

**mlaggner** · 2021-02-20, 17:43

Since we do not use any relational database internally, you will not have any luck querying that from outside tmm (the DB relies on a key/value map by the developers of the H2 database, but it stores all the entities in JSON format do have the best flexibility for migration/speed). We have tried to use a relational database in the past, but you also need an ORM to morph the database entries into Java object (which is a pain in the ass to do it per hand). Long story short -> no direct access to the database possible, but you can use your export engine to bring out almost everything from the database in a format you like.

One more point -> SQLite is native code and would need us to mix Java (tmm itself) and native code which can hit us hard with multi OS compatibility

**mlaggner** · 2021-02-20, 21:56

okay, I've changed the optimization logic a bit - could you test the latest nightly build (should be finished in a few minutes) according to your problem?

@tars @bobrap could you test your use cases too please?

tars · 2021-02-21, 06:02

It works fine for me.

tars · 2021-02-21, 06:15

While scraping metadata, this error popped up though it seems working fine:

Code:
ERROR [Timer-0] org.tinymediamanager.Log4jBackstop:37 - Uncaught exception in thread: Timer-0

java.lang.IllegalMonitorStateException: null

    at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(Unknown Source)

    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown Source)

    at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(Unknown Source)

    at org.tinymediamanager.core.movie.MovieModuleManager.writePendingChanges(MovieModuleManager.java:241)

    at org.tinymediamanager.core.movie.MovieModuleManager$1.run(MovieModuleManager.java:161)

    at java.base/java.util.TimerThread.mainLoop(Unknown Source)

    at java.base/java.util.TimerThread.run(Unknown Source)

**mlaggner** · 2021-02-21, 09:12

@tars okay, I saw a small issue here, but as you said this has no negative influence in the persistence logic. This will be changed in the next build

tars · 2021-02-21, 11:29

If the error is not an important one, can you make it not showing up in a pop up window and just get it logged?

jhoyos · 2021-02-22, 13:40

It seems my database size has been reduced. Thanks for your help.