Kodi Community Forum

Full Version: Local (SQLite) database much slower than MySQL after initial load (on Raspberry Pi)
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hi everyone,

First of all, huge shout-out to everyone involved with XBMC as a whole. Just awesome stuff. Big Grin

I am a long time XBMC user, but a very new Raspberry Pi user, having just got mine last week. So far, everything with Raspbmc has been great - with one exception: the speed of the movie library browser.

I have posted this on the Raspbmc forums too, but it may be something with XBMC core that the Raspberry Pi is just highlighting due to its slower processor speed, so I thought I'd ask over here too.

While I know that there are plenty of threads out there detailing how you speed things up when you have MySQL involved, I have the opposite problem. And I can't seem to find any solutions or comments on it!

Looking at the debug.log, when I first boot the Pi and go into the Movie Library Title list, I get the following:

Code:
RunQuery took 2349 ms for 665 items query: select * from movieview
08:17:39 T:3038351360   DEBUG: Saving fileitems [videodb://1/2/]
08:17:40 T:3038351360   DEBUG:   -- items: 665, sort method: 0, ascending: false

All seems fine and, even though there are 665 movies, it only takes 4-5 seconds and then the screen is displayed.

However, once this initial load is done - i.e. the data is saved from the MySQL central database to the XBMC local SQLite one on the Pi, the next time I go in to the Movies view, I get the following:

Code:
08:18:31 T:3038351360   DEBUG: Loading fileitems [videodb://1/2/]
08:18:31 T:2790732864   DEBUG: RunQuery took 14 ms for 0 items query: select * from musicvideoview  ORDER BY dateAdded desc, idMVideo desc LIMIT 10
08:18:31 T:2965681216   DEBUG: CecLogMessage - key auto-released: select (0)
08:18:31 T:2965681216   DEBUG: PushCecKeypress - received key  b duration 1206
08:18:51 T:2802070592   DEBUG: Thread Jobworker 2802070592 terminating (autodelete)
08:18:51 T:2827322432   DEBUG: Thread Jobworker 2827322432 terminating (autodelete)
08:18:51 T:2936628288   DEBUG: Thread Jobworker 2936628288 terminating (autodelete)
08:19:11 T:3038351360   DEBUG:   -- items: 665, directory: videodb://1/2/ sort method: 0, ascending: false

See the time gap between the first and last line - that would be 40 seconds. May not sound like a lot, but it sure feels like a lot! Especially as it is every time I go into the movie library view.

I'm at a loss to understand this. The initial load from the MySQL database into the SQLite and then the display takes way less time than any view of the same local database from that point on.

On my Core i5 PC, obviously there is next to no delay as it can process locally faster than over the network. But on the comparatively under-powered Pi, it's obviously struggling.

Does anyone have any tips on how to fix this? Or, dare I say it, how to get XBMC to just look at the MySQL database and not the local SQLite one? Like it does on first boot?

Any help is really appreciated.

P.S. for the tech details, I'm running a 512MB ver B Raspberry Pi with a Class 10 SDHC 16GB card. It's connected to the network via a cable to a gigabit switch. The MySQL server is running on a ReadyNAS. I'm running the nightly builds of Raspbmc, and I'm using XBMC Frodo RC 3.
Have I stumped the internet? Smile
Unless RPi works differently to the standard linux build, I don't believe that the data is stored locally at all in sqlite when using MySQL. I believe that all queries and updates are handled by MySQL calls.
According to your Debug Log it's taking 40 seconds to read off the "disk" from it's local cache (note the "Loading fileitems" line).

Perhaps ensure that that cache is on fast media?
Where is this cache located? I note on my x86 system that ~/.xbmc/userdata/Databases/Videos60.db hasn't been touched since October 2012, about the time I moved to mysql databases.

Textures6.db on the other hand has a very current timestamp.
~/.xbmc/cache perhaps, or just in /tmp/
Thanks for your replies!

(2013-01-09, 00:37)nickr Wrote: [ -> ]Unless RPi works differently to the standard linux build, I don't believe that the data is stored locally at all in sqlite when using MySQL. I believe that all queries and updates are handled by MySQL calls.
I'm just trying to interpret the debug.log where it says "DEBUG: Saving fileitems [videodb://1/2/]" when it does that first load, and then afterwards it is "DEBUG: Loading fileitems [videodb://1/2/]" - in other words, doesn't that mean that it is initially loading from MySQL into a local database (assumed SQLite) or cache?

(2013-01-09, 01:10)jmarshall Wrote: [ -> ]According to your debug log it's taking 40 seconds to read off the "disk" from it's local cache (note the "Loading fileitems" line).

Perhaps ensure that that cache is on fast media?
Hmmm...I'm running it on a Class 10 SDHC card, which I would have thought would be fast enough. However, I'll try a USB thumb-drive and see if that's faster.


(2013-01-09, 01:17)nickr Wrote: [ -> ]Where is this cache located? I note on my x86 system that ~/.xbmc/userdata/Databases/Videos60.db hasn't been touched since October 2012, about the time I moved to mysql databases.

Textures6.db on the other hand has a very current timestamp.
For what it's worth, I see that my MyVideos75.db is getting updated pretty frequently.
Ok, so I have some updates, and some more questions.

I tried changing the install over to a USB stick for faster write to the cache, but it didn't make any appreciable difference.

So I started to look into the cause of the slow-down in a bit more detail - including looking at the source code of XBMC.

It seems that when the Video library is displayed, there is a routine that checks to see if the library needs to be cached. If it does, it saves to a file in the xbmc/temp folder with a .fi extension. That's what the "saving file items" line in the log means.

The next time you go to the screen, it sees the file is there and reads from that local cache rather than get the data back from MySQL. For me, the time to write initially and display is about 24 seconds; the second time back (and every subsequent time) that it reads from the cache, it's closer to 45 seconds!

There is a constant that the code refers to - CACHE_IF_SLOW or something like that. There is also CACHE_ALWAYS and CACHE_NEVER. Does anyone know if there is a way to set the video database to never cache without forking and compiling my own copy of the code?
I can't answer that question, but Twilight Zone moment - looks like everyone in this thread is from NZ.

What part you from? ChCh here.
(2013-01-13, 05:32)nickr Wrote: [ -> ]I can't answer that question, but Twilight Zone moment - looks like everyone in this thread is from NZ.

What part you from? ChCh here.

I'm up in Auckland. It's a small internet after all. Smile