[alterior]

loto_bak,

i get the same error when i try to access a large song list(~13k i tested it with lower numers and it worked sporadically after 10k). which is what i thought the thread was about. sorry but i am having the problem with the large song lists with the 2-6 build, same log as you posted originally.
.
there does not seem much in the code that would trigger the catch side so mystiriously.

thanks for all your help.

[kraqh3d]

loto,
i'm able to open songs without issue and at the moment i have 11,281 songs in my database. in the past, i was able to open songs with over 15,000 songs in the database. if i find time tomorrow, i'll rescan all my directories which'll bring the song total back up over 15k.

alterior,
theres not much that can be done about very large databases. there just isnt enough memory. but i think the crash point is around 20,000 songs. 13,000 should be ok.

[Loto_Bak]

watching memory usage it really doesnt seem to be a memory issue. i was able to view all my songs with a older build without issue.

if your tables dont exibit the same behavior ill strip some sensitive information out of my db and send you a copy.

the database really shouldnt take up that much memory. there is 20+ megs of ram available. the database is about 2.06mb (granted its compressed) but still, it should not be an issue to do that query.

[alterior]

kraqh3d,

were you able to load you db to see if you are getting the error with your full list? i tried the 2-12 build and it still has the problem, first reported in this thread. i think my db is somewhere around 15-16k songs, and it stops working around 12k songs.

i think it should work as well since my whole db is 3.75mb i dont think it's a memory issue either.

any help will be greatly appreciated.

[kraqh3d]

sorry, no i didnt, but i will right now. and the physical size of the database is irrelevant. i am pretty certain the issue is memoryu related. additional memory is used by sqlite during the query for its processing, and i dont think this memory usage is being accurately reported by xbmc by the freemem indicator.

[Loto_Bak]

would it be possible to whip up a quick piece of code to dump a bunch of relevent debug info on sql query fail?

[kraqh3d]

15-02-2006 19:48:14 debug cmusicdatabase::getsongsnav() query: select * from songview
15-02-2006 19:48:17 error cmusicdatabase::getsongsnav() failed

yeah, this is definately memory related. i now cant open music with 15,378 songs in it. though, at one time, after some optimizations, i know i was able to. i guess our memory consumption went up again.

loto, i'm not sure. the query uses code in the sqllite library, not in xbmc's code. even if there was, there's not much that can be done about it. the xbox still has a finite amount of memory.

a while back when the upper limit of songs was about 10k items, i was contemplating a "paging" system as a work around. for instance, when you enter songs, it would only fetch the first 1000 songs, and there would be a virtual folder item for "next page" and "previous page" to get the next and previous 1000 songs. the problem is that the sorting routines would only work for the items in the currently displayed page.

** edit **

i just did a test with sqlite3explorer on my pc with my database. the app, itself, when opened takes up 8,736k of memory. just opening the music database kicked up the memory consumption to 8,952k. i then ran the same query. while the query was churning, memory usage peaked over 42,000k. i didnt get the exact value but i did it several times and it was always over 42,000. once the query was finished, the app was using 32,776k of memory to keep the return table in memory.

[Loto_Bak]

how about if it does multiple queries and reassembles the listcontrol in the background? i dont believe xbmc has an issue with the large list, its just sqllite right?

[kraqh3d]

the xbox has finite memory resources so, yes, there is also limit on how large the list/thumb control can be. but, i'll try it out. it'll be slower, but it should work until someone has so many songs that they reach that upper limit. and i can probably code it so that whatever songs its already got before it runs out of memory, it keeps.

[alterior]

loto,

that's excellent idea, i was about to propose the same. i did some tests with the command line query tool and the memory did not go above 3mb.(which is roughly the size of the data). not sure if that makes a diffirence and it was the ver 3 of the sqlite.

in any case i thought checking first to see if the db has over 10k of songs then use the split/reassemble method at 10k at a time, or whatever the best number works out to be, will help folks with smaller databases, form having to deal with the slow down. however, it might not make a speed difirence.

also can we get some kind of progress dialog during all this ;-)

[kraqh3d]

actually, i was going to play around and have it read the data out of the database in 5k chunks regardless of the size of the database.

** edit **
i just did a quick proof of concept using 1k chunks and it works. rather well. i was able to display 15,379 songs:

16-02-2006 11:25:05 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 0
16-02-2006 11:25:06 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 1000
16-02-2006 11:25:07 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 2000
16-02-2006 11:25:08 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 3000
16-02-2006 11:25:09 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 4000
16-02-2006 11:25:10 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 5000
16-02-2006 11:25:11 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 6000
16-02-2006 11:25:12 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 7000
16-02-2006 11:25:14 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 8000
16-02-2006 11:25:15 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 9000
16-02-2006 11:25:17 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 10000
16-02-2006 11:25:19 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 11000
16-02-2006 11:25:21 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 12000
16-02-2006 11:25:23 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 13000
16-02-2006 11:25:25 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 14000
16-02-2006 11:25:27 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 15000
16-02-2006 11:25:28 debug cmusicdatabase::getsongsnav() query: select * from songview limit 1000 offset 16000
16-02-2006 11:25:31 debug saving fileitems [musicdb://4/]
16-02-2006 11:25:33 debug -- items: 15379, sort method: 7, ascending: true

[kraqh3d]

i changed the batch size to 5000 songs, and committed the change to cvs. i *tried* to code it such that if you run out of memory during the process, whatever songs it's already fetched will still be available to you. if anyone has alot of songs (> 20,000), please try to make it fail and let me know if it works. though, be warned that this may leave xbmc in a very precarious situation as there will be very little memory left. (the song list will be using a large portion of the memory.)

[Loto_Bak]

great kraq! thanks very much
ill try and put it through its paces this evening. will let you know if i find any problems

[DaveyWavey]

hello all, my first post. and on an annoying problem at that.
i have been reading the many different forum posts related to daap and itunes sharing. i have experimented with all the possible solutions and from i have seen with my own experience is that you guys in this forum are bang on with the problem. i tried last night installing itunes on my laptop and only added a minimal amount of songs and it worked no problems whatsoever.

my big library has over 17,000 files and sure enough it craps out every time, i am very interested in trying this fix you created.

i consider myself an intermediate when playing around with code, but catch on fast so if i need tools i will get them if i have to edit files i will, i just want to get this to work the way it was supposed to.

i did the smb share and yeah that works but i don't get the album artwork and i have doubles showing up in some groups (clean time i know) but i wouldn't have to do that if the daap works ok.

thanks,
davey wavey

[kraqh3d]

daveywavey,
i just want re-iterate that this only fixes the "songs" option off the root listing of the music library window, and not necessarily any issues with daap. if you are accessing a daap:// share, from files view, this will do nothing for you.