Music Library Scan doesn't capture all songs
#16
Are you sure it's _exactly_ 250? If so, that's an extremely interesting datapoint, but to be honest makes no sense whatsoever.

Are you certain it's not something silly like the NAS failing to send more than 250 items over? (Last time you scanned, it wasn't on the NAS, right?)
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#17
Appreciate the quick response.

When the audio library exceeded the HDD capacity (currently at ~79K files), I moved to using the NAS exclusively - around post-11xxx builds.

At first I thought it might be a PMIII skin cosmetic issue, but switching to a different skin or restarting the XBOX yielding the same results - performed library clean and/or DB deletion with restarts in-between.

As each <artist> folder has a varying amount of <album> sub-folders and audio files, you are correct that it could be concidental that it seems to run into to issues at 250+ <artist> folders. I had a look at the last <album> sub-folder in the last <artist> that was added to the library and the <album> was incomplete. Selecting 'Scan To Library' on the <artist> folder adds the missing tracks and in some cases, the missing <album>s.

Here's a file/folder count the breakdown for \\DNS-323\Music\A\:

Total Folders = 903
<artist> Folders = 255

TBN = 648
JPG = 1,148
MP3 = 3,2


It seemed to have stopped adding the files (MP3, TBN, etc.) to the library once it exceeded 4,000 ... not sure the exact number.
Reply
#18
Most strange. I'm not sure what it could be to be honest. There's another report on trac that is reporting a similar situation, so it appears you're not alone.

http://trac.xbmc.org/ticket/5633

The key is going to get it reproducible. If I can't reproduce, then it's tricky for me to do much about it. Can you run on win32 as well? If so, I'll add copious amounts of debug logging and upload another test build to that trac ticket.

Does it pick up the rest of the songs on an "Update Library"?

Cheers,
Jonathan
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#19
jmarshall Wrote:The key is going to get it reproducible. If I can't reproduce, then it's tricky for me to do much about it. Can you run on win32 as well? If so, I'll add copious amounts of debug logging and upload another test build to that trac ticket.

Not a problem; should I download 'XBMC_for_Windows-8.10.exe' or is there a different build that you need me to run? I'm not sure I have all the correct tools to compile a Win32 build from the PORT SVN tree. I did try to build the Win32 Release from the XBMC SVN tree but it dies a horrible death ... figured the reason was support for it has been dropped.


[EDIT] I had a quick look at the Trac and the log snippets and noticed that some of the folders seem to contain charactes such as comma's (,) - which used to cause me no end of grief on the FATX filesystem. As a result, I pruned my library files & folders so as to not exceed 42 characters (inlcuding .mp3) and only have the following characters: - ' . a-z A-Z 0-9. That said, I suspect you're going to say that it doesn't matter on non-FATX filesystems and that any length and characters are fine.


jmarshall Wrote:Does it pick up the rest of the songs on an "Update Library"?

I ran 'Update Library' and checked each folder (0,A-Z) and it does appear to pick up the missing <artist>s and/or <album>s.
Reply
#20
kzr1y2 Wrote:That said, I suspect you're going to say that it doesn't matter on non-FATX filesystems and that any length and characters are fine.

Each file system has it's own limitations to the length of the path of a file. NTFS for example has a limitation of 255 characters, in the path, not specifically the filename itself.
Reply
#21
Hi Maxim,

Appreciate the comments; yes, each file system does have limitations - EXT2, EXT3, NTFS, FAT, FAT32, FATX, etc.

I currently have the DNS-323 formatted in RAID 1 EXT2 and the XBOX is currently formatted as FATX - 300GB HDD so no special 'mod' formatting was required.

The current path structure (including the filename) in my case would never exceed 170 characters. As you're probably aware, FATX also seemed to be problematic with files/folders that contained non-numeric/alphabetic characters such as commas (,) - which would not necessarily be an issue for other filesystems such as EXT2 and NTFS.

Very early versions of XBMC used to have issues with 'non-standard' file/folder names on external sources; I suspect that isn't so much a problem now ... but just in case Wink
Reply
#22
Excellent, so we now have 2 data points for the same bug. The key now will be getting it reproducible or at least tracking it down a bit further.

I'll try and do up a test exe, and will post it in the trac ticket when I have it ready. Obviously you'll need the rest of a build installed - I suggest grabbing one of Jester's builds as a base.

Filenames and/or folders shouldn't matter: It's not a problem with anything failing to scan, as they all scan in with an update library (or multiple update libraries). The problem appears that during a large scan, something goes wrong which stops things being added to the library. Tracking this down is the tricky bit.

Cheers,
Jonathan
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#23
Hi Jonathan,

I'll need some help tracking down the latest build from Jester ... not sure where to look.

I'll pull the EXE from the Trac once I see your post in this thread.

Cheers,
Reply
#24
kzr1y2 Wrote:Hi Jonathan,

I'll need some help tracking down the latest build from Jester ... not sure where to look.

You can find jester's builds here: http://ocs.nl/xbmc/

-stoli-
Reply
#25
Much appreciated 'stoli' ... jonathan, will download and install 'MCSetup-Rev17788-jester.exe'.

Cheers,
Reply
#26
Good News !!!

I can reproduce the problem with Jester's Win32 Build 17788 ... and issuing 'Update Library' does add the missing content from 'Scan To Library'.

Cheers,

KZ
Reply
#27
Maxim Wrote:Each file system has it's own limitations to the length of the path of a file. NTFS for example has a limitation of 255 characters, in the path, not specifically the filename itself.

Actually that is not true. This limitation is not part of NTFS, but part of old api (from FAT days). Basically, if you access NTFS using FAT api, you get the limitation, and it aint 255, but 251 (dont ask me why).

NTFS, when accessed with the new UTF8 api supports 4000+ character paths. (too bad most applications still use old apis).

cheers
Reply
#28
@jmarshall:

Installed Jester's SVN 18004 (Feb 23, 2009) build, deleted the Music database and Thumbnails from the userdata folder. Issued 'Clean Library' from Music Settings.

- Issued a 'Scan To Library': not all files added to library
- Issued a 'Update Library': remaining (missing) files added to library


Let me know if you would still like me to run a 'full logging' enabled EXE.

Cheers,

KZ
Reply
#29
Hiho,

i played a bit around with the XBMC sources and compiled a version with some SQLite output.

I submitted the logfile and the db to the trac ticket.

It seems that sqlite is executing each query and outputting result is fine. But in fact it isn't! At some point each subsequent db query isn't just executed/stored to the db file?!

The log entry of the last song being inserted is on line 2239. Because my C++ skills are very limited i can't investigate why SQLite shows this behaviour when the audio files are organized in subdirectories instead of being all in one directory.

I hope this information helps

Greetings

TheNME123

P.S.: I deleted some albums that were scanned and inserted correctly before. Afterwards i scanned the directory again and now some of the files that were not inserted before got inserted. In the songs table there are always exactly 502 entries. It stops exactly after this amount of songs (although the album is not completly inserted to the db).
Reply
#30
kzr1y2: What I really need is some idea of why this is occuring, as until I can reproduce it, it's going to be very difficult to fix it.

So useful information would be:

1. Is it the number of directories involved? To test, combine a bunch of your folders and see if it gets to a different folder.

2. Is it the number of files involved? Remove some files early on in the scanning, and see if it gets further.

I'll see if I can do up a test build in a little bit with some more logging that may help us track it down further.

Cheers,
Jonathan
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply

Logout Mark Read Team Forum Stats Members Help
Music Library Scan doesn't capture all songs0