Solved Issue with MySQL & large music libraries resulting in duplicate artists
#13
(2018-06-22, 07:59)DaveBlake Wrote: OK, very useful data @hpcertpro  thank you. I can see the records in part1 are not in id order, hence the next chunck could well return duplicates. However it looks like you have accidentally duplicated the part3 JSON file as part2. I'm sure you have the part2 as you describe ({"end":1500,"start":750}) but it would be nice to have it too for completeness. Just the 17.6 file is all I need.
 Done: downgraded to 17.6 &  re-downloaded the second chunk, sorry for the mess-up.

Here are the full logs again for reference: with part2 corrected: https://filebin.net/9v3kprt15ulh88s9

One thing I notice is that the query responses from mysql (and hence JSON), even if they have the duplicates, at least they are consistent. They don't change from run to run. Also as you mentioned, 17.6 and 18.0 responses are exactly the same.

I tried the SQL queries. The first one works (I think it's the same that Kodi uses today), the second one errors out though:
 
Quote:SQL Error (1064): You have an error in your SQL syntax; check the manual that corresponds to your MaridaDB server version for the right syntax to use near 'ORDER BY artistview.idArtist' at line 9

Line 9 is the 'order' line.

For reference, this is the version of mariadb in the raspberry (OS = OSMC which is raspbian = Debian Stretch)
Quote:$ apt show mariadb-server
Package: mariadb-server
Source: mariadb-10.1
Version: 10.1.26-0+deb9u1
Installed-Size: 62
Maintainer: Debian MySQL Maintainers <[email protected]>
Architecture: all
Depends: mariadb-server-10.1 (>= 10.1.26-0+deb9u1)

So I tweaked the query a bit like so:
Quote:SELECT artistview.* FROM artistview
WHERE ((
EXISTS (SELECT 1 FROM song_artist
WHERE song_artist.idArtist = artistview.idArtist AND song_artist.idRole = 1)
OR EXISTS (SELECT 1 FROM album_artist WHERE album_artist.idArtist = artistview.idArtist))
AND (artistview.strArtist != ''))
AND (artistview.strArtist <> 'Various artists')
ORDER BY artistview.idArtist
LIMIT 750 OFFSET 1500

This one actually actually works and solves the problem!

Part1: LIMIT 750 OFFSET 0
Part2: LIMIT 750 OFFSET 750
Part3: LIMIT 750 OFFSET 1500

(or: LIMIT {[offset,] row_count)

Without the ORDER BY, I could confirm repeated artists in the SQL query output between Part2 and Part3, and ALSO between Part1 and Part3.
Adding the ORDER BY statement solves the issue.

Let me me know if there is a chance to get the change into a nightly so I can test end-to-end (JSON queries and ultimately the app)
Reply


Messages In This Thread
RE: Issue with MySQL & large music libraries resulting in duplicate artists - by htpcero - 2018-06-22, 15:11
Logout Mark Read Team Forum Stats Members Help
Issue with MySQL & large music libraries resulting in duplicate artists0