+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Music Support (https://forum.kodi.tv/forumdisplay.php?fid=263)
+--- Thread: Solved Kodi deleting or modifying previous entry when scanning in similar song (/showthread.php?tid=341950)
Kodi deleting or modifying previous entry when scanning in similar song - cmsimike - 2019-03-13
Hi!
I've been tracking down this, what I believe, to be bug down for a while now.
I've noticed that after a library scan, if Kodi scans in a similarly named song in a different location, it will either modify the previous entry by modifying the userrating and/or iTimesPlayed (at least, this is all I've been tracking) and add a new entry for the newly found song or completely get rid of the previous entry and adding the new song location. I currently have examples of both after a library scan.
I have confirmed that the songs that have had their db song entries modified or deleted to still be on the samba there and working without a problem.
I have the full kodi.log of the music library scan (12 MB - large backlog of scans because I wanted for 18.x to be released), I have CSV snapshots showing any song that has an itimesplayed or userrating > 0 showing the differences between the before and after library scan, and I currently have my previous snapshot of the mysql database and the current snapshot of the mysql database.
As indicated I won't post logs (and I'll take that to not post any images about the problem as well) but I am happy to provide whatever info needed tom help resolve this/prevent this from happening.
RE: Kodi deleting or modifying previous entry when scanning in similar song - DaveBlake - 2019-03-13
This may well be expected behaviour, it all depends on what you mean by "similarly named song in a different location". While I don't want huge logs and db comparisons (at this point if ever), you do need to give me some details to go on. An example some files where this happens and the details of how they are tagged is all I need.
The music library is designed to match albums on either title and artist(s) or Musicbrainz album ID (if one is provided). It does not require all the related music files to be in a unique folder, although that is an obvious way to organise music for humans. Hence if you have music files in separate folders both tagged with same album and artist(s) then Kodi will make one library entry for that album. Likewise with songs on the album, matching is done by filename, title and track number or by Musicbrainz track ID (if provided).
The only way to have multiple copies of songs/albums in the library is by tagging them as different releases and using Musicbrainz Id tags (they are different per release).
If that explanation does not match your experience then provide details of an example please.
RE: Kodi deleting or modifying previous entry when scanning in similar song - cmsimike - 2019-03-13
DaveBlake - thanks for the replyI You've previously helped me with what I thought was a Kodi music library bug (which it wasn't! I just didn't understand how POPM mp3 tags worked) so I am very hopeful this is yet another misunderstanding on my end.
I will try my best to give you the required information without going overboard. Apologies if I don't hit this target.
The song in question is the same *song* but on different albums. Here are the (what I believe to be) relevant columns from the MySQL database for the song.
https://imgur.com/a/DDQVqwn
Same title, different album, different file name, (different different strMusicBrainzTrackID
Here is an interesting note, date added is about the same time, but idSong 29045 was actually added on 2017-10-21 18:59:30 (confirmed in my "good" mysql dump, can post a screen shot if you like) but after the latest import where kodi dropped, the added date is 2019-01-22 18:24:55.
This is one song. I have an instance of another song who's previous entry was completely deleted and instead now I only have the new song entry. I presume this has to do with when Kodi's scan finds the song? But this is speculation on my end.
Please let me know if there's anything else you need from me to continue debugging.
RE: Kodi deleting or modifying previous entry when scanning in similar song - DaveBlake - 2019-03-13
OK, so something for me to dig into with you....
With the example given in the image you have 2 songs in the db with same title (song ids 29045 & 123951), and when the second was scanned into the library the userrating and/or iTimesPlayed of the first was changed? Or did they change at some other point? If you could let me have a copy of those 2 music files (upload to a cloud site and PM me a link) I'll see if I can reproduce.
The "dateadded" field is by default the file modified datetime stamp and not the data that it was actually added to the music library. The naming is misleading but probably made more sense sometime long ago.
Quote:but after the latest import where kodi droppedNot sure what you are saying, by "import" you mean scan or did you use import facility? What did you drop - source, db? Maybe it doesn't matter.
Quote:I have an instance of another song who's previous entry was completely deleted and instead now I only have the new song entry.That sounds like the matching I described. Yes, if a later scan finds a song that matches an existing one (although it could be in a different file) that will overwrite the original entry. Scanning is sequential and the end result is cumulative. But you seem to be saying they are on different albums, and if that is the case can you give the song record details of both if you have them, it sounds like you have backup db so maybe you have this.
To save my poor eyesight, could you paste any db rows as text rather than an image
RE: Kodi deleting or modifying previous entry when scanning in similar song - cmsimike - 2019-03-13
In the example given, 29045 existed since about 2017 and when 123951 was scanned into my library yesterday, Kodi modified the userrating and itimesplayed of 29045 back down to 0. I always take a snapshot of MySQL before and after I do a library scan due to the issues that seem to appear after a scan for me. In this case, I took a snapshot of my MySQL database right before I scanned for new music, then again right after and diffed the two (select certain fields from songview where (itimesplayed > 0 or userrating > 0), which showed Kodi modifying itimesplayed and userrating of a bunch of songs.
I will prep the files and PM you with them. Thank you for taking the time to look into them!
Sorry, I meant music library scan when I said import:
Quote:but after the latest import where kodi dropped [the userrating and itimesplayed],
Quote:That sounds like the matching I described. Yes, if a later scan finds a song that matches an existing one (although it could be in a different file) that will overwrite the original entry. Scanning is sequential and the end result is cumulative. But you seem to be saying they are on different albums, and if that is the case can you give the song record details of both if you have them, it sounds like you have backup db so maybe you have this.You are absolutely right about this case. Looking at the data, the previous song I had was just a one song off the album, and in the latest library scan, I added the whole album. The previous song I had a strMusicBrainzTrackID of f6ea8e8e-317e-4cc7-866e-d5778e139af3 and the new one has the same thing. It is even the same idSong in the database, just a new location for the file. I will opt not to include the DB rows since this mystery is solved.
RE: Kodi deleting or modifying previous entry when scanning in similar song - DaveBlake - 2019-03-14
Just to say I have the files thanks, let's see if I can reproduce the behaviour you describe.
RE: Kodi deleting or modifying previous entry when scanning in similar song - cmsimike - 2019-03-14
(2019-03-14, 18:58)DaveBlake Wrote: Just to say I have the files thanks, let's see if I can reproduce the behaviour you describe.Thank you! If there's anything else you need from me, please let me know. I am very eager to figure this out!
RE: Kodi deleting or modifying previous entry when scanning in similar song - DaveBlake - 2019-03-14
Sorry to say but I am not reproducing this behaviour.
(2019-03-13, 21:26)cmsimike Wrote: In the example given, 29045 existed since about 2017 and when 123951 was scanned into my library yesterday, Kodi modified the userrating and itimesplayed of 29045 back down to 0. I always take a snapshot of MySQL before and after I do a library scan due to the issues that seem to appear after a scan for me. In this case, I took a snapshot of my MySQL database right before I scanned for new music, then again right after and diffed the two (select certain fields from songview where (itimesplayed > 0 or userrating > 0), which showed Kodi modifying itimesplayed and userrating of a bunch of songs.No doubt then something changed the timesplayed and user rating, but I just can't see how the addition of a newly scanned song with the same title and song artist but different everything else including mbid tag could do that. The scan is not mistaking one song for the other, it adds a new song record. If the older music file had been rescanned (because comparison of hash values indicated it may have changed) during the update then it would have a new id. Keeping the old ID 29045 means that scanning process does not update that record.
Or can I have doubt that the older song changed.... If you have both verisons of the db can you paste the before and after records from the song table with songid = 29045. I believe you, but maybe there is something else to see in the full record.
Could something else have run, an addon say? Although I don't know any that update timesplayed, it can be done via JSON API. Oh hang on there is a script that restores playcount and rating from TADB see https://forum.kodi.tv/showthread.php?tid=341772&pid=2832329#pid2832329 , that only matches on song name. Have you been running something?
Quote:which showed Kodi modifying itimesplayed and userrating of a bunch of songs.Did all the songs in the bunch have songs that had just been added with same title? Just looking for a pattern.
More clues needed!!
RE: Kodi deleting or modifying previous entry when scanning in similar song - cmsimike - 2019-03-14
Well, it seems like I've played myself.
I had a whole response written out, then as I went back and tried to fill in info I noticed where I went wrong.
There are two issues from what I was seeing - the song whose previous entry was completely deleted and the song those previous entry was modified after the run.
Thanks to your help, you helped me understand the first issue. The record disappeared because I added a second song tagged exactly as the first. It updated the previous entry to be the new location with 0 for rating and itimesplayed.
Note, in this case, despite it being a new song in a new location being found and the old song still in its old location, Kodi reused the previous idSong (I am looking at both the "bad" and "good" entries right now).
The second issue, turns out, is exactly like the first. My confusion was that I accidentally added same album again in a different location. When I was looking at the "bad" database, the 29045 entry only different in strPath.
Presumably over the years, whenever I've noticed userratings and itimesplayed disappear it was due to me adding the exact same song somehow (which is sort of how I stumbled into figuring out the seemingly duplicate song issue). This I only figured out in Dec '18 but I didn't know this was Kodi working as intended until right now.
What's still tripping me up, and maybe you can help me resolve this, is the re-use of the same idSong for a song in a completely different location. Based on my read on some of your previous statements, I feel like new ids should have been generated in this case? But I'm sure I'm not fully understanding the Kodi music scanning process. But looking at the Kodi log, it definitely updates the previous record with the new song location.
Tonight I will remove the duplicate CD and ensure the scan results make sense (there should be *some* change as I continue to migrate my music collection from iTunes into Kodi but those are a few songs her and there and not the drastic change I saw last scan)
Thank you for all your help in ensuring the sanctity of my music library - my life. And thank you for your help in better understanding how Kodi scans music in. Hopefully these freakouts of mine will become less common
Not a bug related Kodi question:
- Does Kodi have a mode that will blindly add songs without relying on its tags to determine if it should update a previous record or not? For instance, is there a mode Kodi can be in that will always create song records for each song on disk, without regard for tags? Kodi can continue displaying songs in its UI as it does now (and I understand that this would lead to duplicate songs).
- What tags does Kodi take into account in determining if the song is the "same" ? For instance, the issue I'm running into seems due to me tagging all my music with MusicBrainz Picard. If I had two of the exact same mp3s in two different locations, I presume Kodi would never see this as the same?
- Would it make sense to carry over the userrating and itimesplayed in this case? In both my cases it seems like the previous entry was just updated (and the logs indicate that too I think)
RE: Kodi deleting or modifying previous entry when scanning in similar song - cmsimike - 2019-03-15
Ok - confirmed. I put in an album that i shouldn't have (which is why the songs i sent didn't make sense) but at the same time, there were two songs that I've added the full albums of. The three songs i randomly checked were the two songs that, after understanding how kodi scans music, makes sense and the one i sent you. I thought they were the same sort of bug.
Thanks again for the help! I've marked this as solved.
RE: Kodi deleting or modifying previous entry when scanning in similar song - DaveBlake - 2019-03-15
Glad that you found out what happened.
(2019-03-14, 22:13)cmsimike Wrote: Not a bug related Kodi question:No. The library is tag data based not file based - many files of same song, results in one record for that song. It is fundamental to the design.
- Does Kodi have a mode that will blindly add songs without relying on its tags to determine if it should update a previous record or not? For instance, is there a mode Kodi can be in that will always create song records for each song on disk, without regard for tags? Kodi can continue displaying songs in its UI as it does now (and I understand that this would lead to duplicate songs).
(2019-03-14, 22:13)cmsimike Wrote: - What tags does Kodi take into account in determining if the song is the "same" ?When scanning Kodi matches albums on either title and artist(s) or Musicbrainz album ID (if one is provided). It does not require all the related music files to be in a unique folder, although that is an obvious way to organise music for humans. Hence if you have music files in separate folders both tagged with same album and artist(s) then Kodi will make one library entry for that album. Likewise with songs on the album, matching is done by filename, title and track number or by Musicbrainz track ID (if provided).
So to answer exclicitly:
Either
ALBUM (TALB), ALBUMARTIST (TPE2), TITLE (TIT2), TRACKNUMBER and DISCNUMBER (TRCK & TPOS) and physical filename (but not location)
or
MUSICBRAINZ_ALBUMID (called MusicBrainz Release Id by Picard), MUSICBRAINZ_TRACKID (called MusicBrainz Recording Id by Picard) and TRACKNUMBER and DISCNUMBER (TRCK & TPOS).
Note I tend to use the Vorbis tag names that are also used in db etc. rather than the Picard names for tags. MUSICBRAINZ_TRACKID (MusicBrainz Recording Id) is not unique per release hence the use of trac number as well. We could be using MUSICBRAINZ_RELEASETRACKID (MusicBrainz Track Id) that is unique, but historically we didn't and a change would need all users to rescan their library at least and maybe retag.
(2019-03-14, 22:13)cmsimike Wrote: If I had two of the exact same mp3s in two different locations, I presume Kodi would never see this as the same?Again no. If tagged the same (with or without mbids), and named the same then Kodi sees that as one song.
(2019-03-14, 22:13)cmsimike Wrote: - Would it make sense to carry over the userrating and itimesplayed in this case? In both my cases it seems like the previous entry was just updated (and the logs indicate that too I think)Yes, when updating an existing entry although the old record is deleted and a new song id allocated, userrating and playcount are carried forwards from the old record. I think your previous cases are al little confusing, so I suggest you try a test case and convimce yourself it does what I say.
RE: Kodi deleting or modifying previous entry when scanning in similar song - DaveBlake - 2019-03-15
Just to add the per song rather than per music file way the music libray works makes sense if you have media scattered onto multiple removeable sources. You rate and play the song x times, not the music file it was stored in.
RE: Kodi deleting or modifying previous entry when scanning in similar song - cmsimike - 2019-04-05
Thanks again for everything @DaveBlake I'm glad what I was seeing wasn't a bug and was just how my music was tagged. I always appreciate the help.