Kodi Community Forum

Full Version: Duplicates on Every Scan
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Real quick background. I don't use a media manager for TV shows, just make sure the episodes are properly named and rely on the internal scraper. With one exception. I have home movies in my library as a custom TV show. I have them in a "Home Movies" subdirectory with a tvshow.nfo file. In addition, I have the video files each with its own nfo and tbn. The directory looks like this:

Code:
-rw-r--r-- 1 theophile theophile   37055769 2009-09-14 13:43 2009-01-11_17:31.mkv
-rw-r--r-- 1 theophile theophile       1552 2009-09-14 13:41 2009-01-11_17:31.nfo
-rw-r--r-- 1 theophile theophile      54550 2009-09-14 13:43 2009-01-11_17:31.tbn
-rw-r--r-- 1 theophile theophile  125008006 2009-09-14 13:45 2009-01-11_18:11.mkv
-rw-r--r-- 1 theophile theophile       1553 2009-09-14 13:47 2009-01-11_18:11.nfo
-rw-r--r-- 1 theophile theophile      57552 2009-09-14 13:38 2009-01-11_18:11.tbn
-rw-r--r-- 1 theophile theophile   16273772 2009-09-14 13:47 2009-01-12_11:31.mkv
-rw-r--r-- 1 theophile theophile       1560 2009-09-14 13:41 2009-01-12_11:31.nfo
-rw-r--r-- 1 theophile theophile      33314 2009-09-14 13:39 2009-01-12_11:31.tbn
-rw-r--r-- 1 theophile theophile   31685067 2009-09-14 13:45 2009-01-12_17:14.mkv
-rw-r--r-- 1 theophile theophile       1552 2009-09-14 13:39 2009-01-12_17:14.nfo
-rw-r--r-- 1 theophile theophile      42627 2009-09-14 13:39 2009-01-12_17:14.tbn
-rw-r--r-- 1 theophile theophile   25134691 2009-09-14 13:46 2009-01-15_07:53.mkv
-rw-r--r-- 1 theophile theophile       1549 2009-09-14 13:40 2009-01-15_07:53.nfo
-rw-r--r-- 1 theophile theophile      33724 2009-09-14 13:46 2009-01-15_07:53.tbn
-rw-r--r-- 1 theophile theophile  146570153 2009-09-14 13:39 2009-01-15_14:59.mkv
-rw-r--r-- 1 theophile theophile       1553 2009-09-14 13:39 2009-01-15_14:59.nfo
-rw-r--r-- 1 theophile theophile      59944 2009-09-14 13:39 2009-01-15_14:59.tbn
-rw-r--r-- 1 theophile theophile   36854700 2009-09-14 13:38 2009-01-22_23:42.mkv
-rw-r--r-- 1 theophile theophile       1554 2009-09-14 13:41 2009-01-22_23:42.nfo
-rw-r--r-- 1 theophile theophile      41079 2009-09-14 13:45 2009-01-22_23:42.tbn
-rw-r--r-- 1 theophile theophile   84859982 2009-09-14 13:41 2009-01-23_19:50.mkv
-rw-r--r-- 1 theophile theophile       1552 2009-09-14 13:40 2009-01-23_19:50.nfo
-rw-r--r-- 1 theophile theophile      45018 2009-09-14 13:46 2009-01-23_19:50.tbn
-rw-r--r-- 1 theophile theophile   43452957 2009-09-14 13:46 2009-01-26_16:46.mkv
-rw-r--r-- 1 theophile theophile       1552 2009-09-14 13:38 2009-01-26_16:46.nfo
-rw-r--r-- 1 theophile theophile      57605 2009-09-14 13:38 2009-01-26_16:46.tbn
-rw-r--r-- 1 theophile theophile   63223412 2009-09-14 13:39 2009-01-28_19:33.mkv
-rw-r--r-- 1 theophile theophile       1552 2009-09-14 13:47 2009-01-28_19:33.nfo
-rw-r--r-- 1 theophile theophile      49150 2009-09-14 13:40 2009-01-28_19:33.tbn
-rw-r--r-- 1 theophile theophile   44870328 2009-09-14 13:41 2009-01-29_17:28.mkv
-rw-r--r-- 1 theophile theophile       1552 2009-09-14 13:39 2009-01-29_17:28.nfo
-rw-r--r-- 1 theophile theophile      55158 2009-09-14 13:39 2009-01-29_17:28.tbn
-rw-r--r-- 1 theophile theophile     773496 2009-09-14 13:47 2009-01-31_17:09.mkv
-rw-r--r-- 1 theophile theophile        384 2009-09-14 13:43 2009-01-31_17:09.nfo
-rw-r--r-- 1 theophile theophile      42860 2009-09-14 13:46 2009-01-31_17:09.tbn
You get the idea. When I run the scanner, it adds all the videos perfectly.

Now here's the problem. Occasionally I add more videos. When I do this, the scanner adds the new videos, then re-adds all the existing ones so I end up with duplicates. This happens every time the scanner is run. So the first videos I added are in the library 6 or 7 times each.

I can resolve this by deleting the video source and re-adding it. But the next time I run the scanner, the duplicates start piling up again.

Is this a bug or do I have something wrong? How can I prevent the scanner from re-adding "episodes" that already exist?

Thanks!!
That sounds like a bug. You shouldn't get duplicates. The code checks to see if an item is in the database already before adding it. I know this works for "real" TV shows for me. Is this only happening with your home movies fake tv show?
kraqh3d Wrote:That sounds like a bug. You shouldn't get duplicates. The code checks to see if an item is in the database already before adding it. I know this works for "real" TV shows for me. Is this only happening with your home movies fake tv show?

Yes, every other show/episode behaves fine. Though none of my "real" shows have nfos. I've actually been noticing this behavior for a while, since at least last summer. I've only now decided to whine about it. ;-)

How can I help? Would logs be useful?
The most useful thing you can do is reduce it to the bare minimum required to reproduce. I suspect this would consist of a folder with a single "episode" (0 byte .mkv file is fine) and the episode and tvshow .nfo files.

In addition, you'd have a second "episode" (and/or secondary tvshow.nfo files) that you use to copy in to reproduce the issue.

With this, we can see whether it's intentional (eg something is wrong with the nfo files perhaps) or a real bug.

The key is always to reduce to the smallest amount of data and steps to reproduce the fault. The smaller they are, the more likely someone else can reproduce.

Cheers,
Jonathan
OK, I'll set it up and give it a go. In the mean time, here is the format of the .nfo I am using for the "episodes." The stuff after the the close tag is historical info for my own archival purposes.

Code:
<episodedetails>
        <title>July 27, 2009 - 6:24 PM</title>
        <rating></rating>
        <season></season>
        <episode></episode>
        <plot>This video was recorded on July 27, 2009 at 6:24 PM.</plot>
        <credits></credits>
        <director></director>
        <aired>2009-07-27</aired>
    <time>18:24</time>
    <runtime>00:03:01</runtime>
</episodedetails>

========== ORIGINAL FILE INFO ==========

General
Complete name                    : capture011.dv
Format                           : Digital Video
File size                        : 621 MiB
Duration                         : 3mn 1s
Overall bit rate                 : 28.8 Mbps
Recorded date                    : 2009-07-27 18:24:22

Video
Format                           : Digital Video
Duration                         : 3mn 0s
Bit rate                         : 26.1 Mbps
Width                            : 720 pixels
Height                           : 480 pixels
Display aspect ratio             : 16:9
Frame rate mode                  : Constant
Frame rate                       : 29.970 fps
Standard                         : NTSC
Colorimetry                      : 4:1:1
Scan type                        : Interlaced
Bits/(Pixel*Frame)               : 2.519
Stream size                      : 562 MiB (91%)
Encoding settings                : ae mode=full automatic / wb mode=automatic / white balance= / fcm=manual focus

Audio
ID                               : 0
Format                           : PCM
Duration                         : 3mn 1s
Bit rate                         : 1 536 Kbps
Channel(s)                       : 2 channels
Sampling rate                    : 48.0 KHz
Resolution                       : 16 bits
Stream size                      : 33.2 MiB (5%)
jmarshall Wrote:The most useful thing you can do is reduce it to the bare minimum required to reproduce. I suspect this would consist of a folder with a single "episode" (0 byte .mkv file is fine) and the episode and tvshow .nfo files.

In addition, you'd have a second "episode" (and/or secondary tvshow.nfo files) that you use to copy in to reproduce the issue.

With this, we can see whether it's intentional (eg something is wrong with the nfo files perhaps) or a real bug.

The key is always to reduce to the smallest amount of data and steps to reproduce the fault. The smaller they are, the more likely someone else can reproduce.

Cheers,
Jonathan
Sorry it has taken so long for me to actually do this. Lots of stuff going on, forgot, etc., etc.

Okay, to reproduce this behavior, here's what I do:
  1. Create a new subdirectory ("Test Show") within one of your TV Show source locations
  2. Create a tvshow.nfo containing the following:
    Code:
    <tvshow>
            <title>Test Show</title>
            <episodeguideurl><episodeguideurl>
            <rating>9.900</rating>
            <episode>0</episode>
            <plot>The best TV show in the world</plot>
            <genre>Family</genre>
            <premiered></premiered>
            <studio></studio>
        </tvshow>
  3. Create an episode .nfo called "2009-03-19_18:13.nfo" with the following content:
    Code:
    <episodedetails>
            <title>First Episode</title>
            <rating></rating>
            <season></season>
            <episode></episode>
            <plot>This is the first episode.</plot>
            <credits></credits>
            <director></director>
            <aired>2009-03-19</aired>
        <time>18:13</time>
        <runtime>00:01:02</runtime>
    </episodedetails>
  4. Create a 0-byte mkv called "2009-03-19_18:13.mkv"
  5. Copy any .tbn to this directory and rename it "2009-03-19_18:13.tbn"
  6. In XBMC, update library and make sure the new show appears with the single episode.
  7. Now in the new "Test Show" directory, along with the four files created above, create a new .nfo called "2010-04-12_09:37.nfo" with the following content:
    Code:
    <episodedetails>
            <title>Second Episode</title>
            <rating></rating>
            <season></season>
            <episode></episode>
            <plot>This is the second episode.</plot>
            <credits></credits>
            <director></director>
            <aired>2010-04-12</aired>
        <time>09:37</time>
        <runtime>00:01:02</runtime>
    </episodedetails>
  8. Create another 0-byte mkv called "2010-04-12_09:37.mkv"
  9. Copy the existing .tbn to "2010-04-12_09:37.tbn"
  10. The directory now contains the following files:
    Code:
    2009-03-19_18:13.mkv  
    2009-03-19_18:13.nfo  
    2009-03-19_18:13.tbn  
    2010-04-12_09:37.mkv  
    2010-04-12_09:37.nfo  
    2010-04-12_09:37.tbn  
    tvshow.nfo
  11. In XBMC, update library or scan for new content.

When the second episode is added, the first will be duplicated. If you repeat the procedure to make a third episode, upon scanning, each existing episode will be added again, so the first one will appear three times, the second one two times, and so on.

Since I regularly add "episodes," the only way I can get rid of all the duplicates is to go to the Video menu, navigate to the show folder, set content to none, choose "yes" when it asks me if I want to remove all the episodes from the library, change the content back to TV show, then rescan.

This homemade TV show is the only show in my system that uses .nfos. For all the others, I only have the media files and let the scraper take care of the rest.

I'm using Linux AMD64 if that makes a difference. Thanks for taking a look.
Thanks - I'll take a look.
Tested. Your scenario works fine on current trunk. No duplicate entries.

A Debug Log of you reproducing the problem would be useful, assuming you yourself are using current trunk (or Dharma)