[madmax2]

Hi,

Is it possible to find all duplicate movies using Smart Playlist?

Or is there a good way to find all duplicate movies that we have?

[PatK]

Wouldn't think a smart playlist would be needed, just clicking on movies should place duplicates side by side if listed according to title. Be sure you actually have a duplicate, and not just a double listing for the same file. Look in the movie information listing itself and examine carefully the full path. There'll be at least 2 paths referenced for the same file in different locations if you have a real duplicate. Employ the library clean-up utility so extraneous listings without video files are cleaned. In most cases on a windows system if you have extras, trailers, meta-data in the movie folders, just deleting the movie inside Kodi will not eliminate that folder structure and extras, so You'll probably want to handle this outside Kodi with windows file manager.

I think I get the spirit of what you want to do, not having to pour over a huge library looking for duplicates. For this I have used a free program called CCleaner, Install and run it, go to the tools on the sidebar, sub folder called duplicate finder, and choose some realistic settings, ignore small sizes, system files, zero bytes files, or any other excludes, then match by title and pick the drive/folder. The finder will allow you to delete directly, or save the list to a text file.

[madmax2]

Wouldn't think a smart playlist would be needed, just clicking on movies should place duplicates side by side if listed according to title. Be sure you actually have a duplicate, and not just a double listing for the same file. Look in the movie information listing itself and examine carefully the full path. There'll be at least 2 paths referenced for the same file in different locations if you have a real duplicate. Employ the library clean-up utility so extraneous listings without video files are cleaned. In most cases on a windows system if you have extras, trailers, meta-data in the movie folders, just deleting the movie inside Kodi will not eliminate that folder structure and extras, so You'll probably want to handle this outside Kodi with windows file manager.

I think I get the spirit of what you want to do, not having to pour over a huge library looking for duplicates. For this I have used a free program called CCleaner, Install and run it, go to the tools on the sidebar, sub folder called duplicate finder, and choose some realistic settings, ignore small sizes, system files, zero bytes files, or any other excludes, then match by title and pick the drive/folder. The finder will allow you to delete directly, or save the list to a text file.

Yes, you guess correctly about the spirit of my question.

Thanks for the ccleaner suggestion, though how would I just find the movie folder title name duplicates only?
I think if we use ccleaner, it will find duplicate files, which would show all these jpgs and other files which would have heaps of duplicates of the same name..

I just want it to show duplicate movie folder names..
So I can quickly identify duplicates of
-movies that are low quality and delete the lower quality ones
-actual same name and same size duplicates

====
yeah scrolling through a huge movie database just to find the duplicate movie title is not very efficient method.

Just for learning purpose, how would I use to smart playlist to do this?
Is it even possible to use smart playlist to list duplicates?

===

Would trailers, extras be scraped into the movie listing as duplicates even if it doesn't have an NFO file?
How would I ignore these files or get kodi to scrape them as Trailers and Extras?

[PatK]

That's why the sub menu item is called "called duplicate finder", it only looks for duplicate... choice of Title or Dat Size or both, I'd recommend you just list out the duplicates.

Playlist fields section 3.3.1 Fields > Smart playlists (wiki) but looking them over, I don't see a rule that applies (doesn't mean there isn't), but just viewing the library in movie mode with a title sort, will put same named titles to-gether, I like the KISS view point, don't like complicated scripts.

Would trailers, extras be scraped into the movie listing as duplicates even if it doesn't have an NFO file?
How would I ignore these files or get kodi to scrape them as Trailers and Extras?

Good question Yes the trailer would get treated as a video (I've made this mistake), but if you have an exclusion in the advancedsettings, then the trailer will be picked up only when you hit the trailer button if named properly.

You will have to create an advanced settings exclusion listing. advancedsettings.xml (wiki) The advancedsettings.xml file, which does not exist by default, can be created by a user in their Userdata (wiki) folder. I provided an advancedsettings (the code here) for you that when saved in some editor, and saved out as "advancedsettings.xml" into your userdatafolder will create the exclusion you want. This code should exclude trailers if named "movie_name-trailer.mkv" and the extras folder if the folder is called 'extras' To view these extras will need a add-on for that and a skin that supports this add-on. Add-on:Extras (wiki)

section 2.3.14 trailermatching and 2.3.11 excludefromscan and 2.3.12 excludefromlisting

Code:

<advancedsettings>
<video>
<excludefromscan>
        <regexp>[-\._ ](extrafanart|trailer|extrathumbs)[-\._ ]</regexp>
</excludefromscan>
<excludefromlisting><!-- Regular expressions that if evaluated to true won't be displayed in Files View -->
       <regexp>[-._ \\/](extrafanart|trailer|extrathumbs)[-._ \\/]</regexp>
</excludefromlisting>
    <!-- VideoExtras -->
        <excludefromscan action="append">
            <regexp>/extras/</regexp>
            <regexp>[\\/]extras[\\/]</regexp>
        </excludefromscan>
        <excludetvshowsfromscan action="append">
            <regexp>/extras/</regexp>
            <regexp>[\\/]extras[\\/]</regexp>
        </excludetvshowsfromscan>
        <!-- VideoExtras -->
</video>
</advancedsettings>

[madmax2]

That's why the sub menu item is called "called duplicate finder", it only looks for duplicate... choice of Title or Dat Size or both, I'd recommend you just list out the duplicates.

Playlist fields section 3.3.1 Fields > Smart playlists (wiki) but looking them over, I don't see a rule that applies (doesn't mean there isn't), but just viewing the library in movie mode with a title sort, will put same named titles to-gether, I like the KISS view point, don't like complicated scripts.

Would trailers, extras be scraped into the movie listing as duplicates even if it doesn't have an NFO file?
How would I ignore these files or get kodi to scrape them as Trailers and Extras?

Good question Yes the trailer would get treated as a video (I've made this mistake), but if you have an exclusion in the advancedsettings, then the trailer will be picked up only when you hit the trailer button if named properly.

You will have to create an advanced settings exclusion listing. advancedsettings.xml (wiki) The advancedsettings.xml file, which does not exist by default, can be created by a user in their Userdata (wiki) folder. I provided an advancedsettings (the code here) for you that when saved in some editor, and saved out as "advancedsettings.xml" into your userdatafolder will create the exclusion you want. This code should exclude trailers if named "movie_name-trailer.mkv" and the extras folder if the folder is called 'extras' To view these extras will need a add-on for that and a skin that supports this add-on. Add-on:Extras (wiki)

section 2.3.14 trailermatching and 2.3.11 excludefromscan and 2.3.12 excludefromlisting

Code:

<advancedsettings>
<video>
<excludefromscan>
        <regexp>[-\._ ](extrafanart|trailer|extrathumbs)[-\._ ]</regexp>
</excludefromscan>
<excludefromlisting><!-- Regular expressions that if evaluated to true won't be displayed in Files View -->
       <regexp>[-._ \\/](extrafanart|trailer|extrathumbs)[-._ \\/]</regexp>
</excludefromlisting>
    <!-- VideoExtras -->
        <excludefromscan action="append">
            <regexp>/extras/</regexp>
            <regexp>[\\/]extras[\\/]</regexp>
        </excludefromscan>
        <excludetvshowsfromscan action="append">
            <regexp>/extras/</regexp>
            <regexp>[\\/]extras[\\/]</regexp>
        </excludetvshowsfromscan>
        <!-- VideoExtras -->
</video>
</advancedsettings>

Thanks for providing the xml file.

Can you show me how to get ccleaner to just list the duplicate movie title folder names only?

I try use this duplicate finder in ccleaner and it is listing all these insignificant files like jpgs etc.. that just makes the lists of duplicates really long and harder to identify?

===

Yeah I understand KISS method would be easier to just list in movie listing and sorted by title...
but in this case going through a huge database of movies would become a big chore and is very inefficient method

In some cases, a script would be better solution even though initially it might be more complicated to get that script than KISS method,
but the end result it works out to be more better and efficient method and can be reused each time we need to find duplicates.

Even though KISS method is more faster and easier method initially, the actual work is more time consuming to go through the list
and if we need to repeat this method often, it would be very time consuming to keep doing this every time..

Anyway hopefully someone who knows how to do this with smart playlist can offer their solution.

[zerocool_ie]

fwiw, I use these MySQL queries on Krypton to find:

Duplicate movies by IMDB number:
https://pastebin.com/AZNfiJ4b

Duplicate episodes by Title, Season, Episode:
https://pastebin.com/aWwrDX2U

[scott967]

Yes, I think a distinction needs to be made between duplicate files and duplicate titles and IMDB represents a unique title.

scott s.
.

[madmax2]

fwiw, I use these MySQL queries on Krypton to find:

Duplicate movies by IMDB number:
https://pastebin.com/AZNfiJ4b

Duplicate episodes by Title, Season, Episode:
https://pastebin.com/aWwrDX2U

I don't use MySQL for kodi, would this command still work with the default database?

[madmax2]

Yes, I think a distinction needs to be made between duplicate files and duplicate titles and IMDB represents a unique title.

scott s.
.

So what are you trying to say or propose?
Do you have an example of what you mean?

[zerocool_ie]

I don't use MySQL for kodi, would this command still work with the default database?

(tested using SQLiteSpy)

Movies - replace lines 2 & 3 with this:

Code:

c00, dateadded AS DateAdded, strpath, strfilename

(SQLite doesn't suppport the CONCAT function)

TV - query works on SQLite without modification.

[madmax2]

I don't use MySQL for kodi, would this command still work with the default database?

(tested using SQLiteSpy)

Movies - replace lines 2 & 3 with this:

Code:

c00, dateadded AS DateAdded, strpath, strfilename

(SQLite doesn't suppport the CONCAT function)

TV - query works on SQLite without modification.

Just posting this here for people who don't want to view it on pastebin

So like this?
SQLite

Movies

Code:

SELECT
c00, dateadded AS DateAdded, strpath, strfilename
FROM
movie_view a
JOIN (
SELECT
uniqueid_value
FROM
movie_view
GROUP BY
uniqueid_value
HAVING COUNT(*) > 1) b ON a.uniqueid_value = b.uniqueid_value
ORDER BY
strfilename
ASC;

TV

Code:

SELECT y.idShow,y.c12,y.c13,
y.strFileName
FROM episode_view Y
INNER JOIN (
SELECT
c12,c13,idShow, COUNT(*) AS CountOf
FROM episode_view
GROUP BY idShow,c12,c13
HAVING COUNT(*)>1
) dt ON y.c12=dt.c12 AND y.c13=dt.c13 AND y.idShow=dt.idShow;

==========

Do you install MySQL on your main PC with Kodi installed on it?
Or where do you install MySQL?

=========

Thanks your SQLite command works.

It was able to find duplicate movies (different filename and filesize but same movie title + same year) that CCleaner did not find.

Basically it found the same movies but of different resolution/quality that CCleaner did not find,
since ccleaner only matches the exactly same file name + file size..

=====

It would be cool if someone made an addon that can find duplicates within Kodi.

[zerocool_ie]

Movies

Code:

SELECT
c00, dateadded AS DateAdded, strpath, strfilename
FROM
movie_view a
JOIN (
SELECT
uniqueid_value
FROM
movie_view
GROUP BY
uniqueid_value
HAVING COUNT(*) > 1) b ON a.uniqueid_value = b.uniqueid_value
ORDER BY
strfilename
ASC;

TV

Code:

SELECT y.idShow,y.c12,y.c13,
y.strFileName
FROM episode_view Y
INNER JOIN (
SELECT
c12,c13,idShow, COUNT(*) AS CountOf
FROM episode_view
GROUP BY idShow,c12,c13
HAVING COUNT(*)>1
) dt ON y.c12=dt.c12 AND y.c13=dt.c13 AND y.idShow=dt.idShow;

==========

Do you install MySQL on your main PC with Kodi installed on it?
Or where do you install MySQL?

If you want to query for dupes in your current setup (non-MySQL), then just download an SQLite browser (such as SQLiteSpy) and use it to open
userdata (wiki)\Database\MyVideosXX.db

Are you considering moving to MySQL? (I would recommend it only if you have more than one system with Kodi on it)
My setup is:
2 x RPi3's with Kodi, 1 x Samsung tablet with Kodi, 1 x Synology NAS where my media is stored, 1 x old Windows laptop with MySQL installed (acts as my 'poor man's server')

[madmax2]

Movies

Code:

SELECT
c00, dateadded AS DateAdded, strpath, strfilename
FROM
movie_view a
JOIN (
SELECT
uniqueid_value
FROM
movie_view
GROUP BY
uniqueid_value
HAVING COUNT(*) > 1) b ON a.uniqueid_value = b.uniqueid_value
ORDER BY
strfilename
ASC;

TV

Code:

SELECT y.idShow,y.c12,y.c13,
y.strFileName
FROM episode_view Y
INNER JOIN (
SELECT
c12,c13,idShow, COUNT(*) AS CountOf
FROM episode_view
GROUP BY idShow,c12,c13
HAVING COUNT(*)>1
) dt ON y.c12=dt.c12 AND y.c13=dt.c13 AND y.idShow=dt.idShow;

==========

Do you install MySQL on your main PC with Kodi installed on it?
Or where do you install MySQL?

If you want to query for dupes in your current setup (non-MySQL), then just download an SQLite browser (such as SQLiteSpy) and use it to open
userdata (wiki)\Database\MyVideosXX.db

Are you considering moving to MySQL? (I would recommend it only if you have more than one system with Kodi on it)
My setup is:
2 x RPi3's with Kodi, 1 x Samsung tablet with Kodi, 1 x Synology NAS where my media is stored, 1 x old Windows laptop with MySQL installed (acts as my 'poor man's server')

Yeah I already tested your commands before you posted this follow up comment...(using the same program you suggested, I used the portableapps version)

Thanks your SQLite command works.

It was able to find duplicate movies (different filename and filesize but same movie title + same year) that CCleaner did not find.

Basically it found the same movies but of different resolution/quality that CCleaner did not find,
since ccleaner only matches the exactly same file name + file size..

====
I do have more than one kodi installation..
The main issue I have is the movie progress is not sync, which I think having a MySQL database should solve.

I have a main pc and a second HTPC in the living room..

I install MySQL once on my main pc to try using it for kodi, but I noticed it really slowed down my pc.
My PC is not a bad or extremely old pc but it still affected it.

Does your laptop feel slow after putting mysql on it?

Do you use that laptop as a main work/daily pc or is it just for MySQL database only?
And do you leave this laptop on 24/7?

[zerocool_ie]

Does your laptop feel slow after putting mysql on it?

A little bit.. but the laptop is from 2008... Intel Core2 Duo, 4GB RAM.
My main laptop is an i7 4th Gen, 16 GB RAM, SSD drive. I have the same MySQL databases running on it and I wouldn't even know it's running.

Do you use that laptop as a main work/daily pc or is it just for MySQL database only?

MySQL, Ember Media Manager, I connect to it from the office sometimes when I want to test something

And do you leave this laptop on 24/7?

Yep

[madmax2]

Does your laptop feel slow after putting mysql on it?

A little bit.. but the laptop is from 2008... Intel Core2 Duo, 4GB RAM.
My main laptop is an i7 4th Gen, 16 GB RAM, SSD drive. I have the same MySQL databases running on it and I wouldn't even know it's running.

Do you use that laptop as a main work/daily pc or is it just for MySQL database only?

MySQL, Ember Media Manager, I connect to it from the office sometimes when I want to test something

And do you leave this laptop on 24/7?

Yep

My main pc is pretty decent, but I noticed it affected the startup speed and program loading once MySQL was installed..
I think it is because I also have many other startup programs + run browser with many tabs, and when I installed mySQL it really affected the performance more than normal.
So I immediately removed it.

I think if I install MySQL on another PC that does not do much (ie not main PC) just for MySQL then this might not be an issue..

Did you install MySQL so that you can have movie/tv shows progress sync with any kodi devices?
Or what was the reasons for doing this?

I have tried TrackTV to sync the progress, but it is not working 100% good,
basically the feature is flaky..and telling me I haven't finished a movie when I did etc..

Do you have a link for the laptop + NAS you bought?
I am also interested in getting a laptop..

Have many hdd + size do you have in your NAS?
Do you use an RAID on the NAS or have any backup method for your media?
Planning to do something similiar so I can put all my media on one device.