PhotoStick, but for massive amounts

[kartvelo]

I've seen ads for PhotoStick - just plug it into your laptop and it'll search out all your pictures, eliminate duplicates, and store them on the stick. The biggest one stores "60,000 photos!" I have more like 800,000 images collected on an external drive. There are dupes, to be sure, but I'm not confident that the number will reduce to anywhere near 60,000.
I'd like to accomplish what PhotoStick claims to, but I don't think PhotoStick has the horsepower I need. Is there a reliable way to run similar software to locate and extract large numbers of images from one external drive onto another, eliminating dupes?

 

[cgori]

I had been looking at this Kickstarter a while back, it has been delayed (as is typical) and I have no idea how well it will actually work but I think it's sort of claiming to be what you want: View: https://www.kickstarter.com/projects/monument/monument-2-smart-photo-storage-and-organization

 

[kartvelo]

I tried just doing a search on kind: picture on the original drive, hoping to then select all and just copy from one drive to the other - at least everything would be there, even if not as originally arranged, then I could eliminate dupes from that point somehow. The search works great (finds 760K items), the "select all" appears to work great, but it hangs at that point. I tried it with targeted searches that found only maybe 150K items, but the same thing happens.

 

[edoug]

I don't know how well it will work but I have an Idrive cloud account. When uploading files they do not upload duplicates. It hasn't for me anyway. For most it should be good enough. But 800,000 files is just a really imposing number of files. It is an option for maybe one of your problems.
https://www.pcmag.com/reviews/idrive
https://www.idrive.com/idrive/signup/el/ziffdavis