Hmmm, probably a lot of categories in this:
a) Guns that were unpopular and didn't sell
b) Guns that didn't work, or work reliably
c) Guns where the basic concept didn't fly
d) Guns that just never got off the ground for legal reasons
e) Guns that just never got off the ground for business or technical reasons
I can think of a few.
Category a: probably many, but all in shades of grey.
Category b: Similar to a and highly subject to opinion. Neck, the Nambu pistols were pretty dangerous and should have been a discarded design, but they made plenty. I'd say the old "COP .357" pepperbox derringer fits here.
Category c: Probably the Gyro-jet. Least effective at short range, which is not good for a handgun. Among some other pretty glaring problems with the concept.
Category d: The Raging Judge XXVIII. Like, whoops!
Category e: Bren 10 has to be the ultimate.
Subcategory to "e" -- call it "Weird is not a selling point." Dardick Trounds. Maybe the Mateba auto-revolver. Strange, but not in a way useful enough to justify the cost and weirdness.