Well, the trouble for me is that most of the "Nazi" stuff you see is reproduction. There is no historical element to it, other than by an approximation of design. I'm really turned off by that kind of stuff. However, true WWII relics don't bother me at all. The Hitler stamps and the Nazi money are intriguing in a historical view. However, I am fascinated by occupation and counter-occupation currency. I do have a problem with folks who are too enamored by Nazi replica stuff because it shows a fascination with the concept more than the relics. But, to each his own. As to the swastika being on other things, I have Czarist money from Russia printed in the late 1800's on up to WWI with it on them. As a symbol, it is an interesting, eye-catching one - much like bio-hazard and fall-out shelter symbols have a form that is attractive.
As to Confederate heritage, I am very proud of my heritage. I own a Colt model 1849 my Great Great grandfather Daniel carried, at the age of 14, to the Battle of Atlanta. My family is at present exclusively of Confederate descent. Nothing at all in the world wrong with that. They were all farmers/pastors and none of them own slaves, but did live in Georgia, Alabama, and Florida and, particularly the Georgians, saw an invading army destroying their property. They had every right to defend it.
But then, I really like this symbol, kind of an Anti-Nazi symbol.