I don't do IDPA (althought I'd like to give it a try), but I do CCW. I've got a number of different cover garments. In the summer I usually go with a loose fitting Hawaiian/Camp shirt over a tee shirt. Once the weather starts to cool, I go with sweat shirts, hoodies, sweaters, and insulated vests.
I've got a "safari-style" vest that I have been known to wear for CCW on occasion. I usually use it in outdoor settings (hiking, bird watching, nature photography) where its less out of place. On the other hand I've worn it into gas stations, casual resturants, etc. while traveling to or from my outdoor activities and I've not noticed anyone paying particular attention to me or my outfit.
Personally, I think the idea that things like vests or fanny packs "scream gun" is way overblown. Sure, many of the people here might suspect someone with a vest of fany pack is carrying, but we are in a vanishingly small minority. At most, the vast majority of people might think you dress odd, but most of them won't even notice or think about it if they do.