A very wise old gentlemen who attends most of the same local matches I do once said a very important line: "If they keep score, then it's a game.".
Yes, it does have some different ideas behind it, but IDPA (and USPSA, and Steel Challenge, etc) is a
game. In any game you will have people who play just to have fun and people who play to win. There will be those who don't care about the score and those that do. Each individual competitor is absolutely free to choose which of those two types they'll be, but regardless of your choice, you have to respect the other faction and let them play the way they want. That comes with competing in any sport.
It honestly sounds like you just aren't comfortable with the idea of competition shooting (regardless of your past "exploits"). You'd probably do better doing some self defense training classes and/or exercises.
I will say though that Sam1911 does have a very valid point, and competition shooting does a lot to show what will work and what won't in real scenarios. Way back when "defensive" shooting was generally taught as a 1-handed "speed crouch" style. To a large degree it was competition shooting that showed that the more modern two-handed form is faster and more accurate. I'm sure initially though there was some cane waving and accusations about "gaming" though
.