What you’re doing right now sounds perfectly adequate for defensive applications, but not to the exclusion of also working on accuracy.
You should endeavor to shoot “accurately enough” when speed is the pressing matter (and I’m told that, at typical pistol-fighting distances, it often is), but you also need the ability to shoot with precision when precision is the pressing matter.
General rule of thumb these days seems to be eight inches of accuracy across all distances, high center chest. In other words, pretty much what you’re already doing. Adding some precision work to your practice would be a great idea.
I wouldn’t forget head shots, though, as earlier suggested. rcmodel’s cautions duly noted, you absolutely need the ability to shoot what needs to be shot when the shot is available, whether a head or an elbow or a toe.
Does this mean that "absolute precision" is the solution all of the time? Nope. "Accurate enough" will often do. Does this mean that absolute precision is the solution some of the time? Yep. Circumstances dictate which, and when.
Also, as ChaoSS mentions, when it comes time to shoot, you pick your spot and aim to hit it, whether that spot is a bullseye on a piece of paper, a button on a shirt, or an imaginary off switch.