The obvious answer is, whatever you're comfortable with. Look, most of the time I'm happy to have a blade or even a .25ACP pistol. But if you put me in the wrong part of town at night (and I've found myself in those situations), a .25 would seem ridiculous. A .380ACP would be ridiculous.
There was a power outage in New York City years ago. Rioting broke out and mobs of people were breaking into stores and the police were overwhelmed. Some people, finding themselves in bad parts of town, were assaulted and their cars vandalized. I recall reading one story of a fellow with (I think) a Ruger Mini-14 in his trunk, and he was able to comfortably extract himself from his difficulties.
The lowlife anti-gun people think we're safe under the protection of the government, but they didn't even address the New Orleans mess. But when you say, what's the minimum caliber for self defense, you have to be prepared to say where and when, and under what circumstances.
For dining out, walking in the neighborhood and local driving, a .25ACP pistol is a splendid device. At night, downtown, it still will do the job. Go to the outskirts of town in a place like Baltimore and I'd want to up the ante a bit. I'd feel comfortable with a .38 revolver snubnose or a 9mm (the Star BM was an exceptional choice). If your girlfriend gets ticked off at you and throws you out of her car in a run-down part of town with broken streetlights and graffiti and out-of-order pay phones, I'd want to up the anti again with a hi-cap 9mm or .40 badboy ammo with gaping hollowpoints. And perhaps an extra clip.
You can keep going until you're hiking in the great Northwest, in which case you might want to consider a handgun that will kill a bear and put you in the hospital—you know, those calibers that loosen the fillings in your teeth?
The bottom line is there is that the answer is whatever it takes to make you feel secure in the time and place you happen to be.