Got here late and haven't read the all of the responses. I've never had reason to go smaller than my Kel-Tec P32, and it's carried alone while traveling to and from my no-carry job site. Those are the only times it's carried without another, bigger gun. So, I guess that's as small as I go. Incidentally, when it is so carried alone, it's holding eleven rounds of .32 ammunition. I prefer its easier-handling, faster followup, and greater capacity to what's offered by my similarly-sized Taurus TCP.
I do own three guns that are inherently "smaller", and carried one of them decades ago once in a while, before the current crop of polymer-framed, recoil-operated "mouseguns" came on the market. That one is a Jennings J22. Neither it nor my Bauer .25 are carried these days. The third, a NAA mini revolver in .22LR, only does occasional BUG duty, and usually only in the woods or at the river.