Youre going under the assumption that you cant carry and/or conceal a larger gun, not sure based on what.
The smaller guns are almost always harder to get a hold of quickly and positively, harder to shoot well with, usually have harsher recoil, even in the smaller calibers, because they are blowback. The larger guns, even in the larger calibers, are often more comfortable to shoot with due to being locked breech and having grips that you can get a good hold on and handle the recoil better.
If you think the smaller guns are more appropriate, it would be interesting to hear what you are basing things on, and what you do to stay on top of things with it. Ive carried a number of them in the past, as backups, and shot them from how they were carried in regular practice, and other than a very narrow zone of usefulness, they were/are really not something that is a realistic choice for a primary gun that will handle a lot of situations.
What makes you ripe for the pickings is, a lack of training and skill. What you carry really isnt the thing here, its you, and what you can realistically do with what you choose, and in any situation.
We dont get to pick the fight, but we do have to deal with it as best we can if and when it comes. No one can tell you when that will happen, and hopefully, youre prepared for what you get, and you brought both an appropriate gun, and the skills to use it along with you.
I dont know how you look at this but Ive always looked at it as a lifestyle/package deal sort of thing. Its done on a daily basis, and its just another thing on my person that is there if I need it. Its no big deal, and its easily done.
Its also done with something that I shoot well with, and practice with on a regular and realistic basis, and I know will allow me to cover as much as possible that might pop up, not just one or two narrow scenarios.