Do some research before you go and buy a 380.
Massad Ayoob has written several articles about how he has done some slaughter house testing with various 380 loads and that the round barely penetrates the smallest pigs skull. In some cases he reports that the 380 wouldnt penetrate the skull at all and this is why he had to stop testing the round on medium size animals.............just something to think about when it comes to defending yourself or your families lives....
Ayoob's 'testing' seems more dubious than M&S and a lot more anecdotal. While pigs and pig skulls are not quite the myth many hunters make them out to be, the fact remains that there are definitely portions of the pig's skull that are more robust than that found in humans. It has to be in order to support the stressors inflicted on it such as the massive jaw musculature and actual use of the head as a rooting tool.
If Ayoob's tests were side shots with an attempt to penetrate the brain, then his rounds would likely have to pass through some of the heavy jaw muscles before getting to the brain.
Then there is the aspect of unsupported expanses. The human cranium is largely like a hollow ball. There are few folds, angles, ridges on the exteior or interior to really provide any sort of structural support. A pig's cranium has a much smaller area available that is unsupported. I base this on the fact that I have a pig skull with me right now from a male pig that was just reaching adulthood based on eruption of the 3rd molars.
Compare this with a human skull, particularly the cranium (brain compartment). Generally the thickness of a healthy adult cranium will be about 1/4 to just under 1/2 of an inch. The majority of this bone is not solid. Instead, it consists of an interior and exterior thin cortical surface between which is located spongy bone. The spongy bone isn't actually spongy. It just looks like a sponge. You get the same spongy bone at the ends of the long bones where the bone walls are thinnest. Spongy bone provides for excellent lightweight support, but isn't horrible difficult to penetrate with point impacts.
So, the human skull protects the brain, but it isn't terribly thick or tough. Moreover, most isn't covered with heavy musculature or even very thick skin. Folks with hair get a lot more protection to their skulls from the hair than they get from their skull.
Where all this is going is that there is really no legitimate comparison between pig and human skulls when it comes to protection against ballistic impacts in the living animals. To suggest otherwise would be naive.
-----
Getting back to the original query, neither .32 or .380 are at the high end of defensive performance. If you can't handle anything bigger then you go with what you can handle. Otherwise, you will be much better off with larger calibers.