Batjke,
While there would be no end to answers regarding not to do it, I think a Mosin, loaded properly would be actually a great home defense gun. Check out Brassfetcher online for their 203gr Soft Point gel tests for 7.62x54. Neat stuff and penetration seems manageable (however, it is Barnaul ammo, so flash would set the house on fire)
.
Also, Graf and Sons used to have a 125gr Hornady VMax for .311 diameter guns (specifically the x39). This, loaded with a mild load to keep flash down would be great. Failing that, a 125gr soft point would keep overpenetration down, as I would imagine most of the .311 diameter 125gr softpoints are designed for x39, not x54. Anything heavier than that would most likely overpenetrate. As for flash, lower loads are a must. Also, in my loading adventures, Vihtavouri powders always seem VERY low flash and perform superbly.
As for 8mm Mauser, in my opinion, would not use for home defense. Most bullets for loading in that diameter are heavy soft points that are designed for heavier animals. Unless you can find a light weight soft point for it, Mauser would be out of the question in my mind.
I have always been under the idea that a bolt action (if properly loaded for and trained with) can be effective for home defense. It isnt the latest shotgun with lights attached or tacti-crap on it, but one can still attach a light to a Mosin and have an effective home defense. The M44 Mosin, coupled with a scout mount/large diameter red dot, with a light attached (while looking ungainly), good loading and proper technique would be a very effective home defense weapon. Just because it takes a little more practice than a claymore on a stick doesnt mean discount it if that is all one has/limited to.
As far as not using handloads for defense, has there ever been a case regarding this? I have nothing but respect for Ayoob, but filler material for his book would be what I believe this to be. So many people say that but never come up with case one where it was used as an attack against the defendant. Heck, I could imagine that lots of folks that cant afford factory ammo using ball reloads bought from a store (like ultramax or house brands)for defense. I load all my loads for defense very stringently for quality and never scrimp (I dont use a progressive just for that reason, gotta check every one). I have NEVER had one load not go off. YMMV. I feel as long as one doesnt gas on about reloaded ammo and how it made him dead right there, one should be fine. Sorry for the rant, but I get a little hot under the collar when I hear that stuff. To me it is a similar arguement above: Its not the most tacticool round on the planet, so you will get in trouble in court for using it.