Furthermore, he has a wife to protect. Her safety should be his primary concern, not his pride, not his rights, not solving the problems of the world.
While my main argument was based on principle, my ancillary argument was based on practicality. That moving whenever you had an obnoxious neighbor wasn't realistic. What if he encountered a similar neighbor at the next place? Or the next? Managing the safety of oneself or one's family consists of balancing risks and costs. You may suffer the expense of a move and end up no better off safety wise. Obviously, in the OP's risk/reward tables, moving is not an option and I have no option but to take his word for it.
And while I in no way expect and/or suggest he is/be responsible for anyone else's safety, what about the little old lady next door? She's likely more susceptible to intimidation than the OP. If he left that means she'll bear the brunt of this jerk's behaviour and I bet she has less resources to find a new place than the OP does. That's not really the OP's problem, but I would argue that the fact that we're so reluctant to stand together with our decent neighbors, emboldens the bad ones to sully our lives and neighborhoods.