If I were you I wouldn't want bitter neighbors who might lie about me to get me in trouble.
Been there, done that, got many T-Shirts.
Without getting into alot of detail, I quickly managed to make myself persona non grata when I moved in here. At one point they managed to have me cited for a whole slew of violations (they went through the town code book, and found anything that remotely applied, and a few things that didn't). Of course, once I got to court, all the charges were dropped (the judge was smart, and I had more witnesses, as well as a good lawyer). But it cost me some $$$$$ for a lawyer (needed him, to go though all the legalese, so we could rebut all the charges).
They did manage to get me fined $50, for a barking dog ordinance (this all stemmed from training dogs on my 10 acres of property). Even though we trained so far away from anyones house, you could barely hear them if you wer OUTSIDE. Inside my house (obviously the closest) you couldn't here them at all.
But anyway, if your neighbors refuse to compromise about shooting times etc. I would DEFINITELY try to build some kind of sound deadening structure to shoot from. Because, this shows that you made an effort to accomodate their complaints, and did what you could to alleviate the problem, even though you were not required in any way to do so. I would also constrain my shooting times to a few hours in the middle of the day. More CYA.
Now, also make ABSOLUTELY sure that your backstop is containing all the bullets, and any misses/ricochets could only go in a safe direction. Beacuse their next tactic might be to claim that they heard "bullets whizzing past" while they were out in their yard. Bad juju, and impossible to prove or disprove.
Surprisingly, I've not caught any flak from my neighbors, for shooting "out back" (but, again, I have 10 acres, so we get quite aways away)...I also only shoot there infrequently, and mostly with some of my less-loud guns.