It is said that Terry knew that model of pistol and practiced with it and kept his finger off the trigger. Broderick did not.
BTW, on the anniversary of the duel, I went to the spot to watch a reenactment. It was a fun sidetrip and there were plenty of speculators.
I learned of the haunted house thing years later during the 2010 West Coast Civil War Conference in San Francisco. Besides touring Fort Point and Alcatraz (Fort Alcatraz's citadel still stands beneath the prison structure), we went to Fort Mason (the National Park Service put a Dahlgren Gun there) where we were told by the Park Rangers of the ghost.