Set screws are pretty pathetic, in my experience, expecially when they're subject to shock loading. They creep a little with each impact, and gouge whatever they're driven into.
Here's a suggestion: Make the part of the brake that slides over the barrel out of pipe, with a fairly close slip fit over the barrel. You can thread the pipe, and split it, and if you screw a pipe union onto it, it will clamp down amazingly.