Funny you should mention it. I read once that the market hunters, using punt-guns, would mix tallow in with the shot to improve patterns.
So, I experimented with mixing different lubes in with the shot, in those little shot capsules you load in .44 mag/spl cartridges, and .357/.38spl cases. The experiment worked, it did improve patterns. The trouble I found was that the mix had to be right for the temperatures. When it got real cold, the shot and lube would clump together, and if cold enough an almost a solid mass. Too soft, and not much improvement. There was kind of a temperature "zone" where they worked.
ThomasT I think you are on the right track, I believe there's kind of a narrow "window' of velocity where you get your best patterns. With pistol cartridges such as the .44's and .357/38's I seem to recall that the suggestion is to not load them over 1000fps. But I think if you played around with the charge enough one would find the sweet spot. If one's pistol bore would take a modern shotgun wad, that might help a lot. In my Brown Bess Carbine, a 12 gauge plastic wad, inside a paper tube, (and filled with shot of course) fits perfectly in the bore and produces very good patterns. Three dead turkeys will attest to that. I also mix shot sizes, mostly #4 birdshot, but pinches of 6's, and 7's. Possibly in a pistol, just the paper tube would help. And maybe a buffer?