I was afraid of that. When the nose of the bullet hits the feed ramp and the cartridge starts to tilt upward the magazine lips should release the back so that the rim can come up and under the extractor. When they don’t release the round in time, the ramp kicks the cartridge up into a nearly vertical position and the slide then jams it against the barrel face.
Fieldstrip your pistol. Then load one of the “rat-shot†loads into a magazine. Insert the magazine into the frame, and push the cartridge forward until the nose touches the ramp. Mark a line on the magazine lips where the base of the cartridge is. If at this point the magazine lips release the cartridge you should be O.K. If they don’t, (and they likely won’t) the lips need to be shortened to this point. As an alternative you can try all of your magazines to see if one of them happens to work, but I suspect you’ve tried that already.
All automatic pistols are designed around a particular cartridge and configuration. Other configurations may work, but if they do that’s a bonus. Often getting a non-standard cartridge to feed requires some fiddle’n around and modifications here and there. This is one of those times.