Okay, a picture just flashed through my head. How about utilizing the force of the recoil to compress and set a spring much like an open-bolt machinegun presets the bolt with each shot. However, the handgun would fire from the closed position and the entire upper half of the gun would recoil. Once the shot is away, the user must then operate another switch which opens the action energetically and ejects the spent case.
This has yet to be done, Kaylee, unless I am unaware of it. All other recoil systems I know of will return the gun to battery. Well, I can't really say that. I've noted a completely different form of recoil on a 1911 when I fire the last shot. Essentially, this new gun would amplify the effect of semi-automatic pistols with the entire action moving instead of just the slide. This action would store a good ammount of recoil energy in the spring to be released in another impulse when the action opens.
I'm aware of, but haven't examined, a 'recoilless' system where the locked breech and barrel are sent forward prior to the shot. I think it's used on a Trap gun, but the gun eludes me now. This action is similar, but not exactly like, what happens when a large-mass-ratio open bolt gun is fired. Using the M-3 Grease gun as an example, the large bolt accelerates forward and fires the cartridge. The movement of the heavy bolt attenuates the spike in recoil that you get when the gun fires. When the gun is fired for one shot (which is quite do-able with the Grease Gun), the setting of the recoil spring spreads the bolt-mass recoil out over a greater period of time where the peak forces aren't that great.
Problems: Of course to be reliable and versatile, the spring cannot be that strong or else the 'set' will not occur. This would happen with a light load that would fail to send the action completely to the rear. Therefore, some level of 'spiked recoil' would still be necessary for the reliable function of such an arm... or there COULD be a second buffer so that the action would compress a stronger spring after the point at which the action locks. This would further buffer the action against heavy loads and yet still allow a reliable lockup.
So, this gun would fire just like, say, a Magnum Research Lone Eagle pistol. The barrel, bolt, and reciever assembly would travel aproximately 3" compressing a recoil spring and lock to the rear. The user then actuates a release lever, button, or some sort of device which sends the receiver and barrel forward while at the same time ejecting the spent casing from the stationary bolt assembly. Perhaps these are manually actuated like the forward stroke of a pump-action shotgun? One final step would be to release the bolt assembly to close therefore readying the gun for another shot!
I've got a crude idea of the mechanics in my head, but I'd need to draw it and sleep on the idea. I'll conjure up JMB later this winter for some help.