Are you the one leaving this stuff around the range? It drives me crazy when it gets mixed in my precious range pickings.
Sorry, but I don't actually have data to answer your question. I would assume it would make a small difference, how small depending on what powder you use. With a fast-burning powder like you'd use for a target load, I would expect it to be a VERY small difference--as in so small I would try your regular load and see if you can tell any difference. I'm betting you can't.
If you're running hot loads of power pistol or some other medium-burn rate propellant, I'd guess you might have to work back up to your target velocity--but I wouldn't see why you'd have to back down. I mean, it's going to be a reduction in pressure, though a small one.
Hopefully someone will have an answer from a book for you.