You want magnum rifle primers for slow burning rifle powder.
Slow is relative with powders, and primers of different make have differing heat outputs, so for rifle a 'magnum' primer does the job, irrespective of caliber.
The primer affects the pressure curve in the case just after ignition. With slower powders it raises the curve, which is not needed with fast ball powders in pistols.
Because changing the primer can change the pressure in the case, apply due dilligence when doing so for rifles.
For pistols it seems to make no practical difference. I've used PMP primers for small pistol that are designated 'magnum', and can't tell the difference with primers not marked 'magnum'. The Muron small pistol primers gave me slightly higher than average velocities, and they were not marked as magnum.
So to answer the question, when is a magnum primer not a good idea, I'd have to say only in the case of semi-auto pistols that use a large primer [ 10mm, 45ACP ], don't use magnum rifle primers in case you send the pressure too high too fast.