With muzzleloaders, shooting roundballs, too fast a twist rate will hurt accuracy. This depends on the caliber though. I have a few .54 caliber muzzloaders that shoot well with 1-70" and 1-66" twist. 1-48" doesn't work well for .54 (or .50 for that matter), but 1-66" is too slow for .32 or .36 cal roundballs, and 1-48" shoots well.
For centerfire rifles, it seems that it is best to err on the side of the fast twist. But too fast a twist here can cause thin jacket bullets, like varmint bullets, to come apart when they're driven to high velocity. I can do this in a 1-9" .22-250 barrel I have. The shot may be a ten, or an 'X'....or may just end up with a gray puff in the air, not making it to the target.
There is a theoretical ideal for size, based on, I believe it was, Greenhill's (GrenHills?) formula.