I don't think there is any doubt that the barrel-slide relationship is more important for accuracy than the slide-frame relationship.
When a 1911 type pistol is fired, the bullet begins to move forward and at the same time the barrel-slide unit begins to move to the rear. In spite of some erroneous pictures showing otherwise, the barrel and slide begin to move back together BEFORE the bullet leaves the barrel, so that any change in their relationship in that period can adversely affect accuracy. Remember the sights are on the slide. While a barrel pointing a bit off can be accurate and the sights can be adjusted to compensate, no compensation is possible if the relationship between barrel and slide changes from shot to shot. Such a change is entirely possible if the barrel and slide are not properly fitted to each other.
The relationship to the frame is not as important. But in a 1911 type pistol, the outside force that causes the barrel and slide to lock together and unlock is in the frame, in the form of the slide stop pin and the link. The frame thus controls the amount and timing of the locking of the barrel and slide, a very critical interface for both accuracy and safety, as well as wear.
So a very sloppy fit of the slide to the frame can cause variations in the locking and linkdown. It is for that reason, not simply tightness for its own sake, that makes slide-frame fit of importance.
Jim