Although he was in the Eddie Johnson role rather than play by play, George Raveling was far worse doing Pac 10 games than Steve Albert on his worst day while battling the flu, wearing a blindfold just hours after having all his teeth removed without anesthetic. But Albert was indeed awful.
As you said, the color commentator is a different role.
He has to be brisk in his comments. And be knowledgeable,
but that is his (or her) only responsibility.
But the play-by-play announcer has the responsibility
of setting the timing, as well as doing promos. And,
of course, never losing sight of what is happening
on the court and keeping the audience apprised as
the play is going on -- play-by-play (to coin a phrase).