It used to annoy me as well, but various style guides have different guidance for pluralizing acronyms.
There are clear cases where an apostrophe is needed to form a plural, e.g. "Mississippi has four s's and four i's" or in a capitalized title, e.g. "KICKER MISSES THREE FG'S".
My OCD has...