That's all well and good, but it doesn't fit the canon. Gandalf doesn't even come to Middle Earth until the Third Age, so how is he there during the Second Age? I will grant that he is likely alive at this time, but nowhere near this incarnation. From what I can tell, he wasn't even technically an Istar (wizard) until coming to Middle Earth (again in the Third Age). And pretty sure he never traveled by meteor.
I don't know enough about Saruman to speculate, I actually don't figure The Stranger as Saruman. He doesn't really seem to care one way or another about hobbits, which disqualifies him, IMO.
Look, I get it, as a fan it's super simple to just say he's Gandalf and be done with it. But the hints are so in-your-face, it doesn't make sense to me that didn't just confirm it. And so what if the "Who Is Sauron" question is the crux of the series? The whole thing with the Stranger has always been Plot B regardless, and for the most part isn't really related to the "Who is Sauron" plot, even with the 3 Rhunic weirdos saying he was.