I myself erupted at the thought of Ledger as the Joker. Not because of his physicality, but because I didn't think he had the chops for it. Chops don't gain you twenty or so pounds of muscle.
You are one of the few. Not only did I give other examples but there were a ton of people that said Ledger didn't look the part. The first production still came out and people said it looked more like a psycho killer and not anything like the Joker.