Obviously Smokey isn't human, since his human body died. But we already knew that. Smokey now appears to be his real form and Locke is a secondary form. Many believed the opposite.
As far as the spell, I think they can hurt each other, they just can't KILL each other.
As for Jacob staying with his "mother", I think naive as he was, he understood that his brother was the dark, and he couldn't go that way.
EDIT: Another interesting theory is that Jacob's brother is truly dead and the Smoke Monster isn't related to Jacob at all. It can appear as dead people (like their real mother and later as Jacob's brother/Christian/Locke) and it manipulated Jacob's brother. For what purpose, I don't know. Jacob accused his mother of loving the brother more--perhaps the smoke monster believed that he was the one destined to be the new protector, so he set events in motion to eliminate the protector. But it's plan backfired when his brother's dead body appeared in the "cave" or whatever it was and it realized it had made a mistake--and that's why it wants to kill Jacob. But for some reason, the smoke monster cannot kill the protector of the island. Then again, why didn't it just kill Jacob's brother straight off and bother with such an elaborate setup.
Massive FAIL on this episode. IMO the worst of the season. Very disappointing.