No matter how much a team is loved by it's city, participation at the stadium is going to suffer if things go (really) sour. I was critical of the fans in St. Louis when they quit showing up to games because it was one of the only bullets left they had in their fight to keep the Rams. Not that sold out stadiums would have changed the eventual outcome but it would have been one thing less in support of a move.
Which obviously isn't the case for the Cardinals right now. As someone mentioned, there is a difference for teams in Arizona and LA and it's in two parts. One, if the team sucks, there are other things to do. If you don't want to suffer through watching the team struggle, you don't have to. In places like Cleveland, it's the football game or nothing and the weather means you're probably going to be miserable anyway so you might as well be miserable with everyone else at the game.
Second, when you're in a location where people actually want to be, they'll actually show up to watch their team from somewhere else. Getting to go to a game for a fraction of the cost (or at all) against a team that's struggling is a win/win.
For Cardinals fans it sucks to see all that orange but it ultimately isn't a reflection of the home town fans.