I'm just quoting CNET research, I haven't really tested the difference without HDR. My only 4k does have HDR and it does make a big difference vs my 1080p. Not huge, but noticeably more vibrant and colorful.
4k is definitely noticeable, especially in gaming even without HDR. It's so much cleaner. Playing games at 4k then switching to 1080 you see quite a difference. Also many games don't support HDR yet.
What they generally are talking about 'not that noticeable' is a bit off. It depends on the situation, usage, and person by person.
Mostly what they are talking about is watching movies. I don't have a 4k bluray player (but do have some discs lol) so I can't comment 1st hand, but supposedly there isn't a huge difference without HDR.
At this time, gaming is the vast majority of 4k content available for 4ktv's. You can basically play anything in 4k. Even with 4k blu rays you have to wonder if it's really 4k.
Here's a good site to check, or rather to be aware of as there are multiple sites to check on this matter.
http://realorfake4k.com/
Many 4k blurays aren't actually 4k. So you also have to make sure the people reviewing and trying to tell the difference are actually using a native 4k or higher source. Many aren't. Many are 2k, 2.5k upscaled to 4k.
But there's also streaming. If you compare a 4k stream to a 1080p blu ray, the 1080p bluray might actually have a higher bit rate. So 4k streaming is kind of a waste. You'll get the higher resolution, but at a sub par bit rate. Hopefully this improves over time.
Gaming is completely different. It's so obvious even without HDR.
Of course screen size and viewing distance makes an impact, but the whole you don't see any benefit of 1080p under 40" and 4k under 55" is mostly bs. Just look at your cell phone with a ~4-6 inch screen. Various resolution difference are completely noticeable. It's more about sweet spot. That at those screen sizes, you start to see the full benefit. But you of course can see and notice a benefit even at further distances or smaller screen sizes. It's a sliding scale more of a cutoff point.
Also of course, your mileage may vary based on eyesight.
My 4k doesn't have HDR, but it does have a native 120hz screen, so I can choose to play at 4k/60 or 1080/120. I love high framerate. 30 FPS games are a chore for me. Very hard for me to play them. Especially if they dip from 30. Like GTA V on PS4. Fun game, but it was consistently frustrating. Finally got around to picking it up on PC during this year's Steam Summer Sale for another play through and eventually some mods, and it simply is 1000x better on at 100-120FPS rather then 25-30.
So yeah, I'm still incredibly happy even without HDR. Obviously I would love to have it, but the 55" Vizio P series 120hz at 1080p and 4k/60 non-HDR is great for my needs. I can hold out until the format wars calm down, standards set like VRR, and 4k/120 hits the scene. (I do have a feeling OLED is going to be expensive for awhile so probably all of that and NOT OLED)