It doesn't actually happen
Well, this review of the literature suggests an effect.
I've found keeping it reasonably full reduces the duty cycle of my fridge a lot, as does reducing the door openings if it's less full. I live in a warmish climate with no A/C.