"the only person that should have a heart rate of 50 in the middle of the day is an endurance athlete, which he is most certainly not."
That simply isn't true, IMO Looking at heart rates in this overly narrow way is like using BMI to determine obesity. Testing someone's heart rate after telling them they may have a medical condition, is a great way of getting a higher reading.
Resting heart rates can differ greatly - <boast>while I'm in good shape I'm not an elite athlete, I was hooked up to a heart rate monitor at the hospital not long ago (so I assume was accurate) and my resting heart rate was 37.</boast>
There are many things that affect heart rates apart from cardiovascular training, age, sex, diet has a massive effect, stress levels, simple genetics, lifestyle.
I'm guessing your friend perhaps doesn't drink much tea, coffee or alcohol? Perhaps he doesn't eat much red meat or foods that are bad for cholesterol levels? Does he walk to work? - Many factors.
None of this is to say that Fitbits are accurate but that heart rates do vary greatly in perfectly healthy people, much like BMI.