I haven't felt that geek was a derogatory term since the 90s. It's a term that suggests a certain level of intelligence and understanding of a subject that most of the world finds unfathomable. It's a title I proudly wear.
Boffin, likewise, denotes a person who is very intelligent and has knowledge far beyond that of mear mortals. If it's ever used in a derogatory sense then you should be laughing at the gormless prats with the audacity to misuse such an honorable term is such a way rather than feeling derided by them.
Just my two cents from a Merkin point of view.