I highly recommend that you (and everyone) get your hands on the steampunk graphic novel "The Thrilling Adventures of Lovelace and Babbage" by Sydney Padua.
On page 19 you will find a cartoon illustration of an imaginary one of "Babbage's famous parties". It includes everyone you mentioned except Brewster, and also includes others whom Babbage knew:
William Whewell ( polymath, scientist, Anglican priest, philosopher, theologian, historian of scienc, Master of Trinity College, Cambridge)
John and Caroline Herschel
the Duke of Wellington
Alfred, Lord Tennyson
Harriet Martineau ("the first female sociologist")
I cannot recommend it highly enough.