"hydraulic DRUM brakes didn't need any sort of servo assist"
I don't know how old you are but drum brakes certainly did have servos in larger vehicles. High friction shoes are rather prone to fade and lower friction shoes required power actuation. Some degree of self-servo effect could be obtained by dual-cam (twin leading shoe) brakes, which extended to 4LS in some designs, or in the Vincent motorcycle by double drums on the front wheel, but brake fade remained a problem. The more leverage you need to obtain the necessary force, the further the pedal moves as the drum expands. A servo takes care of this by acting as more of a force than a distance multiplier.
Modern traffic is much faster and cars tend to travel much closer together than they did in the 1960s, so what were adequate brakes then would be wholly inadequate now (let alone increase in vehicle weights.)
Modern drum brakes are used as parking brakes; they are not really intended as emergency stoppers from motorway speeds.