One per desktop (et al)
The reason it is the size it is: the range in sizes of the various parts meant that for the smallest to be strong enough (great enough cross-section), the largest are, well, the size they are. It is possible that a 1/4-scale model could be built today, using today's materials, today's fabrication processes, and designed with today's clustered supercomputers running FEA software. However, the _point_ of the original exercise was to determine if _Babbage_ could have built it, with the materials and processes available at the time.
As to "It's a computer, just look at it", well, then EMERAC is a computer :-)
(NOT! Nor is this)
As to the "Swede" who built a difference engine, that one lacked many of the error-checking facilities of Babbage's and was as a result a bit "fiddly", IIRC.
As to crank effort: the current machine has a 4:1 reduction gear. It takes a bit of patience, "feel", or whatever to turn, but I have done so and I am no giant. It would be tiring to turn it at 1:1, and more prone to jams (error-detecting, again), because of the variance in load during the machine cycle, but IMHO, not out of the question. You perhaps underestimate the strength, stamina, and flexibility of the typical Victorian Navvy. :-)