"... the passing gravitational waves only moved the mirrors by ..."
As I understand it, it's not that the mirrors are moved/disturbed but that the 'very fabric of space itself' is compressed then expanded along the direction of propogation of the gravitational wave as it passes by. Hence, a single detector arm can't detect a wave passing at right angles to its axis, so they have two detector arms at right angles to each other. (This rasise the question of what happens if a gravitational wave arrives from vertically above.)
If anyone has a more rigorous or correct explanation then please share it.