Re: how about..
> you'd have to be very careful to ensure it captures only phones inside the designated areas
Perhaps a directional antenna of some sorts is in order... Yes, there's a certain amount of RF leakage in all directions around a directional antenna, but in prisons with nice thick, high walls this is probably entirely manageable. Mount the antennas high up pointing in & slightly down. Muck around with transmit power so they don't overshoot the walls on the other side of the compound by enough to trigger any handover attempts from outside.
All they need to do is make sure that their "fake" cells are the dominant signal inside the prison in most areas, and are not dominant outside the prison.
As to the "mobile flagging up if encryption isn't enabled" - if the operator is deploying these "fake" cells, then they can be deployed in the operators domain, and so encryption needn't be disabled. All they need to do is to configure the cell to not actually provide any service, which is a doddle.