There are Android phones with smaller than 5" screens, especially in your price range.
For starters:
Samsung S3, 4.7" screen, £180
Nokia 1, 4.5" screen, £80
There's also the option of the compact iPhone SE if you're not bothered about Android, a bit more expensive but refurbished models available.