"We keep, or even raise duties on anything we net export - e.g. lamb"
Do you believe EU or other countries won't raise their duties on UK exports, whenever needed?
EU always had strong protections for its agricultural market (if they are good or not is another issue) - once UK is out and without a deal, it will have to face that - and countries like France are not easy to deal with - especially if they see they can have advantages.
The very idea that UK alone could always decide and set whatever it likes was what brought to Brexit, and blocked any sensible deal until now.