Re: [S3] users have to actively turn off security
"Since this keeps happening, it makes me wonder whether perhaps something about the process could be improved, so that users are less likely to turn off rather too much of the security?"
Or, many of the practices that organisations use fall far short of best practice. The existing business process was "turn off anything security related until it works, or just assume it doesn't work until security services are disabled", and the same policy is being applied to the cloud.
i.e host firewalls are generally disabled rather than enabled with relatively simple, open policies for internal networks. Or not installing endpoint protection on servers because it might affect something. Or being years behind in security patching in case something breaks. These are just the examples that quickly spring to mind, there are a lot more to add - while there may be valid reasons for some exceptions to security policies, but the exceptions are generally for more broad than they should be.
A lot of organisations seem to think the cloud is magically secure - it isn't, its just setup to provide many of the services that you wish you could provide with on-prem facilities at either a fraction of the price (logging/storage/DDoS) or with greater resilience/redundancy/flexibility/scalability to allow you to avoid some of the traps with on-premise environments.
If you deploy services to the cloud and follow the same steps (no host hardening, disabling security features, ignoring logging/alerting requirements, bypassing RBAC by giving accounts all roles, deploying services with differing security needs on shared hosts etc) you are likely to see similar results.The only thing that the cloud has really provided is more bandwidth/resources for attackers to do things (i.e. dump your S3 buckets) faster with minimal impact on your services.
TL;DR: this requires an organisational culture change regarding security rather than process changes at cloud providers. While automated best practice wizards and checks may help some, the majority of the organisations will ignore them.
Mines the half empty pint. Not just that, the heat is making it evaporate too...