back to article Amazon’s Away Teams laid bare: How AWS's hivemind of engineers develop and maintain their internal tech

Companies inside and out of Silicon Valley have found their own ways to rapidly develop and deploy features and functionality. Within the belly of Amazon Web Services, the web giant's gigantic cloud beast, though, is a specific digestive system – a concept called Away Teams – that accepts certain weaknesses to achieve maximum …

  1. steelpillow Silver badge
    Pint

    Open, open, open

    A lot of this seems to echo what the Open Source community has been evolving for decades. Source code accessible to all for use, adoption and modification. An architecture built around interfaces (compare the Amazon API mantra with the w3c RFC ecology). A direction driven by a positive tension between crowdsourced value and the influence of a few powerful visionaries. Amazon's only mistake has been not to expose their model before and engage the wider community in making it better.

    Because the history of the Open Source movement teaches us one more thing: have no fear of the competition, for if your competitor is full of sh*t then they won't adopt it anyway.

    1. T. F. M. Reader Silver badge

      Re: Open, open, open

      A lot of this seems to echo what the Open Source community has been evolving for decades.

      This was my first thought, too. And then I found myself wondering about the actual tools of the trade that support this model at AMZN. What languages, source control systems, tools and techniques do they use? How do they help? How common they are between the teams? How do they support the scaled-out development process? To me this was missing from the (very good) article.

      1. Anonymous Coward
        Anonymous Coward

        Re: Open, open, open

        It was only touched in passing but it seems the tools are selected to fit the job(s) at hand rather than be set on high or even required to have commonality cross-team. That's the second step of the way I did things, picking the tools to support the requirements. It was awfully nice if we already had it on hand and in use, but not a necessary condition. And, why I never used the same language twice. Heck, even the editor at hand varied.

  2. Anonymous Coward
    Anonymous Coward

    Two pizza rule - About six people

    I'm ashamed to say that such a team would include me, a wolf pack of one...

  3. Doctor Syntax Silver badge

    "accepts certain weaknesses to achieve maximum velocity"

    That explains a lot.

    1. Bronek Kozicki Silver badge

      An organization which strives for perfection won't survive long, because of the law of diminishing return.

      1. vtcodger Silver badge

        "An organization which strives for perfection won't survive long"

        In what way is that not a recipe for a world where nothing works quite right?

  4. livin' thing
    Stop

    I notice they don't risk using anyone else's cloud

    There's a lesson for us all in there somewhere.

  5. Anonymous Coward
    Anonymous Coward

    Yawn

    What Amazon and it's employees have accomplished is amazing. However, service-oriented collaboration has been around long enough that I remember having such involvement at MCI back in the late 90's as it related to their Frame Relay and ATM services (among others). And I'm sure examples existed before then, like the Apollo program at NASA.

    Anyone who has worked in a large company for any length of time quickly sees how important it is to cross the silos erected to carve out the little empires within a company. It's really hard to react to changing market dynamics and externalities without such a level of collaboration. But kudos to Amazon for codifying how they go about that.

    1. hammarbtyp Silver badge

      Re: Yawn

      The important thing with legacy is not to invent something but to make sure it has a sexy memorable name i.e AJAX

  6. nicomorr

    Interesting article. But surely MAWS is/was "Move to AWS". See https://news.ycombinator.com/item?id=19496719

  7. Anonymous Coward
    Anonymous Coward

    Long time...

    It's interesting that since 2011 things must have changed quite a bit. Amazon/AWS has just been steam-rolling along, while Google is increasingly known for killing services that people use and like if they can't immediately monetise them to same degree as ad-flinging.

    1. Bronek Kozicki Silver badge

      Re: Long time...

      If it has changed then I'd like to hear about it. The article is about principles and these tend not to change in a long time.

  8. rcxb

    > The policy is to do whatever it takes to provide value to the customer and not worry about duplication or deviating from standards. There is no waiting while the perfect shared service is being developed. There is no friction from having to use the perfect shared service.

    Perhaps these superior policies are why the amazon.com website is broken so damn often, with searches turning up zero results, sorting by price resulting in 1/10th as many hits, higher priced items are interspersed through the lower priced items, and utterly irrelevant products show up even though the keywords you used don't show up anywhere in the product page.

  9. J J Carter Silver badge
    Trollface

    Yeah, but...

    Where's my stuff?

  10. Slabfondler
    Joke

    If all you have is CRUD

    You treat everything as an API

    But "Everything is an API" - perhaps not true, but certainly more so today than in the past. The guys at AWS I've talked to do say, it's all API's - when you click something in the Management console, its making an API call - which you can make directly via the SDK, CLI, etc...

    I wonder how many redshirts they have on these away teams?

    1. Anonymous Coward
      Anonymous Coward

      Re: If all you have is CRUD

      Your most valuable skill is to recognize that you are wearing a red shirt and find another job before they force you to do it...

  11. Anonymous Coward
    Anonymous Coward

    "No driver to eradicate technical debt"

    combined with duplication of function and multiple development techniques and languages - I will be interested to see how this evolves because that's the kind of cruft that builds up and usually brings a) huge support issues then b) a clean up project

    1. Permidion
      Alert

      Re: "No driver to eradicate technical debt"

      I was also a bit surprised when I read that.

      Technical debt is not something you can ignore and put under the carpet: sooner or later it will make a sudden comeback and gnaw at your ankle and you will stumble.

      Maybe they made a distinction between the flag development teams who do the new stuff and need to do it fast, and some other grunt teams doing the tedious work of being sure the whole scaffold doesn't break down on its own weight.

      I certainly dont believe AWS can ignore technical debt altogether.

    2. Buzzword

      Re: "No driver to eradicate technical debt"

      When the technical debt gets too big, they just create a new service, move all the clients over to it, and toss out the old one. That's consistent with their philosophy of "no concern about duplication".

  12. hammarbtyp Silver badge

    Seems closer to microservices architecture than SOA, although that line is blurred. Especially because the services are closely aligned to the buisness capabilities

  13. Mike 137

    Sounds a bit like a PR justification for chaos to me - strongly suggesting that the business has grown to big to control.

    The assumption that referring to 'Management' will cause delay is merely an avowal that the said management must be crap. In reality a Manager is there solely to facilitate the work of those being managed, not to gum up the works. However the fashion increasingly seems to be to run an enterprise this way, and I suppose it makes accountability harder to bring home, so maybe there's an advantage there.

    1. Voidstorm
      Holmes

      "Pathalogical avoidance of accepting responsibility when the shit hits the fan"

  14. lxndr

    > The larger worry is that product cohesion and consistency may suffer.

    Heavy user of AWS' public services here, if you look at something simple like 'tag service resource (e.g. Lambda, Table, Bucket) with a k,v', these APIs are vastly different across services, both in naming and workings...

    1. DanWoodsEarly

      Love to find out more about API inconsistencies

      Lxndr,

      Can you drop me a line (@danwoodsearly on twitter)

      I would love to talk more about API inconsistencies.

      Cheers,

      -Dan

    2. Shaheed

      And not just API naming and working, the API tech too

      An example of what @lxndr mentions that annoys me is that sometimes the "tags", which are more-or-less everywhere, can be used to "name" an object, such that the Web management UI considers it a name, and sometimes not.

      Likewise, one can access EC2 through their Python "boto3" library, but not RDS. What even more irritating about having to learn two completely different ways to access these services is that the way they are presented in aws-shell is different again, even though there has clearly been an attempt to unify everything under it.

      Quite honestly, this seems amateurish to me.

  15. AdamWill

    data is never wrong

    "The emphasis on data reduces ideological passion. I may be right or you may be right, but we don’t need to fight it out. We will see in the end because the data is always right."

    This seems like a somewhat naive idea, because the problem with "the data is always right" is that it just begs the question: *what* data? What do you measure, how, and over what timescale? Unless all parties involved can magically agree on that (and how likely is that, if the debate is e.g. about a feature that enhances revenue but compromises user privacy - who's going to draw up a framework for measuring that tradeoff?), all you've done is move the question around a bit...

    1. cbars

      Re: data is never wrong

      I dunno, you introduced the question there. There were two metrics in the article, 1) number of users 2) Money made.

  16. andy 103
    Go

    Not avoiding technical debt and duplication

    The bits about *not* reducing technical debt and *not* avoiding duplication are especially true.

    This is why a lot of developers get bunched into the "you don't understand customers" bag. Because neither of those things are things your customers actually give a shit about. They are the people you make money off. It really is that simple. Amazon's success is because they've - on the whole - given customers what they want whilst avoiding micromanaging every aspect of their operation to the point where that would have been impossible.

  17. DBJDBJ

    SPOF, AMZN and MSFT

    There are good ideas and practices in here but ... Calling Architects "Bar Raizers" might be a sign of the key single point of failure (SPOF).

    I would like to think the AMZN CTO and "Bar Raizers" have developed a common vision and enterprise road map. But I am not hopeful.

    It is perhaps interesting to compare the histories of AMZN and MSFT. Windows is Mount Everest of technical debt, but AMZN are building one that might grow larger.

  18. Anonymous Coward
    Anonymous Coward

    "May I borrow your code?"

    "Sure, of course. But we're going to make this transfer visible to both my boss and your boss. Okay?"

  19. Anonymous Coward
    Anonymous Coward

    its been 48 hours...

    and the AWS partner portal (https://partnercentral.awspartner.com/SiteLogin) remains down due to:

    "...certain technical issues related to a service provider. We have been working closely with our service provider throughout the day, but at this time do not have an ETA on when a resolution will be available."

    the irony.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019