18 Lessons Learned After 18 Years At Amazon

I’ve been at Amazon as a software developer continuously for over 18 years now. That’s longer than most. Lately I’ve been reflecting on my time there. Here are 18 lessons I've learned -- most of which are applicable to any collaborative endeavor.

	Trust is your most important asset. It’s slow to build and easy to lose. This is true inside the company (with teammates, reports, and managers) and outside (with customers, partners, and investors).
	Error correction is at the heart of development. Humans make errors, and a lot of humans make a lot of errors. This is unavoidable, and one should never blame the person – always the process. Swift and honest error correction builds trust. This, too, is true inside the company (with teammates, reports, and managers) and outside (with customers, partners, and investors). Transparency is the best policy.
	If you don’t have enough information, build a tool to gather more. Saying “I don’t know” is not a moral failing. When you don’t know something, say so. Can’t figure out why your program isn’t working for a particular customer, and you’re unable to reproduce the failure mode? Expand your logging and ask your customer to do it again.
	Automate your error correction. Automate your deployments. Automate your data collection (you’re going to need more metrics). Automate your alarms. You can only focus on one thing at a time, so build systems that gather data, and other systems that alert you when the data says things could be going wrong. Any repeated human task will eventually be done incorrectly, so build systems that act as guide rails. If bowling is a metaphor for your job, you want the gutter bumpers on all the time.
	Prioritize. There’s always a cost/benefit decision to be made. There will always be more work to be done than time to do it. Which customers do you say “no” to? Which bugs do you leave unfixed? Figure out which task has the highest ratio of value to time-and-effort, and do that. Remember that "value" can be positive (anticipated gain) or avoiding a negative (mitigating risk).
	There is no safe time to take your system offline. Whenever I log into my bank’s website and see “This system is down for maintenance, come back later”, I think: “That’s appalling. Amazon would never accept this low bar of service.” Amazon aims for figures like 99.95% uptime, leaving just a few hours per year. All of that time is expected to be accidental downtime (outages). Planned downtime is to be avoided.
	To develop is to communicate to future developers, including yourself. The most important thing software must communicate is intent. Don’t document the “how” (anyone reading the code can see the “how”). Document the “why”. Also document the “why not”, when a future reader is likely to think “why didn’t you do this another way that is obviously simpler or easier”. The future reader is likely to be you, perhaps just a few weeks from now.
	Use contracts. For your callers and your customers, document what inputs you expect to handle, what you promise to return, what errors could be produced, and if possible, how long it’s expected to take. For your dependencies and your suppliers, ask for documentation of their contracts. If you have many dependencies, the chances compound that one of them (or more) will be breaking their promises at any given time. Can you still meet your customers’ needs when your dependencies aren’t meeting yours? Do you have a backup plan? Or do you need to adjust your customers’ expectations to match?
	Whatever you build, put a solid API underneath it. Today you might be building a website, but tomorrow you may want to connect another system directly to your control layer. Even if that never happens, a decent API gives you something against which you can write integration tests.
	You can’t please everybody. Sometimes you have to say “no” to someone. It’s easier to say “no” at the beginning, so start with a more modest contract or a more narrow customer base. Customers will understand if you decline to support a feature. If you do offer a feature and then change your mind after your customers depend on it, they will not be happy and you’ll lose trust.
	If your API allows it, some customer will do it. It’s very hard to predict how customers will use what you build. Be clear about what uses are unsupported. If you’re changing the behavior of your system in what seems like an innocuous fashion, be prepared to deal with some irate customers raising objections. See Hyrum’s Law.
	The world changes, so the company’s priorities also change. Nobody can afford to get stuck in the past. Don’t succumb to the sunk cost fallacy.
	Innovation gets all the attention, but most work is maintenance. “Move fast and break things”, if what you’re breaking includes your customers, will also break their trust in you. They’re also trying to build products that depend on yours, and so the foundation you provide them should be solid. Don’t underestimate the cost of necessary upkeep: upgrades, bug fixes, and paying down tech debt. Ignore these at your peril, and never sacrifice quality.
	Don't try to do everything yourself. Rely on your teammates. Develop a plan, seek buy-in, and divide up the work so it can be done in parallel. Sometimes you'll need commitments from other teams, who have competing priorities. If they don't have time for you today, insist on a "date for a date" as a minimum commitment.
	Managers, listen to your engineers. Engineers, talk to your managers. Development does not happen in a vacuum. When developers foresee a problem, such as a system overburdened by tech debt, or a hole in an algorithm, don’t just shrug it off. Raise the alarm. Prioritize the maintenance.
	Making predictions is tough, especially about the future. If your strategy relies on knowing what’s going to happen next, take your best guess, but have a backup strategy in your pocket. In other words, build what you think you’re going to need, and then treat it as another error correction exercise if you prove to be wrong. This is often called a “pivot”, but it’s really a course correction.
	Know which are the two-way doors. Amazonians like to talk about one-way doors and two-way doors. Exposing an API, and allowing your customers to build against it, is a one-way door because it’s expensive to change your mind. Choosing to adopt an agile development process is a two-way door: you can reverse the decision easily. With one-way doors, gather as much information as you can, and make the decision as late as possible (but not too late). Don’t sweat the two-way doors. Make those decisions quickly and move on.
	Don't believe everything you read. Trust, but verify. These days the Internet is filled with nonsense. Take "listicles", for example, like this article. You were promised 18 lessons. But if you count the ones above, you'll see there are only 17! And this last one is obviously filler. Seriously, though: Even with good intentions, human endeavors are messy. Teams will break their own contracts (usually by accident) and their commitments to each other. Sometimes testing others' services is called for. When the company commits to a plan with broad impact, course changes may still happen. You have to roll with the punches.

---

Image attributions:

(1) Mountain climbers: Microsoft 365 stock images catalog
(2) Blue Screen Of Death: Akhristov at English Wikipedia. - Transferred from en.wikipedia to Commons by TFCforever., Public Domain, https://commons.wikimedia.org/w/index.php?curid=8782066
(3) Tool board: By Tom.koehler - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=18057400
(4) Power station computer terminals: By Achim Hering - Own work, Public Domain, https://commons.wikimedia.org/w/index.php?curid=2167047
(5) Task pin-up board: By David Falconer, Photographer (NARA record: 1427627) - U.S. National Archives and Records Administration, Public Domain, https://commons.wikimedia.org/w/index.php?curid=17173503
(6) Chillax: Google Wave maintenance page
(7) Rosetta Stone: By © Hans Hillewaert, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=3153928
(8) Sumerian contract: Public domain
(9) Windows API diagram: The original uploader was Grm wnr at English Wikipedia. Later versions were uploaded by Xyzzy n at en.wikipedia. - Transferred from en.wikipedia to Commons., CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=2242280
(10) Demonstration in Orléans: By DC - Own work, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=11766695
(11) The Incredible Machine: By Sierra On-Line - http://www.mobygames.com/game/incredible-machine/cover-art/gameCoverId,233121/, Fair use, https://en.wikipedia.org/w/index.php?curid=56078433
(12) Cabriolet: By Pearson Scott Foresman - Archives of Pearson Scott Foresman, donated to the Wikimedia Foundation, Public Domain, https://commons.wikimedia.org/w/index.php?curid=2566540
(13) Auto shop: By Butterflystring - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=88866066
(14) Castellers de la Vila de Gràcia: By Montserrat Torres - Castellers de la Vila de Gràcia, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=10265405
(15) NASA engineers, Mission Control, Apollo 13: By NASA - http://spaceflight.nasa.gov/gallery/images/apollo/apollo13/html/s70-35013.html (direct link), Public Domain, https://commons.wikimedia.org/w/index.php?curid=6641252
(16) Crystal ball: By PLBechly - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=107963301
(17) One-way revolving door: By Wfm - eigen werk, gepubliceerd op www.methon.nl (vrijgegeven) 
(18) The Magician: By Pamela Colman Smith - http://muzendo.jp/blog/, Public Domain, https://commons.wikimedia.org/w/index.php?curid=18886161