Awesome Scalability, Availability, and Stability Back-end Design Patterns

A curated list of selected readings to illustrate Scalability, Availability, and Stability Design Patterns in Back-end Development. Concepts are explained in the articles of notable engineers (Werner Vogels, James Hamilton, Martin Fowler, Robert C. Martin, Tom White, etc) and high quality reference sources (highscalability.com, infoq.com, etc). Case studies are taken from battle-tested systems those are serving millions of users (Netflix, Instagram, Riot Games, LINE, Expedia, etc).

What if your Back-end went slow?

Understand your problems: performance problem (slow for a single user) or scalability problem (fast for a single user but slow under heavy load) by reviewing design principles. You can also check some talks of elite engineers from tech giants (Google, Facebook, Netflix, etc) to see how they build and scale their systems.

What if your Back-end went down?

"Even if you lose all one day, you can build all over again if you retain your calm!" - Thuan Pham, CTO at Uber Technologies Inc.

Community Power

Contributions are greatly welcome! You may want to take a look at the contribution guidelines. If you find this project helpful, please help me share it to more and more people! Thank you very much!

Contents

• Principles
• Scalability
• Availability
• Stability
• Performance
• Other Aspects
• Talks
• Books

Principles

• Principles of Chaos Engineering
• Finding the Order in Chaos
• The Clean Architecture - Robert C. Martin (Uncle Bob)
• The Twelve-Factor App
• 10 Common (Large-Scale) Software Architectural Patterns in a Nutshell
• CAP Theorem and Trade-offs
• CAP Twelve Years Later: How the "Rules" Have Changed (2012) - Eric Brewer, VP of Infrastructure at Google
• Scale Up or Scale Out, What it is and Why You Should Care
• Scaling Up vs Scaling Out: Hidden Costs
• ACID and BASE
• Blocking/Non-Blocking and Sync/Async
• Why Non-Blocking?
• SQL versus NoSQL
• SQL or NoSQL - Lesson Learned from Salesforce
• Consistent Hashing - Tom White, author of 'Hadoop: the Definitive Guide'
• Uniform Consistent Hashing
• Eventually Consistent - Werner Vogels, CTO at Amazon
• Cache is King!
• Anti-Caching
• Understand Latency
• Architecture Issues When Scaling Web Applications: Bottlenecks, Database, CPU, IO
• 20 Common Bottlenecks
• Life Beyond Distributed Transactions
• Relying on Software to Redirect Traffic Reliably at Various Layers
• Advantages and Drawbacks of Microservices
• Breaking Things on Purpose
• Avoid Over Engineering
• Scalability Worst Practices
• Use Solid Technologies - Don’t Re-invent the Wheel - Keep It Simple!
• Why Over-Reusing is Bad
• Performance is a Feature
• Make Performance Part of Your Workflow
• The Benefits of Server Side Rendering Over Client Side Rendering
• Writing Code that Scales
• AWS Do's and Don'ts
• (UI) Design Doesn’t Scale - Stanley Wood, Design Director at Spotify
• Design for Loose-coupling
• Design for Resiliency
• Design for Self-healing
• Design for Scaling Out
• Best Practices for Scaling Out
• Design for Evolution
• Learn from Mistakes

Scalability

• Microservices

  • Microservices Resource Guide - Martin Fowler, Chief Scientist at ThoughtWorks
  • Thinking Inside the Container - Riot Games (8 part series)
  • Containerization at Pinterest
  • The Evolution of Container Usage at Netflix
  • Dockerizing MySQL at Uber
  • Testing of Microservices at Spotify
  • Organize Monolith Before Breaking it into Services at Weebly
  • Lessons learned running Docker in production at Treehouse

• Distributed Caching

  • Write-behind and Write-through
  • Eviction Policies
  • Peer-To-Peer Caching
  • Distributed Caching with EVCache at Netflix
  • Robust Memcache Traffic Analyzer at Box.com
  • Caching with Consistent Hashing and Cache Smearing at Etsy
  • An Analysis of Facebook Photo Caching

• Distributed Tracking and Tracing

  • Tracking Service Infrastructure at Scale at Spotify
  • Distributed Tracing with Pintrace at Pinterest
  • Analyzing Distributed Trace Data at Pinterest
  • Distributed Tracing at Uber
  • Data Checking at Dropbox
  • Tracing distributed systems at Showmax

• Distributed Logging

  • The Log: What Every Software Engineer Should Know
  • Scalable and reliable log ingestion at Pinterest
  • Building DistributedLog at Twitter: High-performance replicated log service
  • Logging Service with Spark at CERN Accelerator
  • Logging and Aggregation at Quora
  • BookKeeper: Distributed Log Storage at Yahoo
  • LogDevice: Distributed Data Store for Logs at Facebook

• Distributed Messaging

  • Understanding When to use RabbitMQ or Apache Kafka
  • Running Kafka at scale at Linkedin
  • Delaying Asynchronous Message Processing with RabbitMQ at Indeed
  • Real-time Data Pipeline with Kafka at Yelp
  • Audit Kafka End-to-End at Uber (count each message exactly once, audit a message across tiers)
  • Deduplication Techniques
    • Exactly-once Semantics are Possible: Here’s How Kafka Does it
    • Real-time Deduping at Scale with Kafka-based Pipleline at Tapjoy
    • Delivering Billions of Messages Exactly Once: Deduping at Segment

• Distributed Searching

  • Search Architecture of Instagram
  • Search Architecture of eBay
  • Nautilus: Travel Search Engine of Expedia
  • Galene: Search Architecture of LinkedIn
  • Search at Slack
  • Search Service (Half a Trillion Documents and Query Average Latency < 100ms) at Twitter (2014)

• Distributed Storage

  • In-memory Storage
    • Optimizing Memcached Efficiency at Quora
    • Real-Time Data Warehouse with MemSQL on Cisco UCS
    • Moving to MemSQL at Tapjoy: Horizontally Scalable, ACID Compliant, MySQL Compatibility
  • Durable Storage (typically Object Storage)
    • Amazon S3
      • Reasons for Choosing S3 over HDFS at Databricks
      • S3 in the Data Infrastructure at Airbnb
      • Quantcast File System on Amazon S3
      • Using S3 in Netflix Chukwa
    • Yahoo Cloud Object Store - Object Storage at Exabyte Scale
    • Ambry: Distributed Immutable Object Store at LinkedIn
    • Hammerspace: Persistent, Concurrent, Off-heap Storage at Airbnb

• NoSQL

  • Key-Value Databases (DynamoDB, Voldemort, Manhattan)
    • Scaling Mapbox infrastructure with DynamoDB Streams
    • Manhattan: Twitter’s distributed key-value database
    • Sherpa: Yahoo’s distributed NoSQL key-value store
    • Riak inside Chat Service Architecture at Riot Games
  • Column Databases (Cassandra, HBase, Vertica, Sybase IQ)
    • Consistent Hashing in Cassandra
    • When NOT to use Cassandra?
    • Storing Images in Cassandra at Walmart Scale
    • Cassandra at Instagram
    • How Yelp Scaled Ad Analytics with Cassandra
    • How Discord Stores Billions of Messages with Cassandra
  • Document Databases (MongoDB, CouchDB)
    • eBay: Building Mission-Critical Multi-Data Center Applications with MongoDB
    • MongoDB at Baidu: Multi-Tenant Cluster Storing 200+ Billion Documents across 160 Shards
    • The AWS and MongoDB Infrastructure of Parse (acquired by Facebook)
    • Couchbase Ecosystem at LinkedIn
  • Graph Databases
    • Neo4j case studies with Walmart, eBay, AirBnB, NASA, etc
    • FlockDB: Distributed Graph Database for Storing Adjancency Lists at Twitter
    • JanusGraph: Scalable Graph Database backed by Google, IBM and Hortonworks
    • Amazon Neptune
  • Datastructure Databases (Redis, Hazelcast)
    • Using Redis To Scale at Twitter
    • Scaling Job Queue with Redis at Slack
    • Moving persistent data out of Redis at Github
    • Storing Hundreds of Millions of Simple Key-Value Pairs in Redis at Instagram
    • Redis in Chat Architecture of Twitch (from 27:22)
  • Practical NoSQL resilience design pattern for the enterprise (eBay)

• RDBMS (MySQL, MSSQL, PostgreSQL)

  • MS SQL versus MySQL
  • Why SQL is beating NoSQL, and what this means for the future of data
  • Sharding MySQL at Pinterest
  • How Airbnb Partitioned Main MySQL Database in Two Weeks
  • Replication is the Key for Scalability & High Availability
  • How Twitch uses PostgreSQL
  • Scaling MySQL-based financial reporting system at Airbnb
  • Scaling to 100M at Wix: MySQL is a Better NoSQL
  • Why Uber Engineering Switched from Postgres to MySQL
  • Handling Growth with Postgres at Instagram
  • Scaling the Analytics Database (Postgres) at TransferWise

• Time Series Database (TSDB)

  • Time Series Data: Why and How to Use a Relational Database instead of NoSQL
  • Beringei: High-performance Time Series Storage Engine at Facebook
  • Atlas: In-memory Dimensional Time Series Database at Netflix
  • Heroic: Time Series Database at Spotify
  • Building a Scalable Time Series Database on PostgreSQL
  • Scaling Time Series Data Storage at Netflix

• HTTP Caching (Reverse Proxy, CDN)

  • Reverse Proxy (Nginx, Varnish, Squid, rack-cache)
  • Stop Worrying and Love the Proxy
  • Playing HTTP Tricks with Nginx
  • Using CDN to Improve Site Performance at Coursera
  • Strategy: Caching 404s Saved 66% On Server Time at The Onion
  • Increasing Application Performance with HTTP Cache Headers

• Load Balancing

  • Introduction to Modern Network Load Balancing and Proxying
  • Load Balancing infrastructure to support more than 1.3 billion users at Facebook
  • DHCPLB: Open Source Load Balancer for DHCP at Facebook
  • Load Balancing with Eureka at Netflix
  • Load Balancing at Yelp
  • Load Balancing at Github
  • Consistent Hashing to Improve Load Balancing at Vimeo
  • UDP Load Balancing at 500 pixel

• Concurrency

  • Message-Passing Concurrency
  • Software Transactional Memory
  • Dataflow Concurrency
  • Shared-State Concurrency
  • Concurrency series by Larry Osterman (Principal SDE at Microsoft)
    • Part 8 – Concurrency for scalability
    • Part 9 - APIs that enable scalable programming
    • Part 10 - How do you know if you’ve got a scalability issue?
    • Part 11 – Hidden scalability issues
    • Part 12 – Hidden scalability issues (cont)
  • Concurrency with Erlang
    • Erlang in WhatsApp
    • Erlang in Riot Chat Server
    • How Discord Scaled Elixir to Five Millions Concurrent Users
    • Mnesia: A Distributed DBMS Rooted in Concurrency
    • Mesia and CAP

• Parallel Computing

  • SPMD (Single Program Multiple Data): The Genetic Pattern
  • Master/Worker Pattern
  • Loop Parallelism Pattern: Extracting parallel tasks from loops
  • Fork/Join Pattern: Good for recursive data processing
  • Map-Reduce: Born for Simplified Data Processing on Large Clusters
  • On the Death of Map-Reduce - Henry Robinson, Cloudera
  • Server-side Optimization to Parallelize the Rendering of Web Pages at Yelp

• Event-Driven Architecture

  • Messaging
    • Publish-Subscribe
      • Autoscaling Pub-Sub Consumers at Spotify
      • Pulsar: Pub-Sub Messaging at Scale at Yahoo
      • Wormhole: Pub-Sub system at Facebook (2013)
      • Pub-Sub in Chatting Architecture on LINE LIVE
    • Point-To-Point and Its Differences from Pub-Sub
    • Store-Forward
    • Request-Reply
  • Enterprise Service Bus
  • Domain Events
    • Domain Events: Simple and Reliable Solution
  • Event Stream Processing
    • Kafka Streams on Heroku
    • Bullet: Forward-Looking Query Engine for Streaming Data at Yahoo
    • Benchmarking Streaming Computation Engines at Yahoo
  • Event Sourcing
    • Event Sourced Architectures for High Availability
    • Event Sourcing and Stream Processing at Scale
    • Scaling Event Sourcing for Netflix Downloads
    • Scaling Event-Sourcing at Jet.com
  • Command & Query Responsibility Segregation (CQRS)

• Distributed Machine Learning

  • Scalable Deep Learning Platform On Spark In Baidu
  • Horovod: Uber’s Open Source Distributed Deep Learning Framework for TensorFlow
  • Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at Yelp
  • TensorFlowOnSpark: Distributed Deep Learning on Big Data Clusters at Yahoo
  • CaffeOnSpark: Distributed Deep Learning on Big Data Clusters at Yahoo
  • AIOps in Practice at Baidu
  • Learning with Privacy at Scale - Differential Privacy Team, Apple

Availability

• Failover
  • The Evolution of Global Traffic Routing and Failover
  • Testing for Disaster Recovery Failover Testing
• Replication
  • Master-Slave
  • Tree Replication
  • Master-Master
  • Buddy Replication
• NodeJS High Availability at Yahoo
• Every Day Is Monday in Operations - LinkedIn (11 part series)
• Practical Guide to Monitoring and Alerting with Time Series at Scale
• How Robust Monitoring Powers High Availability for LinkedIn Feed
• Architectural Patterns for High Availability - Adrian Cockcroft, Director of Architecture at Netflix
• Ensuring Resilience to Disaster at Quora

Stability

• Circuit Breaker
  • Circuit Breaking in Distributed Systems
  • Circuit Breakers for Distributed Services at LINE
  • Applying Circuit Breaker to Channel Gateway at LINE
  • Circuit Breaker for Scaling Containers
• Always use timeouts (if possible)
• Let it crash/Supervisors: Embrace failure as a natural state in the life-cycle of the application
• Crash early: An error now is better than a response tomorrow
• Bulkheads: Partition and tolerate failure in one part
• Steady state: Always put logs on separate disk
• Throttling: Maintain a steady pace
• Multi-clustering: Improving Resiliency and Stability of a Large-scale Monolithic API Service at LinkedIn

Performance

• Web Performance: Cache Efficiency Exercise at Facebook
• Improving Performance with Background Data Prefetching at Instagram
• Compression Techniques to Solve Network I/O Bottlenecks at eBay
• Optimizing Web Servers for High Throughput and Low Latency at Dropbox
• Boosting Site Speed Using Brotli Compression at LinkedIn
• Linux Performance Analysis in 60.000 Milliseconds at Netflix
• Optimizing 360 Photos at Scale at Facebook
• Optimizing APIs through Dynamic Polyglot Runtime, Fully Asynchronous, and Reactive Programming at Netflix

Others

• Distributed Git Server at Palantir
• Distributed Cron Architecture at Quora
• Configuration Management for Distributed Systems (using GitHub and cfg4j) at Flickr
• Seagull: Distributed System that Helps Running > 20 Million Tests Per Day at Yelp
• Cloud Bouncer: Distributed Rate Limiting at Yahoo
• Scalable gaming patterns on AWS (Sep 2017)
• Building a Modern Bank Backend at Monzo
• Selecting a cloud provider at Etsy
• Architecture of Tripod (Flickr’s Backend)
• Syscall Auditing at Scale at Slack

Talks

• Talks on Efficiency, Reliability, and Scaling - James Hamilton, Vice President and Distinguished Engineer at AWS
• Building Real Time Infrastructure at Facebook - Jeff Barber and Shie Erlich, Software Engineer at Facebook
• Building Reliable Social Infrastructure for Google - Marc Alvidrez, Senior Manager at Google
• How Google Does Planet-Scale for Planet-Scale Infra - Melissa Binde, SRE Director for Google Cloud Platform
• Netflix Guide to Microservices - Josh Evans, Director of Operations Engineering at Netflix
• Achieving Rapid Response Times in Large Online Services - Jeff Dean, Google Senior Fellow
• How We've Scaled Dropbox - Kevin Modzelewski, Back-end Engineer at Dropbox
• Lessons of Scale at Facebook - Bobby Johnson, Director of Engineering at Facebook
• Scaling Instagram Infrastructure - Lisa Guo, Instagram Engineering
• Scaling Twitter Core Infrastructure - Yao Yue, Staff Software Engineer at Twitter
• Scaling Pinterest - Marty Weiner, Pinterest’s founding engineer
• Scaling Spotify Data Infrastructure - Matti (Lepistö) Pehrs, Spotify
• Scaling Uber's Backend by Breaking Everything - Matt Ranney, Chief Systems Architect at Uber
• Scaling Slack - Bing Wei, Software Engineer (Infrastructure) at Slack

Books

• Google Site Reliability Engineering (Online - Free)
• Distributed Systems for Fun and Profit (Online - Free)
• Beyond the Twelve-Factor App - Exploring the DNA of Highly Scalable, Resilient Cloud Applications (Free)
• Chaos Engineering - Building Confidence in System Behavior through Experiments (Free)
• The Art of Scalability
• Designing Data-Intensive Applications
• Web Scalability for Startup Engineers
• Scalability Rules: 50 Principles for Scaling Web Sites

Special Thanks

• Jonas Bonér, CTO at Lightbend, for the original inspiration

License

This work is licensed under a Creative Commons Attribution 4.0 International License.