Awesome Scalability, Availability, and Stability Back-end Design Patterns

A curated list of selected readings to illustrate Scalability, Availability, and Stability Design Patterns in Back-end Development.

What if your backend went slow?

Understand your problems: performance problem (slow for a single user) or scalability problem (fast for a single user but slow under heavy load) by reviewing some basic design principles.

What if your backend went down?

"Even if you lose all one day, you can build all over again if you retain your calm!" - Thuan Pham, CTO at Uber Technologies Inc.

Contributing

Please take a look at the contribution guidelines first. Contributions are always welcome!

Contents

• Principles
• Scalability
• Availability
• Stability
• Other Design Aspects
• Books
• Talks

Principles

• CAP Theorem and Trade-offs
• Scaling Up and Scaling Out
• ACID and BASE
• Synchronous and Asynchronous
• SQL and NoSQL
• Understand why Cache is King!
• Understand Latency
• Architecture Issues When Scaling Web Applications: Bottlenecks, Database, CPU, IO
• 20 Common Bottlenecks
• Advantages and Drawbacks of Microservices
• Avoid Overengineering
• Don't Repeat Yourself (DRY)
• Design for Loose-coupling
• Design for Resiliency
• Design for Self-healing when failures occur
• Design for Scale out
• Best Practices for Scaling Out
• Design for Evolution

Scalability

• Microservices

  • Thinking Inside the Container - Riot Games (8 part series)
  • Containerization at Pinterest
  • The Evolution of Container Usage at Netflix
  • Dockerizing MySQL at Uber
  • Testing of Microservices at Spotify

• Distributed Caching

  • Write-behind and Write-through
  • Eviction Policies
  • Peer-To-Peer Caching
  • Distributed Caching at Netflix with EVCache
  • Robust Memcache Traffic Analyzer at Box.com
  • How Etsy caches: Consistent Hashing and Cache Smearing

• Distributed Tracing

  • Distributed tracing with Pintrace at Pinterest
  • Analyzing distributed trace data at Pinterest
  • Distributed tracing at Uber

• Distributed Logging

  • Scalable and reliable log ingestion at Pinterest
  • Building DistributedLog at Twitter: High-performance replicated log service
  • Logging Service with Spark at CERN Accelerator
  • Logging and Aggregation at Quora
  • BookKeeper: Distributed Log Storage at Yahoo

• Distributed Messaging

  • Understanding When to use RabbitMQ or Apache Kafka
  • Running Kafka at scale at Linkedin
  • Delaying Asynchronous Message Processing with RabbitMQ at Indeed
  • Real-time Data Pipeline with Kafka at Yelp
  • Audit Kafka End-to-End at Uber (count each message exactly once, audit a message across tiers)
  • Deduplication Techniques
    • Real-time Deduping at Scale with Kafka-based Pipleline at Tapjoy
    • Delivering Billions of Messages Exactly Once: Deduping at Segment

• Storage

  • In-memory Storage
    • Optimizing Memcached Efficiency at Quora
    • Real-Time Data Warehouse with MemSQL on Cisco UCS
    • Moving to MemSQL for solving problems at Tapjoy: horizontally scalable, ACID compliant, MySQL compatibility
  • Durable Storage (typically Object Storage)
    • Amazon S3
      • Reasons for Choosing S3 over HDFS at Databricks
      • S3 in the Data Infrastructure at Airbnb
      • Quantcast File System on Amazon S3
      • Using S3 in Netflix Chukwa
    • Yahoo Cloud Object Store - Object Storage at Exabyte Scale

• NoSQL

  • Key-Value Databases (DynamoDB, Voldemort, Manhattan)
    • Scaling Mapbox infrastructure with DynamoDB Streams
    • Manhattan: Twitter’s distributed key-value database
    • Sherpa: Yahoo’s distributed NoSQL key-value store
  • Column Databases (Cassandra, HBase, Vertica, Sybase IQ)
    • Consistent Hashing in Cassandra
    • When NOT to use Cassandra?
    • Storing Images in Cassandra at Walmart Scale
    • Cassandra at Instagram
    • How Yelp Scaled Ad Analytics with Cassandra
    • How Discord Stores Billions of Messages with Cassandra
  • Document Databases (MongoDB, CouchDB)
    • eBay: Building Mission-Critical Multi-Data Center Applications with MongoDB
    • MongoDB at Baidu: Multi-Tenant Cluster Storing 200+ Billion Documents across 160 Shards
    • The AWS and MongoDB Infrastructure of Parse (acquired by Facebook)
    • Couchbase Ecosystem at LinkedIn
  • Graph Databases
    • Neo4j case studies with Walmart, eBay, AirBnB, NASA, etc
    • FlockDB: Distributed Graph Database for Storing Adjancency Lists at Twitter
    • Amazon Neptune
  • Datastructure Databases (Redis, Hazelcast)
    • How Twitter Uses Redis To Scale
    • How Twitter Uses Redis To Scale - Video
    • Scaling Slack’s Job Queue with Redis
    • Moving persistent data out of Redis at Github
  • Practical NoSQL resilience design pattern for the enterprise (eBay)

• RDBMS (MySQL, PostgreSQL)

  • Why SQL is beating NoSQL, and what this means for the future of data
  • Sharding MySQL at Pinterest
  • How Airbnb Partitioned Main MySQL Database in Two Weeks
  • Replication is the Key for Scalability & High Availability
  • How Twitch uses PostgreSQL
  • Scaling MySQL-based financial reporting system at Airbnb
  • Scaling to 100M at Wix: MySQL is a Better NoSQL
  • Why Uber Engineering Switched from Postgres to MySQL
  • Handling Growth with Postgres at Instagram

• Time Series Database (TSDB)

  • Beringei: high-performance time series storage engine at Facebook
  • Atlas: In-memory dimensional time series database at Netflix
  • Heroic: in-house time series database of Spotify

• HTTP Caching

  • Reverse Proxy (Nginx, Varnish, Squid, rack-cache)
  • CDN (Akamai, Amazon CloudFront)
    • NASA - Streaming 4K Live from the International Space Station Using CloudFront

• Concurrency

  • Message-Passing Concurrency
  • Software Transactional Memory
  • Dataflow Concurrency
  • Shared-State Concurrency
  • Concurrency series by Larry Osterman (Principal SDE at Microsoft)
    • Part 8 – Concurrency for scalability
    • Part 9 - APIs that enable scalable programming
    • Part 10 - How do you know if you’ve got a scalability issue?
    • Part 11 – Hidden scalability issues
    • Part 12 – Hidden scalability issues (cont)

• Event-Driven Architecture

  • Messaging
    • Publish-Subscribe
      • Autoscaling Pub-Sub Consumers at Spotify
      • Pulsar: Pub-Sub Messaging at Scale at Yahoo
      • Wormhole: Pub-Sub system at Facebook (2013)
    • Point-to-Point
    • Store-Forward
    • Request-Reply
  • Actors: Fire-forget and Fire-Receive-Eventually
  • Enterprise Service Bus
  • Domain Events
  • Event Stream Processing
    • Kafka Streams on Heroku
    • Bullet: Forward-Looking Query Engine for Streaming Data at Yahoo
    • Benchmarking Streaming Computation Engines at Yahoo
  • Event Sourcing
  • Command & Query Responsibility Segregation (CQRS)

• Load Balancing

  • Load Balancing infrastructure to support more than 1.3 billion users at Facebook
  • Load Balancing with Eureka at Netflix
  • Load Balancing at Yelp
  • Load Balancing at Github
  • Consistent Hashing to Improve Load Balancing at Vimeo
  • UDP Load Balancing at 500 pixel

• Parallel Computing

  • SPMD (Single Program Multiple Data): The Genetic Pattern
  • Master/Worker Pattern
  • Loop Parallelism Pattern: Extracting parallel tasks from loops
  • Fork/Join Pattern: Good for recursive data processing
  • MapReduce Pattern: Born for Big Data
  • Parallelize the rendering of web pages: Use case of Yelp.com

• Distributed Machine Learning

  • Scalable Deep Learning Platform On Spark In Baidu
  • Horovod: Uber’s Open Source Distributed Deep Learning Framework for TensorFlow
  • Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at Yelp
  • TensorFlowOnSpark: Distributed Deep Learning on Big Data Clusters at Yahoo
  • CaffeOnSpark: Distributed Deep Learning on Big Data Clusters at Yahoo

Availability

• Fail-over
• Replication
  • Master-Slave
  • Tree Replication
  • Master-Master
  • Buddy Replication
• NodeJS High Availability at Yahoo
• Every Day Is Monday in Operations - LinkedIn (11 part series)

Stability

• Circuit Breaker
• Always use timeouts (if possible)
• Let it crash/Supervisors: Embrace failure as a natural state in the life-cycle of the application
• Crash early: An error now is better than a response tomorrow
• Bulkheads: Partition and tolerate failure in one part
• Steady state: Always put logs on separate disk
• Throttling: Maintain a steady pace
• Multi-clustering: Improving Resiliency and Stability of a Large-scale Monolithic API Service at LinkedIn

Others

• Distributed Git server at Palantir
• Configuration management for distributed systems (using GitHub and cfg4j) at Flickr
• Seagull: Distributed system that helps running > 20 million tests per day at Yelp
• Cloud Bouncer: Distributed Rate Limiting at Yahoo
• Scalable gaming patterns on AWS (Sep 2017)
• Building a modern bank backend at Monzo
• Selecting a cloud provider at Etsy
• Architecture of Tripod (Flickr’s Backend)
• How eBay's Shopping Cart used compression techniques to solve network I/O bottlenecks
• Optimizing web servers for high throughput and low latency at Dropbox

Books

• The Art of Scalability
• Designing Data-Intensive Applications
• Web Scalability for Startup Engineers
• Scalability Rules: 50 Principles for Scaling Web Sites

Talks

• Harvard CS75 - Lecture 9: Scalability
• How We've Scaled Dropbox - Kevin Modzelewski, Back-end Engineer at Dropbox
• Lessons of Scale at Facebook - Bobby Johnson, Director of Engineering at Facebook
• Scaling Instagram Infrastructure - Lisa Guo, Instagram Engineering
• Scaling Pinterest - Marty Weiner, Pinterest’s founding engineer
• Designing for Failure: Scaling Uber's Backend by Breaking Everything - Matt Ranney, Chief Systems Architect at Uber
• Netflix Guide to Microservices - Josh Evans, Director of Operations Engineering at Netflix

Special Thanks

• Jonas Bonér, CTO at Lightbend, for the original inspiration