Awesome Scalability, Availability, and Stability Back-end Design Patterns

A curated list of selected readings to illustrate Scalability, Availability, and Stability Design Patterns in Back-end Development.

What if your backend went slow?

Understand your problems: performance problem (slow for a single user) or scalability problem (fast for a single user but slow under heavy load) by reviewing some basic design concepts.

What if your backend went down?

"Even if you lose all one day, you can build all over again if you retain your calm!" - Thuan Pham, CTO at Uber Technologies Inc.

Contributing

Please take a look at the contribution guidelines first. Contributions are always welcome!

Contents

• Basic
• Scalability
• Availability
• Stability
• Other Design Aspects
• Awesome Books
• Awesome Talks

Basic

• CAP theorem and the trade-offs
• Scaling Up versus Scaling Out
  • Best Practices for Scaling Out
• How to deal with latency
  • Dropbox case: Striving for maximal throughput with acceptable latency
• What is ACID?
• Architecture issues: Bottlenecks, Database, CPU, IO
  • How eBay's Shopping Cart used compression techniques to solve network I/O bottlenecks
• Performance and Scalability patterns
• Advantages and drawbacks of Microservices
• Avoid Overengineering
• Don't Repeat Yourself (DRY)
  • DRY in Django
• Design for Loose-coupling
• Design for Resiliency
• Design for Self-healing when failures occur
• Design for Scale out
  • Design for Scale: Three best practices
• Design for Evolution

Scalability

• Microservices

  • Thinking Inside the Container - Riot Games (8 part series)
  • Containerization at Pinterest
  • The Evolution of Container Usage at Netflix
  • Dockerizing MySQL at Uber
  • Testing of Microservices at Spotify

• Distributed Caching

  • Write-behind and Write-through
  • Eviction Policies
  • Peer-To-Peer Caching
  • Distributed Caching at Netflix with EVCache
  • Robust Memcache Traffic Analyzer at Box.com
  • How Etsy caches: Consistent Hashing and Cache Smearing

• Distributed Logging & Tracing

  • Building DistributedLog at Twitter: High-performance replicated log service
  • Distributed tracing at Pinterest with Pintrace
  • Scalable and reliable log ingestion at Pinterest
  • CERN Accelerator Logging Service with Spark
  • Logging and Aggregation at Quora
  • BookKeeper: Distributed Log Storage at Yahoo

• Distributed Messaging

  • Understanding When to use RabbitMQ or Apache Kafka
  • Delaying Asynchronous Message Processing with RabbitMQ at Indeed
  • Yelp's Real-time Data Pipeline with Kafka
  • Real-time Deduping at Scale with Kafka-based Pipleline at Tapjoy

• Storage

  • In-memory Storage
    • Optimizing Memcached Efficiency at Quora
    • Real-Time Data Warehouse with MemSQL on Cisco UCS
    • Moving to MemSQL for solving problems at Tapjoy: horizontally scalable, ACID compliant, MySQL compatibility
  • Object Storage
    • Amazon S3
      • Reasons for Choosing S3 over HDFS at Databricks
      • S3 in the Data Infrastructure at Airbnb
      • Quantcast File System on Amazon S3
      • Using S3 in Netflix Chukwa
    • Other Object Storages
      • Yahoo Cloud Object Store - Object Storage at Exabyte Scale

• NoSQL

  • Key-Value Databases (DynamoDB, Voldemort, Manhattan)
    • Scaling Mapbox infrastructure with DynamoDB Streams
    • Manhattan: Twitter’s distributed key-value database
    • Sherpa: Yahoo’s distributed NoSQL key-value store
  • Column Databases (Cassandra, HBase, Vertica, Sybase IQ)
    • Consistent Hashing in Cassandra
    • When NOT to use Cassandra?
    • Storing Images in Cassandra at Walmart Scale
    • Cassandra at Instagram
    • How Yelp Scaled Ad Analytics with Cassandra
    • How Discord Stores Billions of Messages with Cassandra
  • Document Databases (MongoDB, CouchDB)
    • eBay: Building Mission-Critical Multi-Data Center Applications with MongoDB
    • MongoDB at Baidu: Multi-Tenant Cluster Storing 200+ Billion Documents across 160 Shards
    • The AWS and MongoDB Infrastructure of Parse (acquired by Facebook)
  • Graph Databases (Neo4j)
    • Neo4j at Airbnb
  • Datastructure Databases (Redis, Hazelcast)
    • How Twitter Uses Redis To Scale
    • How Twitter Uses Redis To Scale - Video
    • Scaling Slack’s Job Queue with Redis
    • Moving persistent data out of Redis at Github

• RDBMS (MySQL, PostgreSQL)

  • Why SQL is beating NoSQL, and what this means for the future of data
  • Sharding MySQL at Pinterest
  • How Airbnb Partitioned Main MySQL Database in Two Weeks
  • Replication is the Key for Scalability & High Availability
  • How Twitch uses PostgreSQL
  • Scaling MySQL-based financial reporting system at Airbnb
  • Scaling to 100M at Wix: MySQL is a Better NoSQL
  • Why Uber Engineering Switched from Postgres to MySQL
  • Handling Growth with Postgres at Instagram

• Time Series Database (TSDB)

  • Beringei: high-performance time series storage engine at Facebook
  • Atlas: In-memory dimensional time series database at Netflix

• HTTP Caching

  • Reverse Proxy (Nginx, Varnish, Squid, rack-cache)
  • CDN (Akamai, Amazon CloudFront)
    • NASA - Streaming 4K Live from the International Space Station Using CloudFront

• Concurrency

  • Message-Passing Concurrency
  • Software Transactional Memory
  • Dataflow Concurrency
  • Shared-State Concurrency

• Event-Driven Architecture

  • Messaging
    • Publish-Subscribe
      • Autoscaling Pub-Sub Consumers at Spotify
      • Pulsar: Pub-Sub Messaging at Scale at Yahoo
      • Wormhole: Pub-Sub system at Facebook (2013)
    • Point-to-Point
    • Store-Forward
    • Request-Reply
  • Actors: Fire-forget and Fire-Receive-Eventually
  • Enterprise Service Bus
  • Domain Events
  • Event Stream Processing
    • Kafka Streams on Heroku
    • Bullet: Forward-Looking Query Engine for Streaming Data at Yahoo
    • Benchmarking Streaming Computation Engines at Yahoo
  • Event Sourcing
  • Command & Query Responsibility Segregation (CQRS)

• Load Balancing

  • Round-robin Allocation
  • Random Allocation
  • Weighted Allocation
  • Dynamic Load Balancing
    • Work Stealing
  • Consistent Hashing
  • UDP Load Balancing
  • Cloud Load Balancing
    • AWS ELB: Application Load Balancer, Network Load Balancer, Classic Load Balancer
      • AWS ELB issues at Asana in 2012
    • Google Cloud Load Balancing

• Parallel Computing

  • SPMD (Single Program Multiple Data): The Genetic Pattern
  • Master/Worker Pattern
  • Loop Parallelism Pattern: Extracting parallel tasks from loops
  • Fork/Join Pattern: Good for recursive data processing
  • MapReduce Pattern: Born for Big Data
  • Parallelize the rendering of web pages: Use case of Yelp.com

• Distributed Machine Learning

  • Scalable Deep Learning Platform On Spark In Baidu
  • Horovod: Uber’s Open Source Distributed Deep Learning Framework for TensorFlow
  • Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at Yelp
  • TensorFlowOnSpark: Distributed Deep Learning on Big Data Clusters at Yahoo
  • CaffeOnSpark: Distributed Deep Learning on Big Data Clusters at Yahoo

Availability

• Fail-over
• Replication
  • Master-Slave
  • Tree Replication
  • Master-Master
  • Buddy Replication
• NodeJS High Availability at Yahoo

Stability

• Circuit Breaker
• Always use timeouts (if possible)
• Let it crash/Supervisors: Embrace failure as a natural state in the life-cycle of the application
• Crash early: An error now is better than a response tomorrow
• Bulkheads: Partition and tolerate failure in one part
• Steady state: Always put logs on separate disk
• Throttling: Maintain a steady pace
• Multi-clustering: Improving Resiliency and Stability of a Large-scale Monolithic API Service at LinkedIn

Others

• Distributed Git server at Palantir
• Configuration management for distributed systems (using GitHub and cfg4j) at Flickr
• Seagull: Distributed system that helps running > 20 million tests per day at Yelp
• Cloud Bouncer: Distributed Rate Limiting at Yahoo
• Scalable gaming patterns on AWS (Sep 2017)
• Building a modern bank backend at Monzo
• Selecting a cloud provider at Etsy
• Architecture of Tripod (Flickr’s Backend)

Books

• The Art of Scalability
• Designing Data-Intensive Applications
• Web Scalability for Startup Engineers
• Scalability Rules: 50 Principles for Scaling Web Sites

Talks

• Harvard CS75 - Lecture 9: Scalability
• How We've Scaled Dropbox - Kevin Modzelewski, Back-end Engineer at Dropbox
• Lessons of Scale at Facebook - Bobby Johnson, Director of Engineering at Facebook
• Scaling Instagram Infrastructure - Lisa Guo, Instagram Engineering
• Scaling Pinterest - Marty Weiner, Pinterest’s founding engineer
• Designing for Failure: Scaling Uber's Backend by Breaking Everything - Matt Ranney, Chief Systems Architect at Uber

Special Thanks

• Jonas Bonér, CTO at Lightbend, for the original inspiration