mirror of
https://github.com/binhnguyennus/awesome-scalability.git
synced 2024-10-01 01:06:14 -04:00
The Patterns of Scalable, Reliable, and Performant Large-Scale Systems
architectureawesomeawesome-listbackendbig-datacomputer-sciencedesign-patternsdevopsdistributed-systemsinterviewinterview-practiceinterview-questionslistsmachine-learningprogrammingresourcesscalabilitysystemsystem-designweb-development
CONTRIBUTING.md | ||
README.md |
Awesome Scalability, Availability, and Stability Back-end Design Patterns
A curated list of selected readings to illustrate Scalability, Availability, and Stability Design Patterns in Back-end Development.
What if your backend went slow?
Understand your problems: performance problem (slow for a single user) or scalability problem (fast for a single user but slow under heavy load) by reviewing some basic design concepts.
What if your backend went down?
"Even if you lose all one day, you can build all over again if you retain your calm!" - Thuan Pham, CTO at Uber Technologies Inc.
Contributing
Please take a look at the contribution guidelines first. Contributions are always welcome!
Contents
Basic
- CAP theorem and the trade-offs
- Scaling Up versus Scaling Out
- How to deal with latency
- What is ACID?
- Architecture issues: Bottlenecks, Database, CPU, IO
- Performance and Scalability patterns
- Advantages and drawbacks of Microservices
- Avoid Overengineering
- Don't Repeat Yourself (DRY)
- Design for Loose-coupling
- Design for Resiliency
- Design for Self-healing when failures occur
- Design for Scale out
- Design for Evolution
Scalability
-
- Why SQL is beating NoSQL, and what this means for the future of data
- Sharding MySQL at Pinterest
- How Airbnb Partitioned Main MySQL Database in Two Weeks
- Replication is the Key for Scalability & High Availability
- How Twitch uses PostgreSQL
- Scaling MySQL-based financial reporting system at Airbnb
- Scaling to 100M at Wix: MySQL is a Better NoSQL
- Why Uber Engineering Switched from Postgres to MySQL
- Handling Growth with Postgres at Instagram
-
- Scalable Deep Learning Platform On Spark In Baidu
- Horovod: Uber’s Open Source Distributed Deep Learning Framework for TensorFlow
- Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at Yelp
- TensorFlowOnSpark: Distributed Deep Learning on Big Data Clusters at Yahoo
- CaffeOnSpark: Distributed Deep Learning on Big Data Clusters at Yahoo
Availability
- Fail-over
- Replication
- NodeJS High Availability at Yahoo
- Every Day Is Monday in Operations - LinkedIn (11 part series)
Stability
- Circuit Breaker
- Always use timeouts (if possible)
- Let it crash/Supervisors: Embrace failure as a natural state in the life-cycle of the application
- Crash early: An error now is better than a response tomorrow
- Bulkheads: Partition and tolerate failure in one part
- Steady state: Always put logs on separate disk
- Throttling: Maintain a steady pace
- Multi-clustering: Improving Resiliency and Stability of a Large-scale Monolithic API Service at LinkedIn
Others
- Distributed Git server at Palantir
- Configuration management for distributed systems (using GitHub and cfg4j) at Flickr
- Seagull: Distributed system that helps running > 20 million tests per day at Yelp
- Cloud Bouncer: Distributed Rate Limiting at Yahoo
- Scalable gaming patterns on AWS (Sep 2017)
- Building a modern bank backend at Monzo
- Selecting a cloud provider at Etsy
- Architecture of Tripod (Flickr’s Backend)
Books
- The Art of Scalability
- Designing Data-Intensive Applications
- Web Scalability for Startup Engineers
- Scalability Rules: 50 Principles for Scaling Web Sites
Talks
- Harvard CS75 - Lecture 9: Scalability
- How We've Scaled Dropbox - Kevin Modzelewski, Back-end Engineer at Dropbox
- Lessons of Scale at Facebook - Bobby Johnson, Director of Engineering at Facebook
- Scaling Instagram Infrastructure - Lisa Guo, Instagram Engineering
- Scaling Pinterest - Marty Weiner, Pinterest’s founding engineer
- Designing for Failure: Scaling Uber's Backend by Breaking Everything - Matt Ranney, Chief Systems Architect at Uber
Special Thanks
- Jonas Bonér, CTO at Lightbend, for the original inspiration