⛓🛠 blockchain data engineering

👉 this repository contains my blockchain engineering projects such as scalable event scanners and infrastructure setups for on-chain analysis and machine learning models training (e.g., HFT with deep learning).

🛠 here is a high-level system design chart for a possible blockchain intelligence data platform (all deployed in kubernetes):

---

scanners

• token-scanner-api:
  • a mvp for a scalable event scanner cli and api for ethereum, through indexing and parsing blocks events. this is the first step for training machine learning models on the chains (e.g., high-frequency trading with deep learning).
  • check my mirror post building a scalable event scanner for ethereum.

---

technologies

• the arrow project
• rlp enconding
• spotify's luigi
• google's or-tools

---

external resources

• go-outside-labs ml-htf-agents
• go-outside-labs orchestration-toolkit
• google biquery article on blockchain public datasets