Git-Mirrors/blockchain-data-engineering-toolkit
1
0
Fork 0
mirror of https://github.com/autistic-symposium/blockchain-data-engineering-toolkit.git synced 2025-07-19 21:28:44 -04:00
Code Issues Projects Releases Packages Wiki Activity

Clean up

Browse source
This commit is contained in:
writer 2024-10-15 11:39:29 +09:00
parent 49c2206dc1
commit 5fbab7ede1
13 changed files with 808 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

52
README.md
View file

@ -1,38 +1,33 @@
## ⛓🛠 blockchain data engineering
## ⛓🛠 blockchain infrastructure && data engineering
<br>
##### 👉 this repository contains some pf my blockchain engineering projects such as scalable event scanners and infrastructure setups for on-chain analysis and machine learning models training.
##### 🛠 here is a high-level system design chart for a possible blockchain intelligence data platform (all deployed in kubernetes):
<br>
<p align="center">
<img src="https://user-images.githubusercontent.com/1130416/224561453-274c5066-240d-4cc5-b63b-b4c57388a0e0.png" width="70%" align="center" style="padding:1px;border:1px solid black;"/>
<br>
<br>
---
### in this repo
##### 👉 this repository holds some of my blockchain infrastructure MVPs (such as scalable event scanners and infrastructure setups for on-chain analysis and machine learning models training)
<br>
* **[token-scanner-api](token-scanner-api)**:
- a mvp for a **scalable event scanner cli and api for ethereum**, through indexing and parsing blocks events. this is the first step for training **machine learning models on the chains** (e.g., high-frequency trading with deep learning).
- check my mirror post **[on building a scalable event scanner for ethereum](https://mirror.xyz/steinkirch.eth/vSF18xcLyfXLIWwxjreRa3I_XskwgnjSc6pScegNJWI)**.
* **[apache arrow](technologies/arrow_project.md)**
* **[rlp enconding](technologies/rlp_enconding.md)**
* **[spotify's luigi](technologies/luigi.md)**
* **[google's or-tools](technologies/or_tools.md)**
- a mvp for a **scalable event scanner cli and api for ethereum**, through indexing and parsing blocks events. this is the first step for training **machine learning models on the chains** (e.g., high-frequency trading with deep learning)
- check my mirror post **[on building a scalable event scanner for ethereum](https://mirror.xyz/go-outside.eth/vSF18xcLyfXLIWwxjreRa3I_XskwgnjSc6pScegNJWI)**
* **[implementation of a draft of an EIP for rental NFTs with rights management](eip-9999)**:
- this standard is an extension of ERC-721 and ERC-4907
- it proposes an API for rights management through a rental license
- this framework is fully compatible with the licensing framework proposed on ERC-5218, although it can be implemented independently
* **[a protocol for management and bartering of game (NFT) assets](game-protocol-design)**:
- discussion, design, and implementation route of a protocol for management and bartering of game (NFT) assets
* **research notes**:
- **[apache arrow](technologies/arrow_project.md)**
- **[rlp encoding](technologies/rlp_enconding.md)**
- **[spotify's luigi](technologies/luigi.md)**
- **[google's or-tools](technologies/or_tools.md)**
<br>
@ -43,7 +38,14 @@
<br>
* **[go-outside-labs ml-hft-agents](https://github.com/go-outside-labs/hft-deep-learning-agents)**
* **[go-outside-labs orchestration-toolkit](https://github.com/go-outside-labs/orchestration-toolkit)**
* **[urani trade experiments (on solana)](https://github.com/urani-trade)**:
- a batching orderbook on mongodb: **[orderbook-poc-ts](https://github.com/urani-trade/orderbook-poc-ts)**
- a swap protocol for order intents: **[urani-swap-ts](https://github.com/urani-trade/urani-swap-ts)**
- a mev solver: **[solana-mev-agent-py](https://github.com/urani-trade/solana-mev-agent-py)**
- a graph visualization of order fulfillment: **[urani-arena-graph-ts](https://github.com/urani-trade/urani-arena-graph-ts)**
* **[autistic symposium's ml-ai-agents-py](https://github.com/autistic-symposium/ml-ai-agents-py)**
* **[autistic symposium's backend-and-orchestration-toolkit](https://github.com/autistic-symposium/backend-and-orchestration-toolkit)**
* **[google biquery article on blockchain public datasets](https://cloud.google.com/blog/products/data-analytics/introducing-six-new-cryptocurrencies-in-bigquery-public-datasets-and-how-to-analyze-them)**
* **[paradigm's data portal](https://data.paradigm.xyz/)**