mirror of
https://github.com/autistic-symposium/blockchain-data-engineering-toolkit.git
synced 2025-04-26 02:39:15 -04:00
move web2 infrastrucutre stuff to the other repo
This commit is contained in:
bt3gl
parent
5fbab7ede1
commit
1871892587
131
.gitignore
vendored
131
.gitignore
vendored
@ -1,131 +0,0 @@
|
|||||||
# Byte-compiled / optimized / DLL files
|
|
||||||
__pycache__/
|
|
||||||
*.py[cod]
|
|
||||||
*$py.class
|
|
||||||
|
|
||||||
# C extensions
|
|
||||||
*.so
|
|
||||||
|
|
||||||
# Distribution / packaging
|
|
||||||
.Python
|
|
||||||
build/
|
|
||||||
develop-eggs/
|
|
||||||
dist/
|
|
||||||
downloads/
|
|
||||||
eggs/
|
|
||||||
.eggs/
|
|
||||||
lib/
|
|
||||||
lib64/
|
|
||||||
parts/
|
|
||||||
sdist/
|
|
||||||
var/
|
|
||||||
wheels/
|
|
||||||
pip-wheel-metadata/
|
|
||||||
share/python-wheels/
|
|
||||||
*.egg-info/
|
|
||||||
.installed.cfg
|
|
||||||
*.egg
|
|
||||||
MANIFEST
|
|
||||||
|
|
||||||
# PyInstaller
|
|
||||||
# Usually these files are written by a python script from a template
|
|
||||||
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
|
||||||
*.manifest
|
|
||||||
*.spec
|
|
||||||
|
|
||||||
# Installer logs
|
|
||||||
pip-log.txt
|
|
||||||
pip-delete-this-directory.txt
|
|
||||||
|
|
||||||
# Unit test / coverage reports
|
|
||||||
htmlcov/
|
|
||||||
.tox/
|
|
||||||
.nox/
|
|
||||||
.coverage
|
|
||||||
.coverage.*
|
|
||||||
.cache
|
|
||||||
nosetests.xml
|
|
||||||
coverage.xml
|
|
||||||
*.cover
|
|
||||||
*.py,cover
|
|
||||||
.hypothesis/
|
|
||||||
.pytest_cache/
|
|
||||||
|
|
||||||
# Translations
|
|
||||||
*.mo
|
|
||||||
*.pot
|
|
||||||
|
|
||||||
# Django stuff:
|
|
||||||
*.log
|
|
||||||
local_settings.py
|
|
||||||
db.sqlite3
|
|
||||||
db.sqlite3-journal
|
|
||||||
|
|
||||||
# Flask stuff:
|
|
||||||
instance/
|
|
||||||
.webassets-cache
|
|
||||||
|
|
||||||
# Scrapy stuff:
|
|
||||||
.scrapy
|
|
||||||
|
|
||||||
# Sphinx documentation
|
|
||||||
docs/_build/
|
|
||||||
|
|
||||||
# PyBuilder
|
|
||||||
target/
|
|
||||||
|
|
||||||
# Jupyter Notebook
|
|
||||||
.ipynb_checkpoints
|
|
||||||
|
|
||||||
# IPython
|
|
||||||
profile_default/
|
|
||||||
ipython_config.py
|
|
||||||
|
|
||||||
# pyenv
|
|
||||||
.python-version
|
|
||||||
|
|
||||||
# pipenv
|
|
||||||
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
|
|
||||||
# However, in case of collaboration, if having platform-specific dependencies or dependencies
|
|
||||||
# having no cross-platform support, pipenv may install dependencies that don't work, or not
|
|
||||||
# install all needed dependencies.
|
|
||||||
#Pipfile.lock
|
|
||||||
|
|
||||||
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
|
|
||||||
__pypackages__/
|
|
||||||
|
|
||||||
# Celery stuff
|
|
||||||
celerybeat-schedule
|
|
||||||
celerybeat.pid
|
|
||||||
|
|
||||||
# SageMath parsed files
|
|
||||||
*.sage.py
|
|
||||||
|
|
||||||
# Environments
|
|
||||||
.env
|
|
||||||
.venv
|
|
||||||
env/
|
|
||||||
venv/
|
|
||||||
ENV/
|
|
||||||
env.bak/
|
|
||||||
venv.bak/
|
|
||||||
|
|
||||||
# Spyder project settings
|
|
||||||
.spyderproject
|
|
||||||
.spyproject
|
|
||||||
|
|
||||||
# Rope project settings
|
|
||||||
.ropeproject
|
|
||||||
|
|
||||||
# mkdocs documentation
|
|
||||||
/site
|
|
||||||
|
|
||||||
# mypy
|
|
||||||
.mypy_cache/
|
|
||||||
.dmypy.json
|
|
||||||
dmypy.json
|
|
||||||
|
|
||||||
# Pyre type checker
|
|
||||||
.pyre/
|
|
||||||
.vercel
|
|
||||||
.DS_Store
|
|
||||||
12
README.md
12
README.md
@ -1,4 +1,4 @@
|
|||||||
## ⛓🛠 blockchain infrastructure && data engineering
|
## blockchain infrastructure
|
||||||
|
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
@ -7,7 +7,7 @@
|
|||||||
|
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
##### 👉 this repository holds some of my blockchain infrastructure MVPs (such as scalable event scanners and infrastructure setups for on-chain analysis and machine learning models training)
|
### chapters
|
||||||
|
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
@ -24,21 +24,17 @@
|
|||||||
- discussion, design, and implementation route of a protocol for management and bartering of game (NFT) assets
|
- discussion, design, and implementation route of a protocol for management and bartering of game (NFT) assets
|
||||||
|
|
||||||
* **research notes**:
|
* **research notes**:
|
||||||
- **[apache arrow](technologies/arrow_project.md)**
|
|
||||||
- **[rlp encoding](technologies/rlp_enconding.md)**
|
- **[rlp encoding](technologies/rlp_enconding.md)**
|
||||||
- **[spotify's luigi](technologies/luigi.md)**
|
|
||||||
- **[google's or-tools](technologies/or_tools.md)**
|
|
||||||
|
|
||||||
|
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### external resources
|
### cool resources
|
||||||
|
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
* **[urani trade experiments (on solana)](https://github.com/urani-trade)**:
|
* **[urani.trade's experiments on solana](https://github.com/urani-trade)**:
|
||||||
- a batching orderbook on mongodb: **[orderbook-poc-ts](https://github.com/urani-trade/orderbook-poc-ts)**
|
- a batching orderbook on mongodb: **[orderbook-poc-ts](https://github.com/urani-trade/orderbook-poc-ts)**
|
||||||
- a swap protocol for order intents: **[urani-swap-ts](https://github.com/urani-trade/urani-swap-ts)**
|
- a swap protocol for order intents: **[urani-swap-ts](https://github.com/urani-trade/urani-swap-ts)**
|
||||||
- a mev solver: **[solana-mev-agent-py](https://github.com/urani-trade/solana-mev-agent-py)**
|
- a mev solver: **[solana-mev-agent-py](https://github.com/urani-trade/solana-mev-agent-py)**
|
||||||
|
|||||||
@ -1,13 +0,0 @@
|
|||||||
## the arrow project
|
|
||||||
|
|
||||||
<br>
|
|
||||||
|
|
||||||
* the [arrow project](https://arrow.apache.org/) is an open-source, cross-language columnar in-memory data representation that is designed to accelerate big data processing. it was initially developed by the Apache Software Foundation and is now a top-level project of the foundation.
|
|
||||||
|
|
||||||
* arrow provides a standard for representing data in a columnar format that can be used across different programming languages and different computing platforms. this enables more efficient data exchange between different systems, as well as faster processing of data using modern hardware such as CPUs, GPUs, and FPGAs.
|
|
||||||
|
|
||||||
* one of the key benefits of Arrow is its memory-efficient design. because data is stored in a columnar format, it can be compressed more effectively than with traditional row-based storage methods. this can result in significant reductions in memory usage and faster processing times.
|
|
||||||
|
|
||||||
* arrow is also designed to be extensible, with support for a wide range of data types and operations. it supports many programming languages, including C++, Java, Python, and Rust, among others. Arrow also integrates with popular big data frameworks such as Apache Spark, Apache Kafka, and Apache Flink.
|
|
||||||
|
|
||||||
* arrow is a powerful tool for accelerating big data processing across different systems and programming languages. its columnar data format and memory-efficient design make it an attractive option for data-intensive applications that require fast and efficient data processing.
|
|
||||||
@ -1,6 +0,0 @@
|
|||||||
## luigi
|
|
||||||
|
|
||||||
<br>
|
|
||||||
|
|
||||||
* a python package that helps you build complex pipelines of batch jobs.
|
|
||||||
* [here is how a task looks like](https://luigi.readthedocs.io/en/stable/tasks.html).
|
|
||||||
@ -1,11 +0,0 @@
|
|||||||
## google's or-tools
|
|
||||||
|
|
||||||
<br>
|
|
||||||
|
|
||||||
### tl; dr
|
|
||||||
|
|
||||||
* the goal of optimization is to find the best solution to a problem out of a large set of possible solutions (or any feasible solution).
|
|
||||||
* all optimization problems have the following elements:
|
|
||||||
* the **objective**: the quantity you want to optimize. an optimal solution is one for which the value of the objective function is the best, i.e. max or min.
|
|
||||||
* the **constraints**: restrictions on the set of possible solutions, based on the specific requirements of the problem. a feasible solution is one that satisfies all the given constraints for the problem, without necessarily being optimal.
|
|
||||||
* [google's or-tools](https://developers.google.com/optimization/introduction) is an open-source software for combinatorial optimization, which seeks to find the best solution to a problem out of a very large set of possible solutions.
|
|
||||||
@ -2,8 +2,13 @@
|
|||||||
|
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
* RLP encoding is a key component of many blockchain and distributed ledger technologies, enabling efficient and secure storage and processing of complex data structures.
|
|
||||||
* RLP (Recursive Length Prefix) is a method for encoding arbitrary nested data structures in a compact binary format, commonly used in blockchain and distributed ledger technologies, such as Ethereum.
|
* rlp (recursive length prefix) is a method for encoding arbitrary nested data structures in a compact binary format, commonly used in blockchain and distributed ledger technologies, such as ethereum.
|
||||||
* RLP encoding works by recursively encoding the length and contents of each item in a data structure. each item is first encoded as a byte array, and then the length of the byte array is encoded in a prefix, followed by the byte array itself. this process is repeated for each item in the data structure, including any nested structures.
|
|
||||||
* because RLP encoding is deterministic, it also ensures that the encoded data is consistent and can be easily verified.
|
* rlp encoding is a key component of many blockchain and distributed ledger technologies, enabling efficient and secure storage and processing of complex data structures.
|
||||||
* in ethereum, RLP encoding is used to encode various types of data, such as transactions, blocks, and account state data. RLP encoding is also used to encode smart contract bytecode, which is then stored on the blockchain and executed by the Ethereum Virtual Machine (EVM).
|
|
||||||
|
* rlp encoding works by recursively encoding the length and contents of each item in a data structure. each item is first encoded as a byte array, and then the length of the byte array is encoded in a prefix, followed by the byte array itself. this process is repeated for each item in the data structure, including any nested structures.
|
||||||
|
|
||||||
|
* because rlp encoding is deterministic, it also ensures that the encoded data is consistent and can be easily verified.
|
||||||
|
|
||||||
|
* in ethereum, rlp encoding is used to encode various types of data, such as transactions, blocks, and account state data. rlp encoding is also used to encode smart contract bytecode, which is then stored on the blockchain and executed by the ethereum virtual machine.
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user