mirror of
https://repo.getmonero.org/monero-project/ccs-proposals.git
synced 2024-08-27 21:11:54 +00:00
217 lines
7.2 KiB
Markdown
217 lines
7.2 KiB
Markdown
---
|
|
layout: wip
|
|
title: Monero Kubernetes Operator
|
|
author: Ciro S. Costa (utxobr)
|
|
date: May 3, 2021
|
|
amount: 22.86
|
|
milestones:
|
|
- name: Proof of concept
|
|
funds: 0
|
|
done: 02 May 2021
|
|
status: finished
|
|
- name: Prototype refactoring, installation improvements and docs
|
|
funds: 2.47
|
|
done:
|
|
status: unfinished
|
|
- name: Support annonimity networks
|
|
funds: 3.71
|
|
done:
|
|
status: unfinished
|
|
- name: Improve observability of nodes
|
|
funds: 3.71
|
|
done:
|
|
status: unfinished
|
|
payouts:
|
|
- date:
|
|
amount:
|
|
- date:
|
|
amount:
|
|
- date:
|
|
amount:
|
|
---
|
|
|
|
|
|
## Brief Intro
|
|
|
|
My name is Ciro S. Costa (https://github.com/cirocosta,
|
|
https://twitter.com/utxobr), I'm currently a staff engineer, having previously
|
|
being a core contributor to https://concourse-ci.org.
|
|
|
|
Monero-wise, I've been mostly focused on the networking side of it, having
|
|
implemented the basics of Levin's handshake in Go
|
|
(https://github.com/cirocosta/go-monero) with full support for the
|
|
Portablestorage format, which lets me create some interesting reports on node
|
|
distribution (see https://twitter.com/utxobr/status/1386458317405540360) by
|
|
crawling the P2P network.
|
|
|
|
|
|
## Problem
|
|
|
|
_**tl;dr**: there's no good solution for running a large number of monero
|
|
nodes_
|
|
|
|
For those with more than a machine or two to run Monero nodes (or even miners),
|
|
there's not a good solution out there for having those up and running in an
|
|
easy to upgrade fashion.
|
|
|
|
It's great that folks like Seth provide wonderful guides on how to run Monero
|
|
nodes (see https://sethsimmons.me/guides/run-a-monero-node-advanced/), and that
|
|
within the functional tests in the codebase we can tell how to run regtest, but
|
|
none of that helps with running a larger-scale setup.
|
|
|
|
|
|
## Proposal
|
|
|
|
_**tl;dr**: extend the Kubernetes API via its common extension system to provide
|
|
semantics that make deploying clusters of monero nodes or miners with ease. See
|
|
proof of concept at https://github.com/cirocosta/monero-operator_
|
|
|
|
Kubernetes (see [what is kubernetes]) provides us with this vendor-neutral API
|
|
for expressing what the desired state should be, and then behind the scenes,
|
|
having that state achieved (and maintained) through the use of small
|
|
programs whose whole job is to deal with going from current state to desired state.
|
|
|
|
Aside from being offered by pretty much every cloud provider (and many VPS
|
|
offerings out there too) and still remaining not vendor-specific, its API is
|
|
open for extension, which we can leverage to provide extra functionality that
|
|
it didn't have before.
|
|
|
|
By extending the Kubernetes API via the use of [Custom Resources], we're able
|
|
to provide a new semantics for the users of those clusters so that we simplify
|
|
*a lot* running, say a few Monero nodes all configured the same across
|
|
different machines
|
|
|
|
|
|
```yaml
|
|
kind: MoneroNodeSet
|
|
apiVersion: utxo.com.br/v1alpha1
|
|
metadata:
|
|
name: nodes
|
|
spec:
|
|
replicas: 3
|
|
hardAntiAffinity: true
|
|
monerod:
|
|
image: utxobr/monerod:v0.17.2.0 # if testing a release candidate, then
|
|
args: # just bump the image and the operator
|
|
- --public # will take care of rolling out, preserving
|
|
- --enable-dns-blocklist # the data already synced.
|
|
- --enforce-dns-checkpointing
|
|
- --out-peers=1024
|
|
- --in-peers=1024
|
|
- --limit-rate=128000
|
|
```
|
|
|
|
which could be very useful for businesses like CakeWallet that run sets of full
|
|
nodes (or literally anyone wanting to run highly-available monerod
|
|
deployments), but it can be also useful for folks doing research like me,
|
|
wanting to roll out a regtest network with many peers:
|
|
|
|
```yaml
|
|
kind: MoneroNetwork
|
|
apiVersion: utxo.com.br/v1alpha1
|
|
metadata:
|
|
name: regtest
|
|
spec:
|
|
replicas: 20
|
|
|
|
template:
|
|
spec:
|
|
monerod:
|
|
args: # each replica has these args
|
|
- --regtest # plus `--add-exclusive-node`
|
|
- --fixed-difficulty=1 # pointing just at the other
|
|
# peers, forming a closed net
|
|
```
|
|
|
|
_(^ which under the hood gets materialized in the form of `monerod` instances
|
|
pointing one at each other, with volumes attached and everything you'd want for
|
|
a real setup.)_
|
|
|
|
Naturally, we can do the same for miners, for instance, we can get to run 10
|
|
replicas of `xmrig` against a pool like so:
|
|
|
|
```yaml
|
|
kind: MoneroMiningNodeSet
|
|
apiVersion: utxo.com.br/v1alpha1
|
|
metadata:
|
|
name: miners
|
|
spec:
|
|
replicas: 10
|
|
hardAntiAffinity: true
|
|
|
|
xmrig:
|
|
args:
|
|
- -o
|
|
- cryptonote.social:5556
|
|
- -u
|
|
- 891B5keCnwXN14hA9FoAzGFtaWmcuLjTDT5aRTp65juBLkbNpEhLNfgcBn6aWdGuBqBnSThqMPsGRjWVQadCrhoAT6CnSL3.node-$(id)
|
|
- --tls
|
|
```
|
|
|
|
and then, if we regret chosing that pool, all it takes is patching the object
|
|
and under the hood, our extension to Kubernetes takes care of rolling the
|
|
updates out.
|
|
|
|
_(aside: couple this with [horizontal pod autoscaler (HPA)] and you don't even
|
|
need to pre-provision any underlying machines - if your provider supports HPA -
|
|
as by making use of proper resource reservation, asking for extra replicas
|
|
would trigger the creation of new machines)._
|
|
|
|
|
|
[what is kubernetes]: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes
|
|
[Custom Resources]: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources
|
|
[horizontal pod autoscaler (HPA)]: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
|
|
[OpenMetrics]: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md
|
|
[Prometheus]: https://prometheus.io/
|
|
|
|
|
|
## The scope
|
|
|
|
I currently have a working proof of concept
|
|
(https://github.com/cirocosta/monero-operator) that implements those three
|
|
custom resources mentioned above (`MoneroMiningNodeSet`, `MoneroNodeSet`, and
|
|
`MoneroNetwork`).
|
|
|
|
This CCS would cover:
|
|
|
|
1. boosting the confidence in the codebase by providing more tests to cover
|
|
edge cases glanced over while building the prototype, as well as improving
|
|
installation and documentation as a whole
|
|
|
|
2. adding support for Tor and I2P so that nodes and networks can be deployed on
|
|
annonimity networks with a line or two in the yaml while still running the
|
|
services with high availability
|
|
|
|
3. improving the observability of the deployed `monerod` instances introducing a
|
|
sidecar to expose `monerod` metrics for any [OpenMetrics] consumer (like
|
|
[Prometheus])
|
|
|
|
|
|
As a result, the community will end up with:
|
|
|
|
- a Kubernetes extension that lets anyone deploy highly-available `monerod`
|
|
(and miners) on any Kubernetes-enabled platform
|
|
|
|
- a Go package that they can rely on for interacting with `monerod`
|
|
|
|
|
|
## The structure, milestones, and price.
|
|
|
|
Working on this during my personal hours, I plan to do the work a few hours a
|
|
day on the side (with a few healthy periods of break) until completion.
|
|
|
|
The proposal is structured to be paid along with the delivery of the three points above:
|
|
|
|
1. confidence in the codebase + installation/doc guides: ~10Hr
|
|
2. support for Tor and I2P for full nodes and whole networks: ~15Hr
|
|
3. observability of `monerod`: ~15Hr
|
|
|
|
Assuming a rate of 100$/hr and a current rate of 404 USD/xmr (May 3rd, 2021):
|
|
|
|
| deliverable | hours | usd | xmr |
|
|
|-----|------|-----|-----|
|
|
| 1 | 10 | $ 1000 | XMR 2.47 |
|
|
| 2 | 15 | $ 1500 | XMR 3.71 |
|
|
| 3 | 15 | $ 1500 | XMR 3.71 |
|
|
|