mirror of
https://gitlab.com/veilid/veilid.git
synced 2024-10-01 01:26:08 -04:00
167 lines
11 KiB
HTML
167 lines
11 KiB
HTML
<!doctype html>
|
|
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
|
|
<title>Veilid Architecture Guide</title>
|
|
<meta name="description" content="a guide to the architecture of Veilid">
|
|
<meta name="author" content="beka">
|
|
|
|
<!-- <link rel="stylesheet" href="guide.css"> -->
|
|
|
|
</head>
|
|
|
|
<body>
|
|
<div id="content">
|
|
|
|
<h1>Veilid Architecture Guide</h1>
|
|
|
|
<hr/>
|
|
|
|
<ul class="section-toc">
|
|
<li>
|
|
<a class="section-name" href="#from-orbit">From Orbit</a>
|
|
</li>
|
|
<li>
|
|
<a class="section-name" href="#birds-eye-view">Bird's Eye View</a>
|
|
<ul class="subsection-toc">
|
|
<li>
|
|
<a class="subsection-name" href="#peer-network-for-data-storage">Peer Network for Data Storage</a>
|
|
</li>
|
|
<li>
|
|
<a class="subsection-name" href="#block-store">Block Store</a>
|
|
</li>
|
|
<li>
|
|
<a class="subsection-name" href="#key-value-store">Key-Value Store</a>
|
|
</li>
|
|
<li>
|
|
<a class="subsection-name" href="#structuring-data">Structuring Data</a>
|
|
</li>
|
|
<li>
|
|
<a class="subsection-name" href="#peer-and-user-identity">Peer and User Identity</a>
|
|
</li>
|
|
<li>
|
|
<a class="subsection-name" href="#peer-privacy">Peer Privacy</a>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<a class="section-name" href="#on-the-ground">On The Ground</a>
|
|
<ul class="subsection-toc">
|
|
<li>Section 2a</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
|
|
<hr/>
|
|
|
|
<h2 id="from-orbit">From Orbit</h2>
|
|
|
|
<p>
|
|
The first matter to address is the question "What is Veilid?" The highest-level description is that Veilid is a peer-to-peer network for easily sharing various kinds of data.
|
|
</p>
|
|
|
|
<p>
|
|
Veilid is designed with a social dimension in mind, so that each user can have their personal content stored on the network, but also can share that content with other people of their choosing, or with the entire world if they want.
|
|
</p>
|
|
|
|
<p>
|
|
The primary purpose of the Veilid network is to provide the infrastructure for a specific kind of shared data: social media in various forms. That includes light-weight content such as Twitter's tweets or Mastodon's toots, medium-weight content like images and songs, and heavy-weight content like videos. Meta-content such as personal feeds, replies, private messages, and so forth are also intended to run atop Veilid.
|
|
</p>
|
|
|
|
<hr/>
|
|
|
|
<h2 id="birds-eye-view">Bird's Eye View</h2>
|
|
|
|
<p>
|
|
Now that we know what Veilid is and what we intend to put on it, the second order of business is to address the parts of the question of how Veilid achieves that. Not at a very detailed level, of course, that will come later, but rather at a middle level of detail such that all of it can fit in your head at the same time.
|
|
</p>
|
|
|
|
<h3 id="peer-network-for-data-storage">Peer Network for Data Storage</h3>
|
|
|
|
<p>
|
|
The bottom-most level of Veilid is a network of peers communicating to one another over the internet. Peers send each other messages (remote procedure calls) about the data being stored on the network, and also messages about the network itself. For instance, one peer might ask another for some file, or it might ask for info about what other nodes exist in the network.
|
|
</p>
|
|
|
|
<p>
|
|
The data stored in the network is segmented into two kinds of data: file-like data, which typically is large, and textual data, which typically is small. Each kind of data is stored in its own subsystem specifically chosen to optimize for that kind of data.
|
|
</p>
|
|
|
|
<h3 id="block-store">Block Store</h3>
|
|
|
|
<p>
|
|
File-like content is stored in a content-addressable block store. Each block is just some arbitrary blob of data (for instance, a JPEG or an MP4) of whatever size. The hash of that block acts as the unique identifier for the block, and can be used by peers to request particular blocks. Technically, textual data can be stored as a block as well, and this is expected to be done when the textual data is thought of as a document or file of some sort.
|
|
</p>
|
|
|
|
<h3 id="key-value-store">Key-Value Store</h3>
|
|
|
|
<p>
|
|
Smaller, more ephemeral textual content generally, however, is stored in a key-value-store (KV store). Things like status updates, blog posts, user bios, etc. are all thought of as being suited for storage in this part of the data store. KV store data is not simply "on the Veilid network", but also owned/controlled by users, and identified by an arbitrary name chosen by the owner the data. Any group of users can add data, but can only change the data they've added.
|
|
</p>
|
|
|
|
<p>
|
|
For instance, we might talk about Boone's bio vs. Boone's blogpost titled "Hi, I'm Boone!", which are two things owned by the same user but with different identifiers, or on Boone's bio vs. Marquette's bio, which are two things owned by distinct users but with the same identifier.
|
|
</p>
|
|
|
|
<p>
|
|
KV store data is also versioned, so that updates to it can be made. Boone's bio, for instance, would not be fixed in time, but rather is likely to vary over time as he changes jobs, picks up new hobbies, etc. Versioning, together with arbitrary user-chosen identifiers instead of content hashes, means that we can talk about "Boone's Bio" as an abstract thing, and subscribe to updates to it.
|
|
</p>
|
|
|
|
<h3 id="structuring-data">Structuring Data</h3>
|
|
|
|
<p>
|
|
The combination of block storage and key-value storage together makes it possible to have higher-level concepts as well. A song, for instance, might be represented in two places in Veilid: the block store would hold the raw data, while the KV store would store a representation of the idea of the song. Maybe that would consist of a JSON object with metadata about the song, like the title, composer, date, encoding information, etc. as well as the ID of the block store data. We can then also store different <em>versions</em> of that JSON data, as the piece is updated, upsampled, remastered, or whatever, each one pointing to a different block in the block store. It's still "the same song", at a conceptual level, so it has the same identifier in the KV store, but the raw bits associated with each version differ.
|
|
</p>
|
|
|
|
<p>
|
|
Another example of this, but with even more tenuous connection between the block store data, is the notion of a profile picture. "Marquette's Profile Picture" is a really abstracted notion, and precisely which bits it corresponds to can vary wildly over time, not just being different versions of the picture but completely different pictures entirely. Maybe one day its a photo of Marquette and the next day it's a photo of a flower.
|
|
</p>
|
|
|
|
<p>
|
|
Social media offers many examples of these concepts. Friends lists, block lists, post indexes, favorites. These are all stateful notions, in a sense: a stable reference to a thing, but the precise content of the thing changes over time. These are exactly what we would put in the KV store, as opposed to in the block store, even if this data makes reference to content in the block store.
|
|
</p>
|
|
|
|
<h3 id="peer-and-user-identity">Peer and User Identity</h3>
|
|
|
|
<p>
|
|
Two notions of identity are at play in the above network: peer identity and user identity. Peer identity is simple enough: each peer has a cryptographic key pair that it uses to communicate securely with other peers, both through traditional encrypted communication, and also through the various encrypted routes. Peer identity is just the identity of the particular instance of the Veilid software running on a computer.
|
|
</p>
|
|
|
|
<p>
|
|
User identity is a slightly richer notion. Users, that is to say, *people*, will want to access the Veilid network in a way that has a consistent identity across devices and apps. But since Veilid doesn't have servers in any traditional sense, we can't have a normal notion of "account". Doing so would also introduce points of centralization, which federated systems have shown to be a source of trouble. Many Mastodon users have found themselves in a tricky situation when their instance sysadmins burned out and suddenly shut down the instance without enough warning.
|
|
</p>
|
|
|
|
<p>
|
|
To avoid this re-centralization of identity, we use cryptographic identity for users as well. The user's key pair is used to sign and encrypt their content as needed for publication to the data store. A user is said to be "logged in" to a client app whenever that app has a copy of their private key. When logged in a client app act like any other of the user's client apps, able to decrypt and encrypt content, sign messages, and so forth. Keys can be added to new apps to sign in on them, allowing the user to have any number of clients they want, on any number of devices they want.
|
|
</p>
|
|
|
|
<h3 id="peer-privacy">Peer Privacy</h3>
|
|
|
|
<p>
|
|
In order to ensure that peers can participate in Veilid with some amount of privacy, we need to address the fact that being connected to Veilid entails communicating with other peers, and therefore sharing IP addresses.
|
|
</p>
|
|
|
|
<p>
|
|
The approach that Veilid takes to privacy is two sided: privacy of the sender of a message, and privacy of the receiver of a message. Either or both sides can want privacy or opt out of privacy. To achieve sender privacy, we use something called a Safety Route: a sequence of any number of other peers, chosen by the sender, who will relay messages. The sequence of addresses is put into a nesting doll of encryption, so that each hope can see the previous and next hops, while no hop can see the whole route. This is similar to a Tor route, except only the addresses are hidden from view. Additionally, the route can be chosen at random for each send.
|
|
</p>
|
|
|
|
<p>
|
|
Receiver privacy is similar, in that we have a nesting doll of encrypted peer address, except because it's for incoming messages, the various addresses have to be shared ahead of time. We call such things Private Routes, and they are published to the key-value store as part of a peer's public data. For full privacy on both ends, a Private Route will be used as the final destination of a Safety Route, so that a total of four intermediate hops are used to send a message so that neither the sender nor receiver knows the IP address of the other.
|
|
</p>
|
|
|
|
<hr/>
|
|
|
|
<h2 id="on-the-ground">On The Ground</h2>
|
|
|
|
<p>
|
|
The bird's eye view of things makes it possible to hold it all in mind at once, but leaves out lots of information about implementation choice. It's now time to come down to earth and get our hands dirty.
|
|
</p>
|
|
|
|
<p>
|
|
TODO
|
|
</p>
|
|
|
|
</div>
|
|
</body>
|
|
</html> |