Go to file
Rivka Segan 5dd0a3793b call set_record_data_size with accumulated size
set_subkey corrupts record_data_size in a Record struct by calling
set_record_data_size with a value that depends only on the length of
the new subkey value. This leads to various undesirable outcomes, such
as: applications can write more than MAX_RECORD_DATA_SIZE without
encountering the intended "veilid.error.VeilidAPIErrorGeneric:
label='Generic' message='dht record too large'" error message, and
"panicked at 'attempt to subtract with overflow'" (i.e., an attempt to
set a negative value of a size) if a subkey's new length is less than
a subkey's old length. Typically, record_data_size in a Record struct
will be incorrect if a value was set for more than one subkey. Some
users might want to start over with a table_store that doesn't have
any incorrect record_data_size values.

The issue begins here:
6f71c6a00a/veilid-core/src/storage_manager/record_store.rs (L583-L586)

and is triggered here:
6f71c6a00a/veilid-core/src/storage_manager/record_store.rs (L613-L615)

It should be clear that new_record_data_size is only related to the
subkey that is currently being set. The amount of data in the record,
before set_subkey is called, is ignored. It appears that
new_total_size, not new_record_data_size, was intended to be used for
set_record_data_size, and this change succeeds for me in limited
testing but I don't have a comprehensive test suite.

One way to reproduce is by running the code from
https://gitlab.com/vatueil/veilid-file on a greater than 1 MB file
while watching variable values within
veilid-core/src/storage_manager/record_store.rs. For example: "poetry
run file put /usr/bin/tcpdump" (1.3 MB on Ubuntu 23.04). With the
original Veilid code, each of the dozens of subkey writes is checking
whether a roughly 32K value is greater than 1048576, it never is, and
thus there is never a "dht record too large" error. With the patch in
this MR, each of the dozens of subkey writes is checking whether an
ever-increasing value is greater than 1048576, it eventually is, and
the "dht record too large" error is printed. With the patch, one can
work with smaller files, e.g., do "poetry run file put /usr/bin/ssh"
(0.8 MB) followed by "poetry run file get VLD0:<_insert_key_here_>
ssh-copy" and the retrieved file ssh-copy is identical to
/usr/bin/ssh.

The more detailed behavior is that the modified code has
record.total_size of 350 on the first iteration, then 33596, 66842,
100088, etc. The original code also has record.total_size of 350 on
the first iteration, but then stays at 33246 forever (33246 is the
user-supplied subkey size of 32768, plus 128, plus the minimum record
size of 350),
2023-08-26 07:08:47 +00:00
.cargo wasm fixes 2022-11-29 22:51:51 -05:00
dev-setup Fix setup_linux.sh to use scripts from the parent directory 2023-08-19 12:29:30 -04:00
doc network fixes 2023-07-19 10:07:51 -04:00
docs/guide Fix minor typos 2023-08-21 20:12:20 -05:00
external update submodules 2023-07-19 14:41:10 -04:00
files Add Test CA and simple certs for testing 2021-11-22 09:02:41 -05:00
package service restart if running, instructs to start if not 2023-07-17 20:55:45 -05:00
scripts Fix minor typos 2023-08-21 20:12:20 -05:00
veilid-cli veilid-server with Clap v4 2023-08-22 21:12:23 +00:00
veilid-core call set_record_data_size with accumulated size 2023-08-26 07:08:47 +00:00
veilid-flutter flutter fix 2023-08-24 18:35:37 -04:00
veilid-python veilid-python: update_schema.sh now checks in a few places for veilid-server 2023-08-24 23:30:44 +00:00
veilid-server veilid-server with Clap v4 2023-08-22 21:12:23 +00:00
veilid-tools Version update: v0.1.9 → v0.1.10 2023-08-20 11:37:18 -05:00
veilid-wasm xfer 2023-08-22 15:11:45 -04:00
.bumpversion.cfg Version update: v0.1.9 → v0.1.10 2023-08-20 11:37:18 -05:00
.earthlyignore add eartly build 2022-01-09 15:17:36 -05:00
.gitignore Ignore emacs backup files 2023-08-12 18:10:32 +02:00
.gitlab-ci.yml Tests passed. This commit is a clean up of .gitlab-ci.yml 2023-08-22 21:49:55 +00:00
.gitmodules make submodule path absolute 2023-08-18 01:13:09 -04:00
Cargo.lock flutter fix 2023-08-24 18:35:37 -04:00
Cargo.toml version bump 2023-07-16 14:35:51 -04:00
CHANGELOG.md add more changelog and clean up test output 2023-08-20 12:10:54 -04:00
code_of_conduct.md Add docs and mod version bumping 2023-08-01 21:37:38 -05:00
CONTRIBUTING.md Update CONTRIBUTING.md 2023-08-10 19:32:36 +00:00
DEVELOPMENT.md Update DEVELOPMENT.md to reflect new paths. 2023-08-18 09:27:17 -04:00
Earthfile decompression fix for OOM 2023-08-19 18:13:30 -04:00
INSTALL.md Update file INSTALL.md 2023-08-24 17:41:35 +00:00
LICENSE Add LICENSE 2023-08-10 19:34:50 +00:00
README.md fix: README typos 2023-08-12 21:17:31 -07:00
RELEASING.md Revise release process 2023-08-19 17:19:17 -05:00
version_bump.sh version 2023-07-19 13:01:55 -04:00

Welcome to Veilid

From Orbit

The first matter to address is the question "What is Veilid?" The highest-level description is that Veilid is a peer-to-peer network for easily sharing various kinds of data.

Veilid is designed with a social dimension in mind, so that each user can have their personal content stored on the network, but also can share that content with other people of their choosing, or with the entire world if they want.

The primary purpose of the Veilid network is to provide the infrastructure for a specific kind of shared data: social media in various forms. That includes light-weight content such as Twitter's tweets or Mastodon's toots, medium-weight content like images and songs, and heavy-weight content like videos. Meta-content such as personal feeds, replies, private messages, and so forth are also intended to run atop Veilid.

Run a Node

The easiest way to help grow the Veilid network is to run your own node. Every user of Veilid is a node, but some nodes help the network more than others. These network support nodes are heavier than the node a user would establish on their phone in the form of a chat or social media application. A cloud based virtual private server (VPS), such as Digital Ocean Droplets or AWS EC2, with high bandwidth, processing resources, and up time availability is crucial for building the fast, secure, and private routing that Veilid is built to provide.

To run such a node, establish a Debian or Fedora based VPS and install the veilid-server service. To make this process simple we are hosting package manager repositories for .deb and .rpm packages. See the installing guide for more information.

Development

If you're inclined to get involved in code and non-code development, please check out the contributing guide. We're striving for this project to be developed in the open and by people for people. Specific areas in which we are looking for help include:

  • Rust
  • Flutter/Dart
  • Python
  • Gitlab DevOps and CI/CD
  • Documentation
  • Security reviews
  • Linux packaging