Bootstrap V1

BOOTSTRAP-SETUP.md

@@ -1,4 +1,11 @@
-# Starting a Generic/Public Veilid Bootstrap Server
+# Creating a Veilid Bootstrap Server
+
+There are two versions of the Veilid bootstrap:
+
+ * Version 0: Unsigned bootstrap - Any node can bootstrap with your server, but no guarantees are provided to those nodes about which network was connected, as DNS is subject to MITM attacks
+ * Version 1: Signed bootstrap - Any node can bootstrap with your server, and if they have your bootstrap signing public key in their trusted keys config, they will get a verified bootstrap to a specific network
+
+These instructions will cover both versions, however Version 1 is preferable, as Version 0 will eventually be deprecated.
 
 ## Instance Recommended Setup
 
@@ -8,6 +15,8 @@ Storage: 25GB<br>
 IP: Static v4 & v6<br>
 Firewall: 5150/TCP/UDP inbound allow all<br>
 
+You will need to ensure your bootstrap server has PTR records for its hostname, as the TXT records generated for the bootstrap will rely on the hostname existing and being consistent for all of its IP addresses.
+
 ## Install Veilid
 
 Follow instructions in [INSTALL.md](./INSTALL.md)
@@ -20,11 +29,36 @@ Follow instructions in [INSTALL.md](./INSTALL.md)
 sudo systemctl stop veilid-server.service
 ```
 
+### Create a bootstrap signing key
+
+You need a 'bootstrap signing key' to sign your bootstrap server records. A single signing key can be used to sign multiple bootstrap server records. If you don't have one yet, with `veilid-server` not already running:
+
+```shell
+sudo -u veilid veilid-server --generate-key-pair VLD0
+```
+which outputs a key in this form (example: `VLD0:NAPctwUP5NNynWdkX8rcUz_yk44v-cHuDM9ZzvsDXnQ:-ncghvgw2NFQK2RH2vCfvCJj3M3gTVOD-UM08-7n6kQ`):
+```
+VLD0:PUBLIC_KEY:SECRET_KEY
+```
+
+Copy down the generated keypair and store it in a secure location, preferably offline.
+Remove the part after the second colon (the SECRET_KEY), and this is your 'Bootstrap Signing Public Key' (should look like: `VLD0:NAPctwUP5NNynWdkX8rcUz_yk44v-cHuDM9ZzvsDXnQ`)
+
 ### Setup the config
 
-In `/etc/veilid-server/veilid-server.conf`, ensure `bootstrap: ['bootstrap.<your.domain>']` in the `routing_table:` section.
+In `/etc/veilid-server/veilid-server.conf` ensure these keys are in the in the `routing_table:` section
 
-If you came here from the [dev network setup](./dev-setup/dev-network-setup.md) guide, this is when you set the network key.
+- `bootstrap: ['bootstrap.<your.domain>']`
+
+- V0: Use an empty bootstrap key list to enable unverified bootstrap
+  - `bootstrap_keys: []`
+- V1: Add your bootstrap signing public key to this list.
+  - If your signing key is the only one:
+    - `bootstrap_keys: ['VLD0:<your_bootstrap_signing_public_key>']`
+  - You may also want to include any other signing keys for bootstraps you trust. If this is a bootstrap for the main Veilid network, include Veilid Foundation's signing keys here as well
+    - `bootstrap_keys: ['VLD0:<your_bootstrap_signing_public_key>', 'VLD0:Vj0lKDdUQXmQ5Ol1SZdlvXkBHUccBcQvGLN9vbLSI7k', 'VLD0:QeQJorqbXtC7v3OlynCZ_W3m76wGNeB5NTF81ypqHAo','VLD0:QNdcl-0OiFfYVj9331XVR6IqZ49NG-E18d5P7lwi4TA']`
+
+(If you came here from the [dev network setup](./dev-setup/dev-network-setup.md) guide, this is when you set the network key as well in the `network_key_password` field of the `network:` section)
 
 **Switch to veilid user**
 
@@ -53,7 +87,12 @@ veilid-server --set-node-id [PUBLIC_KEY] --delete-table-store
 Copy the output to secure storage. This information will be use to setup DNS records.
 
 ```shell
-veilid-server --dump-txt-record
+veilid-server --dump-txt-record <FULL BOOTSTRAP SIGNING KEY PAIR>
+```
+
+(will look like this, but with your own key:)
+```shell
+veilid-server --dump-txt-record VLD0:NAPctwUP5NNynWdkX8rcUz_yk44v-cHuDM9ZzvsDXnQ:-ncghvgw2NFQK2RH2vCfvCJj3M3gTVOD-UM08-7n6kQ
 ```
 
 ### Start the Veilid service
@@ -78,12 +117,25 @@ Create the following DNS Records for your domain:
 
 (This example assumes two bootstrap servers are being created)
 
-| Record    | Value                       | Record Type |
-|-----------|-----------------------------|-------------|
-|bootstrap  | 1,2                         | TXT         |
-|1.bootstrap| IPv4                        | A           |
-|1.bootstrap| IPv6                        | AAAA        |
-|1.bootstrap| output of --dump-txt-record | TXT         |
-|2.bootstrap| IPv4                        | A           |
-|2.bootstrap| IPv6                        | AAAA        |
-|2.bootstrap| output of --dump-txt-record | TXT         |
+V1:
+| Record       | Value                        | Record Type |
+| ------------ | ---------------------------- | ----------- |
+| bootstrap-v1 | IPv4 of bootstrap 1          | A           |
+| bootstrap-v1 | IPv4 of bootstrap 2          | A           |
+| bootstrap-v1 | IPv6 of bootstrap 1          | AAAA        |
+| bootstrap-v1 | IPv6 of bootstrap 2          | AAAA        |
+| bootstrap-v1 | TXTRecord v0 for bootstrap 1 | TXT         |
+| bootstrap-v1 | TXTRecord v1 for bootstrap 1 | TXT         |
+| bootstrap-v1 | TXTRecord v0 for bootstrap 2 | TXT         |
+| bootstrap-v1 | TXTRecord v1 for bootstrap 2 | TXT         |
+
+V0:
+| Record      | Value                        | Record Type |
+| ----------- | ---------------------------- | ----------- |
+| bootstrap   | 1,2                          | TXT         |
+| 1.bootstrap | IPv4                         | A           |
+| 1.bootstrap | IPv6                         | AAAA        |
+| 1.bootstrap | TXTRecord v0 for bootstrap 1 | TXT         |
+| 2.bootstrap | IPv4                         | A           |
+| 2.bootstrap | IPv6                         | AAAA        |
+| 2.bootstrap | TXTRecord v0 for bootstrap 2 | TXT         |

CHANGELOG.md

@@ -10,13 +10,14 @@
     - `VeilidConfigInner` -> `VeilidConfig`
 
 - veilid-core:
+  - **Security** Signed bootstrap v1 added which closes #293: https://gitlab.com/veilid/veilid/-/issues/293
   - Allow shutdown even if tables are closed
   - New, more robust, watchvalue implementation
   - Consensus is now counted from the nodes closest to the key, excluding attempts that have failed, but including new nodes that show up, requiring N out of the M closest nodes to have      succeeded and all have been attempted.
   - Watching a node now also triggers an background inspection+valueget to detect if values have changed online
   - Fanout queue disqualifaction for distance-based rejections reimplemented
   - Local rehydration implemented. DHT record subkey data that does not have sufficient consensus online is re-pushed to keep it alive when records are opened.
-  - Direct bootstrap now filters out Relayed nodes correctly
+  - Direct bootstrap v0 now filters out Relayed nodes correctly
   - Closed issue #400: https://gitlab.com/veilid/veilid/-/issues/400
   - Closed issue #377: https://gitlab.com/veilid/veilid/-/issues/377
   - Add the `veilid_features()` API, which lists the compile-time features that were enabled when `veilid-core` was built (available in language bindings as well). ([!401](https://gitlab.com/veilid/veilid/-/issues/400))

dev-setup/dev-network-setup.md

@@ -6,7 +6,7 @@ There will be times when a contibutor wishes to dynamically test their work on l
 
 This document outlines the process of using the steps found in [INSTALL.md](../INSTALL.md) and [BOOTSTRAP-SETUP.md](../BOOTSTRAP-SETUP.md) with some modifications which results in a reasonably isolated and independent network of Veilid development nodes which do not communicate with nodes on the actual Veilid network.
 
-The minimum topology of a dev network is 1 bootstrap server and 4 nodes, all with public IP addresses with port 5150/TCP open. This allows enabling public address detection and private routing. The minimum specifications are 1 vCPU, 1GB RAM, and 25 GB storage.
+The minimum topology of a dev network is 1 bootstrap server and 4 nodes, all with public IP addresses with port 5150/TCP open. It is preferable to open port 5150/UDP as well. This allows enabling public address detection and private routing. The minimum specifications are 1 vCPU, 1GB RAM, and 25 GB storage.
 
 ## Quick Start
 

doc/config/sample.config

@@ -55,7 +55,8 @@ core:
         routing_table:
             node_id: null
             node_id_secret: null
-            bootstrap: ['bootstrap.veilid.net']
+            bootstrap: ['bootstrap-v1.veilid.net']
+            bootstrap_keys: ['VLD0:Vj0lKDdUQXmQ5Ol1SZdlvXkBHUccBcQvGLN9vbLSI7k','VLD0:QeQJorqbXtC7v3OlynCZ_W3m76wGNeB5NTF81ypqHAo','VLD0:QNdcl-0OiFfYVj9331XVR6IqZ49NG-E18d5P7lwi4TA']
             limit_over_attached: 64
             limit_fully_attached: 32
             limit_attached_strong: 16

doc/config/veilid-bootstrap-config

@@ -17,17 +17,18 @@ logging:
     enabled: false
 core:
   capabilities:
-    disable: ['TUNL','SGNL','RLAY','DIAL','DHTV','DHTW','APPM','ROUT']
+    disable: ["TUNL", "SGNL", "RLAY", "DIAL", "DHTV", "DHTW", "APPM", "ROUT"]
   network:
     upnp: false
     dht:
       min_peer_count: 2
     detect_address_changes: false
     routing_table:
-      bootstrap: ['bootstrap.<your.domain>']
+      bootstrap: ["bootstrap.<your.domain>"]
+      bootstrap_keys: ["VLD0:<your bootstrap signing public key>"]
   protected_store:
-    insecure_fallback_directory: '/var/db/veilid-server/protected_store'
+    insecure_fallback_directory: "/var/db/veilid-server/protected_store"
   table_store:
-    directory: '/var/db/veilid-server/table_store'
+    directory: "/var/db/veilid-server/table_store"
   block_store:
-    directory: '/var/db/veilid-server/block_store'
+    directory: "/var/db/veilid-server/block_store"

doc/config/veilid-dev-bootstrap-config

@@ -22,18 +22,19 @@ logging:
     enabled: false
 core:
   capabilities:
-    disable: ['TUNL','SGNL','RLAY','DIAL','DHTV','DHTW','APPM']
+    disable: ["TUNL", "SGNL", "RLAY", "DIAL", "DHTV", "DHTW", "APPM"]
   network:
     upnp: false
     dht:
       min_peer_count: 2
     detect_address_changes: false
     routing_table:
-      bootstrap: ['bootstrap.<your.domain>']
-    network_key_password: '<your-chosen-passkey>'
+      bootstrap: ["bootstrap.<your.domain>"]
+      bootstrap_keys: ["VLD0:<your bootstrap signing public key>"]
+    network_key_password: "<your-chosen-passkey>"
   protected_store:
-    insecure_fallback_directory: '/var/db/veilid-server/protected_store'
+    insecure_fallback_directory: "/var/db/veilid-server/protected_store"
   table_store:
-    directory: '/var/db/veilid-server/table_store'
+    directory: "/var/db/veilid-server/table_store"
   block_store:
-    directory: '/var/db/veilid-server/block_store'
+    directory: "/var/db/veilid-server/block_store"

doc/config/veilid-dev-node-config

@@ -3,10 +3,10 @@
 #
 # Private Development Node Configuration
 #
-# This config is templated to setup a Velid node with a 
+# This config is templated to setup a Velid node with a
 # network_key_password. Set the network key to whatever you
-# set within your private bootstrap server's config. Treat it 
-# like a password. 
+# set within your private bootstrap server's config. Treat it
+# like a password.
 # -----------------------------------------------------------
 
 ---
@@ -21,18 +21,19 @@ logging:
     enabled: false
 core:
   capabilities:
-    disable: ['APPM']
+    disable: ["APPM"]
   network:
     upnp: false
     dht:
       min_peer_count: 10
     detect_address_changes: false
     routing_table:
-      bootstrap: ['bootstrap.<your.domain>']
-    network_key_password: '<your-chosen-passkey>'
+      bootstrap: ["bootstrap.<your.domain>"]
+      bootstrap_keys: ["VLD0:<your bootstrap signing public key>"]
+    network_key_password: "<your-chosen-passkey>"
   protected_store:
-    insecure_fallback_directory: '/var/db/veilid-server/protected_store'
+    insecure_fallback_directory: "/var/db/veilid-server/protected_store"
   table_store:
-    directory: '/var/db/veilid-server/table_store'
+    directory: "/var/db/veilid-server/table_store"
   block_store:
-    directory: '/var/db/veilid-server/block_store'
+    directory: "/var/db/veilid-server/block_store"

doc/config/veilid-server-config.md

@@ -0,0 +1,321 @@
+---
+title: Veilid Server Configuration
+keywords:
+- config
+- veilid-server
+status: Draft
+---
+# Veilid Server Configuration
+
+## Configuration File
+
+`veilid-server` may be run using configuration from both command-line arguments
+and the `veilid-server.conf` file.
+
+## Global Directives
+
+| Directive                    | Description                           |
+| ---------------------------- | ------------------------------------- |
+| [daemon](#daemon)            | Run `veilid-server` in the background |
+| [client\_api](#client_api)   |                                       |
+| [auto\_attach](#auto_attach) |                                       |
+| [logging](#logging)          |                                       |
+| [testing](#testing)          |                                       |
+| [core](#core)                |                                       |
+
+
+### daemon
+
+```yaml
+daemon:
+    enabled: false
+```
+
+### client_api
+
+```yaml
+client_api:
+    enabled: true
+    listen_address: 'localhost:5959'
+```
+
+| Parameter                                    | Description |
+| -------------------------------------------- | ----------- |
+| [enabled](#client_apienabled)                |             |
+| [listen\_address](#client_apilisten_address) |             |
+
+#### client\_api:enabled
+
+**TODO**
+
+#### client\_api:listen\_address
+
+**TODO**
+
+### auto\_attach
+
+```yaml
+auto_attach: true
+```
+
+### logging
+
+```yaml
+logging:
+    system:
+        enabled: false
+        level: 'info'
+    terminal:
+        enabled: true
+        level: 'info'
+    file:
+        enabled: false
+        path: ''
+        append: true
+        level: 'info'
+    api:
+        enabled: true
+        level: 'info'
+    otlp:
+        enabled: false
+        level: 'trace'
+        grpc_endpoint: 'localhost:4317'
+```
+
+| Parameter                    | Description |
+| ---------------------------- | ----------- |
+| [system](#loggingsystem)     |             |
+| [terminal](#loggingterminal) |             |
+| [file](#loggingfile)         |             |
+| [api](#loggingapi)           |             |
+| [otlp](#loggingotlp)         |             |
+
+#### logging:system
+
+```yaml
+system:
+    enabled: false
+    level: 'info'
+```
+
+#### logging:terminal
+
+```yaml
+terminal:
+    enabled: true
+    level: 'info'
+```
+
+#### logging:file
+
+```yaml
+file:
+    enabled: false
+    path: ''
+    append: true
+    level: 'info'
+```
+
+#### logging:api
+
+```yaml
+api:
+    enabled: true
+    level: 'info'
+```
+
+#### logging:otlp
+
+```yaml
+otlp:
+    enabled: false
+    level: 'trace'
+    grpc_endpoint: 'localhost:4317'
+```
+
+### testing
+
+```yaml
+testing:
+    subnode_index: 0
+    subnode_count: 1
+```
+
+### core
+
+| Parameter                                | Description |
+| ---------------------------------------- | ----------- |
+| [protected\_store](#coreprotected_store) |             |
+| [table\_store](#coretable_store)         |             |
+| [block\_store](#block_store)             |             |
+| [network](#corenetwork)                  |             |
+
+#### core:protected\_store
+
+```yaml
+protected_store:
+    allow_insecure_fallback: true
+    always_use_insecure_storage: true
+    directory: '%DIRECTORY%'
+    delete: false
+```
+
+#### core:table\_store
+
+```yaml
+table_store:
+    directory: '%TABLE_STORE_DIRECTORY%'
+    delete: false
+```
+
+#### core:block\_store
+
+```yaml
+block_store:
+    directory: '%BLOCK_STORE_DIRECTORY%'
+    delete: false
+```
+
+#### core:network
+
+```yaml
+network:
+    connection_initial_timeout_ms: 2000
+    connection_inactivity_timeout_ms: 60000
+    max_connections_per_ip4: 32
+    max_connections_per_ip6_prefix: 32
+    max_connections_per_ip6_prefix_size: 56
+    max_connection_frequency_per_min: 128
+    client_allowlist_timeout_ms: 300000
+    reverse_connection_receipt_time_ms: 5000
+    hole_punch_receipt_time_ms: 5000
+    network_key_password: null
+    disable_capabilites: []
+    node_id: null
+    node_id_secret: null
+    upnp: true
+    detect_address_changes: true
+    enable_local_peer_scope: false
+    restricted_nat_retries: 0
+```
+
+| Parameter                                   | Description |
+| ------------------------------------------- | ----------- |
+| [routing\_table](#corenetworkrouting_table) |             |
+| [rpc](#corenetworkrpc)                      |             |
+| [dht](#corenetworkdht)                      |             |
+| [tls](#corenetworktls)                      |             |
+| [application](#corenetworkapplication)      |             |
+| [protocol](#corenetworkprotocol)            |             |
+
+#### core:network:routing\_table
+
+```yaml
+routing_table:
+    node_id: null
+    node_id_secret: null
+    bootstrap: ['bootstrap-v1.veilid.net']
+    bootstrap_keys: ['VLD0:Vj0lKDdUQXmQ5Ol1SZdlvXkBHUccBcQvGLN9vbLSI7k','VLD0:QeQJorqbXtC7v3OlynCZ_W3m76wGNeB5NTF81ypqHAo','VLD0:QNdcl-0OiFfYVj9331XVR6IqZ49NG-E18d5P7lwi4TA']
+    limit_over_attached: 64
+    limit_fully_attached: 32
+    limit_attached_strong: 16
+    limit_attached_good: 8
+    limit_attached_weak: 4
+```
+
+#### core:network:rpc
+
+```yaml
+rpc:
+    concurrency: 0
+    queue_size: 1024
+    max_timestamp_behind_ms: 10000
+    max_timestamp_ahead_ms: 10000
+    timeout_ms: 5000
+    max_route_hop_count: 4
+    default_route_hop_count: 1
+```
+
+#### core:network:dht
+
+```yaml
+dht:
+    max_find_node_count: 20
+    resolve_node_timeout_ms: 10000
+    resolve_node_count: 1
+    resolve_node_fanout: 4
+    get_value_timeout_ms: 10000
+    get_value_count: 3
+    get_value_fanout: 4
+    set_value_timeout_ms: 10000
+    set_value_count: 5
+    set_value_fanout: 4
+    min_peer_count: 20
+    min_peer_refresh_time_ms: 60000
+    validate_dial_info_receipt_time_ms: 2000
+    local_subkey_cache_size: 128
+    local_max_subkey_cache_memory_mb: 256
+    remote_subkey_cache_size: 1024
+    remote_max_records: 65536
+    remote_max_subkey_cache_memory_mb: %REMOTE_MAX_SUBKEY_CACHE_MEMORY_MB%
+    remote_max_storage_space_mb: 0
+    public_watch_limit: 32
+    member_watch_limit: 8
+    max_watch_expiration_ms: 600000
+```
+
+#### core:network:tls
+
+```yaml
+tls:
+    certificate_path: '%CERTIFICATE_PATH%'
+    private_key_path: '%PRIVATE_KEY_PATH%'
+    connection_initial_timeout_ms: 2000
+```
+
+#### core:network:application
+
+```yaml
+application:
+    https:
+        enabled: false
+        listen_address: ':5150'
+        path: 'app'
+        # url: 'https://localhost:5150'
+    http:
+        enabled: false
+        listen_address: ':5150'
+        path: 'app'
+        # url: 'http://localhost:5150'
+```
+
+#### core:network:protocol
+
+```yaml
+protocol:
+    udp:
+        enabled: true
+        socket_pool_size: 0
+        listen_address: ':5150'
+        # public_address: ''
+    tcp:
+        connect: true
+        listen: true
+        max_connections: 32
+        listen_address: ':5150'
+        #'public_address: ''
+    ws:
+        connect: true
+        listen: true
+        max_connections: 16
+        listen_address: ':5150'
+        path: 'ws'
+        # url: 'ws://localhost:5150/ws'
+    wss:
+        connect: true
+        listen: false
+        max_connections: 16
+        listen_address: ':5150'
+        path: 'ws'
+        # url: ''
+```

veilid-core/src/attachment_manager.rs

@@ -23,7 +23,7 @@ impl Default for AttachmentManagerStartupContext {
 #[derive(Debug)]
 struct AttachmentManagerInner {
     last_attachment_state: AttachmentState,
-    last_routing_table_health: Option<RoutingTableHealth>,
+    last_routing_table_health: Option<Arc<RoutingTableHealth>>,
     maintain_peers: bool,
     started_ts: Timestamp,
     attach_ts: Option<Timestamp>,
@@ -123,14 +123,14 @@ impl AttachmentManager {
             // Check if the routing table health is different
             if let Some(last_routing_table_health) = &inner.last_routing_table_health {
                 // If things are the same, just return
-                if last_routing_table_health == &health {
+                if last_routing_table_health.as_ref() == &health {
                     return;
                 }
             }
 
             // Swap in new health numbers
             let opt_previous_health = inner.last_routing_table_health.take();
-            inner.last_routing_table_health = Some(health.clone());
+            inner.last_routing_table_health = Some(Arc::new(health.clone()));
 
             // Calculate new attachment state
             let config = self.config();
@@ -414,10 +414,6 @@ impl AttachmentManager {
         }
     }
 
-    // pub fn get_attachment_state(&self) -> AttachmentState {
-    //     self.inner.lock().last_attachment_state
-    // }
-
     fn get_veilid_state_inner(inner: &AttachmentManagerInner) -> Box<VeilidStateAttachment> {
         let now = Timestamp::now();
         let uptime = now - inner.started_ts;
@@ -444,4 +440,14 @@ impl AttachmentManager {
         let inner = self.inner.lock();
         Self::get_veilid_state_inner(&inner)
     }
+
+    #[expect(dead_code)]
+    pub fn get_attachment_state(&self) -> AttachmentState {
+        self.inner.lock().last_attachment_state
+    }
+
+    #[expect(dead_code)]
+    pub fn get_last_routing_table_health(&self) -> Option<Arc<RoutingTableHealth>> {
+        self.inner.lock().last_routing_table_health.clone()
+    }
 }

veilid-core/src/intf/native/system.rs

@@ -104,13 +104,14 @@ pub async fn txt_lookup<S: AsRef<str>>(host: S) -> EyreResult<Vec<String>> {
                     if (*p_record).wType == DNS_TYPE_TEXT.0 {
                         let count:usize = (*p_record).Data.TXT.dwStringCount.try_into().unwrap();
                         let string_array: *const PSTR = &(*p_record).Data.TXT.pStringArray[0];
+                        let mut record_out = Vec::<u8>::new();
                         for n in 0..count {
                             let pstr: PSTR = *(string_array.add(n));
                             let c_str: &CStr = CStr::from_ptr(pstr.0 as *const i8);
-                            if let Ok(str_slice) = c_str.to_str() {
-                                let str_buf: String = str_slice.to_owned();
-                                out.push(str_buf);
-                            }
+                            record_out.extend_from_slice(c_str.to_bytes());
+                        }
+                        if let Ok(s) = String::from_utf8(record_out) {
+                            out.push(s);
                         }
                     }
                     p_record = (*p_record).pNext;
@@ -148,8 +149,12 @@ pub async fn txt_lookup<S: AsRef<str>>(host: S) -> EyreResult<Vec<String>> {
                 })).await?;
             let mut out = Vec::new();
             for x in txt_result.iter() {
-                for s in x.txt_data() {
-                    out.push(String::from_utf8(s.to_vec()).wrap_err("utf8 conversion error")?);
+                let mut record_out = Vec::<u8>::new();
+                for txtd in x.txt_data() {
+                    record_out.extend_from_slice(txtd);
+                }
+                if let Ok(s) = String::from_utf8(record_out) {
+                    out.push(s);
                 }
             }
             Ok(out)

veilid-core/src/network_manager/bootstrap/bootstrap_record.rs

@@ -0,0 +1,492 @@
+use super::*;
+
+impl_veilid_log_facility!("net");
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct BootstrapRecord {
+    node_ids: TypedKeyGroup,
+    envelope_support: Vec<u8>,
+    dial_info_details: Vec<DialInfoDetail>,
+    timestamp_secs: Option<u64>,
+    extra: Vec<String>,
+}
+
+impl BootstrapRecord {
+    pub fn new(
+        node_ids: TypedKeyGroup,
+        mut envelope_support: Vec<u8>,
+        mut dial_info_details: Vec<DialInfoDetail>,
+        timestamp_secs: Option<u64>,
+        extra: Vec<String>,
+    ) -> Self {
+        envelope_support.sort();
+        dial_info_details.sort();
+
+        Self {
+            node_ids,
+            envelope_support,
+            dial_info_details,
+            timestamp_secs,
+            extra,
+        }
+    }
+
+    pub fn node_ids(&self) -> &TypedKeyGroup {
+        &self.node_ids
+    }
+    pub fn envelope_support(&self) -> &[u8] {
+        &self.envelope_support
+    }
+    pub fn dial_info_details(&self) -> &[DialInfoDetail] {
+        &self.dial_info_details
+    }
+    pub fn timestamp_secs(&self) -> Option<u64> {
+        self.timestamp_secs
+    }
+    #[expect(dead_code)]
+    pub fn extra(&self) -> &[String] {
+        &self.extra
+    }
+
+    pub fn merge(&mut self, other: BootstrapRecord) {
+        self.node_ids.add_all(&other.node_ids);
+        for x in other.envelope_support {
+            if !self.envelope_support.contains(&x) {
+                self.envelope_support.push(x);
+                self.envelope_support.sort();
+            }
+        }
+        for did in other.dial_info_details {
+            if !self.dial_info_details.contains(&did) {
+                self.dial_info_details.push(did);
+            }
+        }
+        self.dial_info_details.sort();
+        if let Some(ts) = self.timestamp_secs.as_mut() {
+            if let Some(other_ts) = other.timestamp_secs {
+                // Use earliest timestamp if merging
+                ts.min_assign(other_ts);
+            } else {
+                // Do nothing
+            }
+        } else {
+            self.timestamp_secs = other.timestamp_secs;
+        }
+        self.extra.extend_from_slice(&other.extra);
+    }
+
+    async fn to_vcommon_string(
+        &self,
+        dial_info_converter: &dyn DialInfoConverter,
+    ) -> EyreResult<String> {
+        let valid_envelope_versions = self
+            .envelope_support()
+            .iter()
+            .map(|x| x.to_string())
+            .collect::<Vec<_>>()
+            .join(",");
+
+        let node_ids = self
+            .node_ids
+            .iter()
+            .map(|x| x.to_string())
+            .collect::<Vec<_>>()
+            .join(",");
+
+        let mut short_urls = Vec::new();
+        let mut some_hostname = Option::<String>::None;
+        for did in self.dial_info_details() {
+            let ShortDialInfo {
+                short_url,
+                hostname,
+            } = dial_info_converter.to_short(did.dial_info.clone()).await;
+            if let Some(h) = &some_hostname {
+                if h != &hostname {
+                    bail!(
+                        "Inconsistent hostnames for dial info: {} vs {}",
+                        some_hostname.unwrap(),
+                        hostname
+                    );
+                }
+            } else {
+                some_hostname = Some(hostname);
+            }
+
+            short_urls.push(short_url);
+        }
+        if some_hostname.is_none() || short_urls.is_empty() {
+            bail!("No dial info for bootstrap host");
+        }
+        short_urls.sort();
+        short_urls.dedup();
+
+        let vcommon = format!(
+            "|{}|{}|{}|{}",
+            valid_envelope_versions,
+            node_ids,
+            some_hostname.as_ref().unwrap(),
+            short_urls.join(",")
+        );
+
+        Ok(vcommon)
+    }
+
+    pub async fn to_v0_string(
+        &self,
+        dial_info_converter: &dyn DialInfoConverter,
+    ) -> EyreResult<String> {
+        let vcommon = self.to_vcommon_string(dial_info_converter).await?;
+        Ok(format!("{}{}", BOOTSTRAP_TXT_VERSION_0, vcommon))
+    }
+
+    pub async fn to_v1_string(
+        &self,
+        network_manager: &NetworkManager,
+        dial_info_converter: &dyn DialInfoConverter,
+        signing_key_pair: TypedKeyPair,
+    ) -> EyreResult<String> {
+        let vcommon = self.to_vcommon_string(dial_info_converter).await?;
+        let ts = if let Some(ts) = self.timestamp_secs() {
+            ts
+        } else {
+            bail!("timestamp required for bootstrap v1 format");
+        };
+        let mut v1 = format!("{}{}|{}|", BOOTSTRAP_TXT_VERSION_1, vcommon, ts);
+
+        let crypto = network_manager.crypto();
+
+        let sig = match crypto.generate_signatures(v1.as_bytes(), &[signing_key_pair], |kp, sig| {
+            TypedSignature::new(kp.kind, sig).to_string()
+        }) {
+            Ok(v) => {
+                let Some(sig) = v.first().cloned() else {
+                    bail!("No signature generated");
+                };
+                sig
+            }
+            Err(e) => {
+                bail!("Failed to generate signature: {}", e);
+            }
+        };
+
+        v1 += &sig;
+        Ok(v1)
+    }
+
+    pub fn new_from_v0_str(
+        network_manager: &NetworkManager,
+        dial_info_converter: &dyn DialInfoConverter,
+        record_str: &str,
+    ) -> EyreResult<Option<BootstrapRecord>> {
+        // All formats split on '|' character
+        let fields: Vec<String> = record_str
+            .trim()
+            .split('|')
+            .map(|x| x.trim().to_owned())
+            .collect();
+
+        // Bootstrap TXT record version
+        let txt_version: u8 = match fields[0].parse::<u8>() {
+            Ok(v) => v,
+            Err(e) => {
+                bail!(
+                    "invalid txt_version specified in bootstrap node txt record: {}",
+                    e
+                );
+            }
+        };
+        let bootstrap_record = match txt_version {
+            BOOTSTRAP_TXT_VERSION_0 => {
+                match Self::process_bootstrap_fields_v0(
+                    network_manager,
+                    dial_info_converter,
+                    &fields,
+                ) {
+                    Err(e) => {
+                        bail!(
+                            "couldn't process v0 bootstrap records from {:?}: {}",
+                            fields,
+                            e
+                        );
+                    }
+                    Ok(Some(v)) => v,
+                    Ok(None) => {
+                        // skipping
+                        return Ok(None);
+                    }
+                }
+            }
+            _ => return Ok(None),
+        };
+
+        Ok(Some(bootstrap_record))
+    }
+
+    pub fn new_from_v1_str(
+        network_manager: &NetworkManager,
+        dial_info_converter: &dyn DialInfoConverter,
+        record_str: &str,
+        signing_keys: &[TypedKey],
+    ) -> EyreResult<Option<BootstrapRecord>> {
+        // All formats split on '|' character
+        let fields: Vec<String> = record_str
+            .trim()
+            .split('|')
+            .map(|x| x.trim().to_owned())
+            .collect();
+
+        // Bootstrap TXT record version
+        let txt_version: u8 = match fields[0].parse::<u8>() {
+            Ok(v) => v,
+            Err(e) => {
+                bail!(
+                    "invalid txt_version specified in bootstrap node txt record: {}",
+                    e
+                );
+            }
+        };
+        let bootstrap_record = match txt_version {
+            BOOTSTRAP_TXT_VERSION_1 => {
+                match Self::process_bootstrap_fields_v1(
+                    network_manager,
+                    dial_info_converter,
+                    record_str,
+                    &fields,
+                    signing_keys,
+                ) {
+                    Err(e) => {
+                        bail!(
+                            "couldn't process v1 bootstrap records from {:?}: {}",
+                            fields,
+                            e
+                        );
+                    }
+                    Ok(Some(v)) => v,
+                    Ok(None) => {
+                        // skipping
+                        return Ok(None);
+                    }
+                }
+            }
+            _ => return Ok(None),
+        };
+
+        Ok(Some(bootstrap_record))
+    }
+
+    /// Process bootstrap version 0
+    ///
+    /// Bootstrap TXT Record Format Version 0:
+    /// txt_version|envelope_support|node_ids|hostname|dialinfoshort*
+    ///
+    /// Split bootstrap node record by '|' and then lists by ','. Example:
+    /// 0|0|VLD0:7lxDEabK_qgjbe38RtBa3IZLrud84P6NhGP-pRTZzdQ|bootstrap-1.dev.veilid.net|T5150,U5150,W5150/ws
+    fn process_bootstrap_fields_v0(
+        network_manager: &NetworkManager,
+        dial_info_converter: &dyn DialInfoConverter,
+        fields: &[String],
+    ) -> EyreResult<Option<BootstrapRecord>> {
+        if fields.len() != 5 {
+            bail!("invalid number of fields in bootstrap v0 txt record");
+        }
+
+        // Envelope support
+        let mut envelope_support = Vec::new();
+        for ess in fields[1].split(',') {
+            let ess = ess.trim();
+            let es = match ess.parse::<u8>() {
+                Ok(v) => v,
+                Err(e) => {
+                    bail!(
+                        "invalid envelope version specified in bootstrap node txt record: {}",
+                        e
+                    );
+                }
+            };
+            envelope_support.push(es);
+        }
+        envelope_support.sort();
+        envelope_support.dedup();
+
+        // Node Id
+        let mut node_ids = TypedKeyGroup::new();
+        for node_id_str in fields[2].split(',') {
+            let node_id_str = node_id_str.trim();
+            let node_id = match TypedKey::from_str(node_id_str) {
+                Ok(v) => v,
+                Err(e) => {
+                    bail!(
+                        "Invalid node id in bootstrap node record {}: {}",
+                        node_id_str,
+                        e
+                    );
+                }
+            };
+            node_ids.add(node_id);
+        }
+
+        // Hostname
+        let hostname_str = fields[3].trim();
+
+        // Resolve each record and store in node dial infos list
+        let mut dial_info_details = Vec::new();
+        for rec in fields[4].split(',') {
+            let rec = rec.trim();
+            let short_dial_info = ShortDialInfo {
+                short_url: rec.to_string(),
+                hostname: hostname_str.to_string(),
+            };
+            let dial_infos = match dial_info_converter.try_vec_from_short(&short_dial_info) {
+                Ok(dis) => dis,
+                Err(e) => {
+                    veilid_log!(network_manager warn "Couldn't resolve bootstrap node dial info {}: {}", rec, e);
+                    continue;
+                }
+            };
+
+            for di in dial_infos {
+                dial_info_details.push(DialInfoDetail {
+                    dial_info: di,
+                    class: DialInfoClass::Direct,
+                });
+            }
+        }
+
+        Ok(Some(BootstrapRecord::new(
+            node_ids,
+            envelope_support,
+            dial_info_details,
+            None,
+            vec![],
+        )))
+    }
+
+    /// Process bootstrap version 1
+    ///
+    /// Bootstrap TXT Record Format Version 1:
+    /// txt_version|envelope_support|node_ids|hostname|dialinfoshort*|timestamp|extra..|....| typedsignature
+    ///
+    /// Split bootstrap node record by '|' and then lists by ','. Example:
+    /// 1|0|VLD0:7lxDEabK_qgjbe38RtBa3IZLrud84P6NhGP-pRTZzdQ|bootstrap-1.dev.veilid.net|T5150,U5150,W5150/ws|1746308366
+    /// timestamp is a uint64 number of seconds since epoch (unix time64)
+    /// extra is any extra data to be covered by the signature, any number of extra '|' fields
+    /// the signature is over all of the byte data in the string that precedes the signature itself, including all delimeters and/or whitespace
+    fn process_bootstrap_fields_v1(
+        network_manager: &NetworkManager,
+        dial_info_converter: &dyn DialInfoConverter,
+        record_str: &str,
+        fields: &[String],
+        signing_keys: &[TypedKey],
+    ) -> EyreResult<Option<BootstrapRecord>> {
+        if fields.len() < 7 {
+            bail!("invalid number of fields in bootstrap v1 txt record");
+        }
+
+        // Get signature from last record
+        let sigstring = fields.last().unwrap();
+        let sig = TypedSignature::from_str(sigstring)
+            .wrap_err("invalid signature for bootstrap v1 record")?;
+
+        // Get slice that was signed
+        let signed_str = &record_str[0..record_str.len() - sigstring.len()];
+
+        // Validate signature against any signing keys if we have them
+        if !signing_keys.is_empty() {
+            let mut validated = false;
+            for key in signing_keys.iter().copied() {
+                if let Some(valid_keys) = network_manager.crypto().verify_signatures(
+                    &[key],
+                    signed_str.as_bytes(),
+                    &[sig],
+                )? {
+                    if valid_keys.contains(&key) {
+                        validated = true;
+                        break;
+                    }
+                }
+            }
+            if !validated {
+                bail!(
+                    "bootstrap record did not have valid signature: {}",
+                    record_str
+                );
+            }
+        }
+
+        // Envelope support
+        let mut envelope_support = Vec::new();
+        for ess in fields[1].split(',') {
+            let ess = ess.trim();
+            let es = match ess.parse::<u8>() {
+                Ok(v) => v,
+                Err(e) => {
+                    bail!(
+                        "invalid envelope version specified in bootstrap node txt record: {}",
+                        e
+                    );
+                }
+            };
+            envelope_support.push(es);
+        }
+        envelope_support.sort();
+        envelope_support.dedup();
+
+        // Node Id
+        let mut node_ids = TypedKeyGroup::new();
+        for node_id_str in fields[2].split(',') {
+            let node_id_str = node_id_str.trim();
+            let node_id = match TypedKey::from_str(node_id_str) {
+                Ok(v) => v,
+                Err(e) => {
+                    bail!(
+                        "Invalid node id in bootstrap node record {}: {}",
+                        node_id_str,
+                        e
+                    );
+                }
+            };
+            node_ids.add(node_id);
+        }
+
+        // Hostname
+        let hostname_str = fields[3].trim();
+
+        // DialInfos
+        let mut dial_info_details = Vec::new();
+        for rec in fields[4].split(',') {
+            let rec = rec.trim();
+            let short_dial_info = ShortDialInfo {
+                short_url: rec.to_string(),
+                hostname: hostname_str.to_string(),
+            };
+            let dial_infos = match dial_info_converter.try_vec_from_short(&short_dial_info) {
+                Ok(dis) => dis,
+                Err(e) => {
+                    veilid_log!(network_manager warn "Couldn't resolve bootstrap node dial info {}: {}", rec, e);
+                    continue;
+                }
+            };
+
+            for di in dial_infos {
+                dial_info_details.push(DialInfoDetail {
+                    dial_info: di,
+                    class: DialInfoClass::Direct,
+                });
+            }
+        }
+
+        // Timestamp
+        let secs_u64 = u64::from_str(&fields[5]).wrap_err("invalid timestamp")?;
+
+        // Extra fields
+        let extra = fields[6..fields.len() - 1].to_vec();
+
+        Ok(Some(BootstrapRecord::new(
+            node_ids,
+            envelope_support,
+            dial_info_details,
+            Some(secs_u64),
+            extra,
+        )))
+    }
+}

veilid-core/src/network_manager/bootstrap/debug.rs

@@ -0,0 +1,55 @@
+use super::*;
+
+impl_veilid_log_facility!("net");
+
+impl NetworkManager {
+    pub async fn debug_info_txtrecord(&self, signing_key_pair: TypedKeyPair) -> String {
+        let routing_table = self.routing_table();
+
+        let dial_info_details = routing_table.dial_info_details(RoutingDomain::PublicInternet);
+        if dial_info_details.is_empty() {
+            return "No PublicInternet DialInfo for TXT Record".to_owned();
+        }
+        let envelope_support = VALID_ENVELOPE_VERSIONS.to_vec();
+        let node_ids = routing_table.node_ids();
+
+        let mut out = "Bootstrap TXT Records:\n".to_owned();
+
+        let bsrec = BootstrapRecord::new(
+            node_ids,
+            envelope_support,
+            dial_info_details,
+            Some(Timestamp::now().as_u64() / 1_000_000u64),
+            vec![],
+        );
+
+        let dial_info_converter = BootstrapDialInfoConverter::default();
+
+        match bsrec.to_v0_string(&dial_info_converter).await {
+            Ok(v) => {
+                //
+                out += &format!("V0:\n{}\n", v);
+            }
+            Err(e) => {
+                //
+                out += &format!("V0 error: {}\n", e);
+            }
+        }
+
+        match bsrec
+            .to_v1_string(self, &dial_info_converter, signing_key_pair)
+            .await
+        {
+            Ok(v) => {
+                //
+                out += &format!("V1:\n{}\n", v);
+            }
+            Err(e) => {
+                //
+                out += &format!("V1 error: {}\n", e);
+            }
+        }
+
+        out
+    }
+}

veilid-core/src/network_manager/bootstrap/dial_info_converter.rs

@@ -0,0 +1,249 @@
+use super::*;
+
+pub struct ShortDialInfo {
+    pub short_url: String,
+    pub hostname: String,
+}
+
+trait DialInfoConverterResolver: Send + Sync {
+    fn ptr_lookup(&self, ip_addr: IpAddr) -> PinBoxFuture<'_, EyreResult<String>>;
+    fn to_socket_addrs(
+        &self,
+        host: &str,
+        default: SocketAddr,
+    ) -> std::io::Result<std::vec::IntoIter<SocketAddr>>;
+}
+
+pub trait DialInfoConverter: Send + Sync {
+    fn try_vec_from_short(&self, short_dial_info: &ShortDialInfo)
+        -> VeilidAPIResult<Vec<DialInfo>>;
+    fn try_vec_from_url(&self, url: &str) -> VeilidAPIResult<Vec<DialInfo>>;
+    fn to_short(&self, dial_info: DialInfo) -> PinBoxFuture<'_, ShortDialInfo>;
+    #[expect(dead_code)]
+    fn to_url(&self, dial_info: DialInfo) -> PinBoxFuture<'_, String>;
+}
+
+impl<C> DialInfoConverter for C
+where
+    C: DialInfoConverterResolver,
+{
+    fn try_vec_from_short(
+        &self,
+        short_dial_info: &ShortDialInfo,
+    ) -> VeilidAPIResult<Vec<DialInfo>> {
+        let short = &short_dial_info.short_url;
+        let hostname = &short_dial_info.hostname;
+
+        if short.len() < 2 {
+            apibail_parse_error!("invalid short url length", short);
+        }
+        let url = match &short[0..1] {
+            "U" => {
+                format!("udp://{}:{}", hostname, &short[1..])
+            }
+            "T" => {
+                format!("tcp://{}:{}", hostname, &short[1..])
+            }
+            "W" => {
+                format!("ws://{}:{}", hostname, &short[1..])
+            }
+            "S" => {
+                format!("wss://{}:{}", hostname, &short[1..])
+            }
+            _ => {
+                apibail_parse_error!("invalid short url type", short);
+            }
+        };
+        self.try_vec_from_url(&url)
+    }
+
+    fn try_vec_from_url(&self, url: &str) -> VeilidAPIResult<Vec<DialInfo>> {
+        let split_url = SplitUrl::from_str(url)
+            .map_err(|e| VeilidAPIError::parse_error(format!("unable to split url: {}", e), url))?;
+
+        let port = match split_url.scheme.as_str() {
+            "udp" | "tcp" => split_url
+                .port
+                .ok_or_else(|| VeilidAPIError::parse_error("Missing port in udp url", url))?,
+            "ws" => split_url.port.unwrap_or(80u16),
+            "wss" => split_url.port.unwrap_or(443u16),
+            _ => {
+                apibail_parse_error!("Invalid dial info url scheme", split_url.scheme);
+            }
+        };
+
+        let socket_addrs = match split_url.host {
+            SplitUrlHost::Hostname(_) => self
+                .to_socket_addrs(
+                    &split_url.host_port(port),
+                    SocketAddr::new(IpAddr::V4(Ipv4Addr::new(0, 0, 0, 0)), port),
+                )
+                .map_err(|_| VeilidAPIError::parse_error("couldn't resolve hostname in url", url))?
+                .collect(),
+            SplitUrlHost::IpAddr(a) => vec![SocketAddr::new(a, port)],
+        };
+
+        let mut out = Vec::new();
+        for sa in socket_addrs {
+            out.push(match split_url.scheme.as_str() {
+                "udp" => DialInfo::udp_from_socketaddr(sa),
+                "tcp" => DialInfo::tcp_from_socketaddr(sa),
+                "ws" => DialInfo::try_ws(
+                    SocketAddress::from_socket_addr(sa).canonical(),
+                    url.to_string(),
+                )?,
+                "wss" => DialInfo::try_wss(
+                    SocketAddress::from_socket_addr(sa).canonical(),
+                    url.to_string(),
+                )?,
+                _ => {
+                    unreachable!("Invalid dial info url scheme")
+                }
+            });
+        }
+        Ok(out)
+    }
+
+    fn to_short(&self, dial_info: DialInfo) -> PinBoxFuture<'_, ShortDialInfo> {
+        pin_dyn_future!(async move {
+            match dial_info {
+                DialInfo::UDP(di) => ShortDialInfo {
+                    short_url: format!("U{}", di.socket_address.port()),
+                    hostname: self
+                        .ptr_lookup(di.socket_address.ip_addr())
+                        .await
+                        .unwrap_or_else(|_| di.socket_address.to_string()),
+                },
+                DialInfo::TCP(di) => ShortDialInfo {
+                    short_url: format!("T{}", di.socket_address.port()),
+                    hostname: self
+                        .ptr_lookup(di.socket_address.ip_addr())
+                        .await
+                        .unwrap_or_else(|_| di.socket_address.to_string()),
+                },
+                DialInfo::WS(di) => {
+                    let mut split_url =
+                        SplitUrl::from_str(&format!("ws://{}", di.request)).unwrap();
+                    if let SplitUrlHost::IpAddr(a) = split_url.host {
+                        if let Ok(host) = self.ptr_lookup(a).await {
+                            split_url.host = SplitUrlHost::Hostname(host);
+                        }
+                    }
+                    ShortDialInfo {
+                        short_url: format!(
+                            "W{}{}",
+                            split_url.port.unwrap_or(80),
+                            split_url
+                                .path
+                                .map(|p| format!("/{}", p))
+                                .unwrap_or_default()
+                        ),
+                        hostname: split_url.host.to_string(),
+                    }
+                }
+                DialInfo::WSS(di) => {
+                    let mut split_url =
+                        SplitUrl::from_str(&format!("wss://{}", di.request)).unwrap();
+                    if let SplitUrlHost::IpAddr(a) = split_url.host {
+                        if let Ok(host) = self.ptr_lookup(a).await {
+                            split_url.host = SplitUrlHost::Hostname(host);
+                        }
+                    }
+                    ShortDialInfo {
+                        short_url: format!(
+                            "S{}{}",
+                            split_url.port.unwrap_or(443),
+                            split_url
+                                .path
+                                .map(|p| format!("/{}", p))
+                                .unwrap_or_default()
+                        ),
+                        hostname: split_url.host.to_string(),
+                    }
+                }
+            }
+        })
+    }
+
+    fn to_url(&self, dial_info: DialInfo) -> PinBoxFuture<'_, String> {
+        pin_dyn_future!(async move {
+            match dial_info {
+                DialInfo::UDP(di) => self
+                    .ptr_lookup(di.socket_address.ip_addr())
+                    .await
+                    .map(|h| format!("udp://{}:{}", h, di.socket_address.port()))
+                    .unwrap_or_else(|_| format!("udp://{}", di.socket_address)),
+                DialInfo::TCP(di) => self
+                    .ptr_lookup(di.socket_address.ip_addr())
+                    .await
+                    .map(|h| format!("tcp://{}:{}", h, di.socket_address.port()))
+                    .unwrap_or_else(|_| format!("tcp://{}", di.socket_address)),
+                DialInfo::WS(di) => {
+                    let mut split_url =
+                        SplitUrl::from_str(&format!("ws://{}", di.request)).unwrap();
+                    if let SplitUrlHost::IpAddr(a) = split_url.host {
+                        if let Ok(host) = self.ptr_lookup(a).await {
+                            split_url.host = SplitUrlHost::Hostname(host);
+                        }
+                    }
+                    split_url.to_string()
+                }
+                DialInfo::WSS(di) => {
+                    let mut split_url =
+                        SplitUrl::from_str(&format!("wss://{}", di.request)).unwrap();
+                    if let SplitUrlHost::IpAddr(a) = split_url.host {
+                        if let Ok(host) = self.ptr_lookup(a).await {
+                            split_url.host = SplitUrlHost::Hostname(host);
+                        }
+                    }
+                    split_url.to_string()
+                }
+            }
+        })
+    }
+}
+
+#[derive(Debug, Clone, Copy, Default)]
+pub struct BootstrapDialInfoConverter {}
+
+impl DialInfoConverterResolver for BootstrapDialInfoConverter {
+    fn ptr_lookup(&self, ip_addr: IpAddr) -> PinBoxFuture<'_, EyreResult<String>> {
+        pin_dyn_future!(async move { intf::ptr_lookup(ip_addr).await })
+    }
+
+    #[allow(unused_variables)]
+    fn to_socket_addrs(
+        &self,
+        host: &str,
+        default: SocketAddr,
+    ) -> std::io::Result<std::vec::IntoIter<SocketAddr>> {
+        // Resolve if possible, WASM doesn't support resolution and doesn't need it to connect to the dialinfo
+        // This will not be used on signed dialinfo, only for bootstrapping, so we don't need to worry about
+        // the '0.0.0.0' address being propagated across the routing table
+
+        cfg_if::cfg_if! {
+            if #[cfg(all(target_arch = "wasm32", target_os = "unknown"))] {
+                Ok(vec![default].into_iter())
+            } else {
+                host.to_socket_addrs()
+            }
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy, Default)]
+pub struct MockDialInfoConverter {}
+
+impl DialInfoConverterResolver for MockDialInfoConverter {
+    fn ptr_lookup(&self, _ip_addr: IpAddr) -> PinBoxFuture<'_, EyreResult<String>> {
+        pin_dyn_future!(async move { Ok("fake_hostname".to_string()) })
+    }
+
+    fn to_socket_addrs(
+        &self,
+        _host: &str,
+        default: SocketAddr,
+    ) -> std::io::Result<std::vec::IntoIter<SocketAddr>> {
+        Ok(vec![default].into_iter())
+    }
+}

veilid-core/src/network_manager/bootstrap/direct_bootstrap/mod.rs

@@ -0,0 +1,150 @@
+mod v0;
+mod v1;
+
+use super::*;
+use v1::*;
+
+impl_veilid_log_facility!("net");
+
+impl NetworkManager {
+    /// Direct bootstrap request
+    /// Sends a bootstrap request to a dialinfo and returns the list of peers to bootstrap with
+    /// If no bootstrap keys are specified, uses the v0 mechanism, otherwise uses the v1 mechanism
+    #[instrument(level = "trace", target = "net", err, skip(self))]
+    pub async fn direct_bootstrap(&self, dial_info: DialInfo) -> EyreResult<Vec<Arc<PeerInfo>>> {
+        let direct_boot_version = self.config().with(|c| {
+            if c.network.routing_table.bootstrap_keys.is_empty() {
+                0
+            } else {
+                1
+            }
+        });
+
+        if direct_boot_version == 0 {
+            self.direct_bootstrap_v0(dial_info).await
+        } else {
+            self.direct_bootstrap_v1(dial_info).await
+        }
+    }
+
+    /// Uses the bootstrap v0 (BOOT) mechanism
+    #[instrument(level = "trace", target = "net", err, skip(self))]
+    async fn direct_bootstrap_v0(&self, dial_info: DialInfo) -> EyreResult<Vec<Arc<PeerInfo>>> {
+        let timeout_ms = self.config().with(|c| c.network.rpc.timeout_ms);
+        // Send boot magic to requested peer address
+        let data = BOOT_MAGIC.to_vec();
+
+        let out_data: Vec<u8> = network_result_value_or_log!(self self
+            .net()
+            .send_recv_data_unbound_to_dial_info(dial_info, data, timeout_ms)
+            .await? => [ format!(": dial_info={}, data.len={}", dial_info, data.len()) ]
+        {
+            return Ok(Vec::new());
+        });
+
+        let bootstrap_peerinfo_str =
+            std::str::from_utf8(&out_data).wrap_err("bad utf8 in boot peerinfo")?;
+
+        let bootstrap_peerinfo: Vec<PeerInfo> = match deserialize_json(bootstrap_peerinfo_str) {
+            Ok(v) => v,
+            Err(e) => {
+                error!("{}", e);
+                return Err(e).wrap_err("failed to deserialize peerinfo");
+            }
+        };
+
+        Ok(bootstrap_peerinfo.into_iter().map(Arc::new).collect())
+    }
+
+    /// Uses the bootstrap v1 (B01T) mechanism
+    #[instrument(level = "trace", target = "net", err, skip(self))]
+    async fn direct_bootstrap_v1(&self, dial_info: DialInfo) -> EyreResult<Vec<Arc<PeerInfo>>> {
+        let timeout_ms = self.config().with(|c| c.network.rpc.timeout_ms);
+
+        // Send boot magic to requested peer address
+        let data = B01T_MAGIC.to_vec();
+
+        let out_data: Vec<u8> = network_result_value_or_log!(self self
+            .net()
+            .send_recv_data_unbound_to_dial_info(dial_info, data, timeout_ms)
+            .await? => [ format!(": dial_info={}, data.len={}", dial_info, data.len()) ]
+        {
+            return Ok(Vec::new());
+        });
+
+        let bootv1response_str =
+            std::str::from_utf8(&out_data).wrap_err("bad utf8 in bootstrap v1 records")?;
+
+        veilid_log!(self debug "Direct bootstrap v1 response: {}", bootv1response_str);
+
+        let bootv1response: BootV1Response = match deserialize_json(bootv1response_str) {
+            Ok(v) => v,
+            Err(e) => {
+                error!("{}", e);
+                return Err(e).wrap_err("failed to deserialize bootstrap v1 response");
+            }
+        };
+
+        // Parse v1 records
+        let bsrecs = match self.parse_bootstrap_v1(&bootv1response.records) {
+            Ok(v) => v,
+            Err(e) => {
+                veilid_log!(self debug "Direct bootstrap v1 parsing failure: {}", e);
+                return Err(e);
+            }
+        };
+
+        veilid_log!(self debug "Direct bootstrap v1 resolution: {:#?}", bsrecs);
+
+        // Returned bootstrapped peers
+        let routing_table = self.routing_table();
+
+        let peers: Vec<Arc<PeerInfo>> = bsrecs
+            .into_iter()
+            .filter_map(|bsrec| {
+                if routing_table.matches_own_node_id(bsrec.node_ids()) {
+                    veilid_log!(self debug "Ignoring own node in bootstrap list");
+                    None
+                } else {
+                    // If signed peer info exists for this record, use it
+                    // This is important for browser websocket bootstrapping where the
+                    // dialinfo in the bootstrap record has an unspecified IP address,
+                    // and as such, a routing domain can not be determined for it
+                    // by the code that receives the FindNodeA result
+                    for pi in bootv1response.peers.iter().cloned() {
+                        if pi.node_ids().contains_any(bsrec.node_ids()) {
+                            return Some(pi);
+                        }
+                    }
+
+                    // Otherwise use an unsigned peerinfo and try to resolve it directly from the bootstrap record
+                    // The bootstrap will be rejected if a FindNodeQ could not resolve the peer info
+
+                    // Get crypto support from list of node ids
+                    let crypto_support = bsrec.node_ids().kinds();
+
+                    // Make unsigned SignedNodeInfo
+                    let sni = SignedNodeInfo::Direct(SignedDirectNodeInfo::with_no_signature(
+                        NodeInfo::new(
+                            NetworkClass::InboundCapable, // Bootstraps are always inbound capable
+                            ProtocolTypeSet::all(), // Bootstraps are always capable of all protocols
+                            AddressTypeSet::all(),  // Bootstraps are always IPV4 and IPV6 capable
+                            bsrec.envelope_support().to_vec(), // Envelope support is as specified in the bootstrap list
+                            crypto_support, // Crypto support is derived from list of node ids
+                            vec![],         // Bootstrap needs no capabilities
+                            bsrec.dial_info_details().to_vec(), // Dial info is as specified in the bootstrap list
+                        ),
+                    ));
+
+                    Some(Arc::new(PeerInfo::new(
+                        RoutingDomain::PublicInternet,
+                        bsrec.node_ids().clone(),
+                        sni,
+                    )))
+                }
+            })
+            .collect();
+
+        Ok(peers)
+    }
+}

veilid-core/src/network_manager/bootstrap/direct_bootstrap/v0.rs

@@ -0,0 +1,146 @@
+use super::*;
+
+impl_veilid_log_facility!("net");
+
+impl NetworkManager {
+    /// Direct bootstrap request handler (separate fallback mechanism from cheaper TXT bootstrap mechanism)
+    #[instrument(level = "trace", target = "net", skip(self), ret, err)]
+    pub async fn handle_boot_v0_request(&self, flow: Flow) -> EyreResult<NetworkResult<()>> {
+        // Get a bunch of nodes with a range of crypto kinds, protocols and capabilities
+        let bootstrap_nodes = self.find_bootstrap_nodes_filtered(2);
+
+        // Serialize out peer info
+        let bootstrap_peerinfo: Vec<Arc<PeerInfo>> = bootstrap_nodes
+            .iter()
+            .filter_map(|nr| nr.get_peer_info(RoutingDomain::PublicInternet))
+            .collect();
+        let json_bytes = serialize_json(bootstrap_peerinfo).as_bytes().to_vec();
+
+        veilid_log!(self trace "BOOT reponse: {}", String::from_utf8_lossy(&json_bytes));
+
+        // Reply with a chunk of signed routing table
+        let net = self.net();
+        match pin_future_closure!(net.send_data_to_existing_flow(flow, json_bytes)).await? {
+            SendDataToExistingFlowResult::Sent(_) => {
+                // Bootstrap reply was sent
+                Ok(NetworkResult::value(()))
+            }
+            SendDataToExistingFlowResult::NotSent(_) => Ok(NetworkResult::no_connection_other(
+                "bootstrap reply could not be sent",
+            )),
+        }
+    }
+
+    /// Retrieve up to N of each type of protocol capable nodes for a single crypto kind
+    fn find_bootstrap_nodes_filtered_per_crypto_kind(
+        &self,
+        crypto_kind: CryptoKind,
+        max_per_type: usize,
+    ) -> Vec<NodeRef> {
+        let protocol_types = [
+            ProtocolType::UDP,
+            ProtocolType::TCP,
+            ProtocolType::WS,
+            ProtocolType::WSS,
+        ];
+
+        let protocol_types_len = protocol_types.len();
+        let mut nodes_proto_v4 = [0usize, 0usize, 0usize, 0usize];
+        let mut nodes_proto_v6 = [0usize, 0usize, 0usize, 0usize];
+
+        let filter = Box::new(
+            move |rti: &RoutingTableInner, entry: Option<Arc<BucketEntry>>| {
+                let entry = entry.unwrap();
+                entry.with(rti, |_rti, e| {
+                    // skip nodes on our local network here
+                    if e.has_node_info(RoutingDomain::LocalNetwork.into()) {
+                        return false;
+                    }
+
+                    // Ensure crypto kind is supported
+                    if !e.crypto_kinds().contains(&crypto_kind) {
+                        return false;
+                    }
+
+                    // Only nodes with direct publicinternet node info
+                    let Some(signed_node_info) = e.signed_node_info(RoutingDomain::PublicInternet)
+                    else {
+                        return false;
+                    };
+
+                    // Only direct node info
+                    let SignedNodeInfo::Direct(signed_direct_node_info) = signed_node_info else {
+                        return false;
+                    };
+                    let node_info = signed_direct_node_info.node_info();
+
+                    // Bootstraps must have -only- inbound capable network class and direct dialinfo
+                    if !node_info.is_fully_direct_inbound() {
+                        return false;
+                    }
+
+                    // Must have connectivity capabilities
+                    if !node_info.has_all_capabilities(CONNECTIVITY_CAPABILITIES) {
+                        return false;
+                    }
+
+                    // Check for direct dialinfo and a good mix of protocol and address types
+                    let mut keep = false;
+                    for did in node_info.dial_info_detail_list() {
+                        if matches!(did.dial_info.address_type(), AddressType::IPV4) {
+                            for (n, protocol_type) in protocol_types.iter().enumerate() {
+                                if nodes_proto_v4[n] < max_per_type
+                                    && did.dial_info.protocol_type() == *protocol_type
+                                {
+                                    nodes_proto_v4[n] += 1;
+                                    keep = true;
+                                }
+                            }
+                        } else if matches!(did.dial_info.address_type(), AddressType::IPV6) {
+                            for (n, protocol_type) in protocol_types.iter().enumerate() {
+                                if nodes_proto_v6[n] < max_per_type
+                                    && did.dial_info.protocol_type() == *protocol_type
+                                {
+                                    nodes_proto_v6[n] += 1;
+                                    keep = true;
+                                }
+                            }
+                        }
+                    }
+                    keep
+                })
+            },
+        ) as RoutingTableEntryFilter;
+
+        let filters = VecDeque::from([filter]);
+
+        self.routing_table().find_preferred_fastest_nodes(
+            protocol_types_len * 2 * max_per_type,
+            filters,
+            |_rti, entry: Option<Arc<BucketEntry>>| {
+                NodeRef::new(self.registry(), entry.unwrap().clone())
+            },
+        )
+    }
+
+    /// Retrieve up to N of each type of protocol capable nodes for all crypto kinds
+    fn find_bootstrap_nodes_filtered(&self, max_per_type: usize) -> Vec<NodeRef> {
+        let mut out =
+            self.find_bootstrap_nodes_filtered_per_crypto_kind(VALID_CRYPTO_KINDS[0], max_per_type);
+
+        // Merge list of nodes so we don't have duplicates
+        for crypto_kind in &VALID_CRYPTO_KINDS[1..] {
+            let nrs =
+                self.find_bootstrap_nodes_filtered_per_crypto_kind(*crypto_kind, max_per_type);
+            'nrloop: for nr in nrs {
+                for nro in &out {
+                    if nro.same_entry(&nr) {
+                        continue 'nrloop;
+                    }
+                }
+                out.push(nr);
+            }
+        }
+        out
+    }
+}

veilid-core/src/network_manager/bootstrap/direct_bootstrap/v1.rs

@@ -0,0 +1,107 @@
+use super::*;
+
+impl_veilid_log_facility!("net");
+
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct BootV1Response {
+    pub records: Vec<String>,
+    pub peers: Vec<Arc<PeerInfo>>,
+}
+
+impl NetworkManager {
+    /// Direct bootstrap v1 request handler
+    /// This is a proxy mechanism to the TXT bootstrap mechanism
+    /// that is intended for supporting WS/WSS nodes that can not perform DNS TXT lookups,
+    /// however this does work over straight UDP and TCP protocols as well.
+    #[instrument(level = "trace", target = "net", skip(self), ret, err)]
+    pub async fn handle_boot_v1_request(&self, flow: Flow) -> EyreResult<NetworkResult<()>> {
+        let bootstraps = self
+            .config()
+            .with(|c| c.network.routing_table.bootstrap.clone());
+
+        // Don't bother if bootstraps aren't configured
+        if bootstraps.is_empty() {
+            return Ok(NetworkResult::service_unavailable(
+                "no bootstraps configured",
+            ));
+        }
+
+        // Extract only the TXT hostnames
+        let dial_info_converter = BootstrapDialInfoConverter::default();
+        let mut bootstrap_txt_names = Vec::<String>::new();
+        for bootstrap in bootstraps {
+            if dial_info_converter.try_vec_from_url(&bootstrap).is_ok() {
+                // skip direct bootstraps here
+            } else {
+                bootstrap_txt_names.push(bootstrap);
+            }
+        }
+
+        // Process bootstraps into TXT strings
+        let mut unord = FuturesUnordered::<PinBoxFuture<EyreResult<Vec<String>>>>::new();
+
+        for bstn in bootstrap_txt_names {
+            // TXT bootstrap
+            unord.push(pin_dyn_future!(async {
+                let bstn = bstn;
+                self.resolve_bootstrap_txt_strings(bstn).await
+            }));
+        }
+
+        let mut txt_strings_set = HashSet::<String>::new();
+        while let Some(res) = unord.next().await {
+            match res {
+                Ok(txt_strings) => {
+                    for txt_string in txt_strings {
+                        txt_strings_set.insert(txt_string);
+                    }
+                }
+                Err(e) => {
+                    veilid_log!(self debug "Direct bootstrap resolution error: {}", e);
+                }
+            }
+        }
+
+        let mut records = txt_strings_set.into_iter().collect::<Vec<_>>();
+        records.sort();
+
+        // Add peer infos if we have them, only for the peers present in the records
+        let routing_table = self.routing_table();
+        let routing_domain = RoutingDomain::PublicInternet;
+        let bsrecs = self.parse_bootstrap_v1(&records)?;
+
+        let peers: Vec<Arc<PeerInfo>> = bsrecs
+            .into_iter()
+            .filter_map(|bsrec| {
+                if routing_table.matches_own_node_id(bsrec.node_ids()) {
+                    routing_table.get_published_peer_info(routing_domain)
+                } else if let Some(best_node_id) = bsrec.node_ids().best() {
+                    if let Some(nr) = routing_table.lookup_node_ref(best_node_id).ok().flatten() {
+                        nr.get_peer_info(routing_domain)
+                    } else {
+                        None
+                    }
+                } else {
+                    None
+                }
+            })
+            .collect();
+
+        // Serialize out bootstrap response
+        let bootv1response = BootV1Response { records, peers };
+        let json_bytes = serialize_json(bootv1response).as_bytes().to_vec();
+        veilid_log!(self trace "B01T reponse: {}", String::from_utf8_lossy(&json_bytes));
+
+        // Reply with bootstrap records
+        let net = self.net();
+        match pin_future_closure!(net.send_data_to_existing_flow(flow, json_bytes)).await? {
+            SendDataToExistingFlowResult::Sent(_) => {
+                // Bootstrap reply was sent
+                Ok(NetworkResult::value(()))
+            }
+            SendDataToExistingFlowResult::NotSent(_) => Ok(NetworkResult::no_connection_other(
+                "bootstrap reply could not be sent",
+            )),
+        }
+    }
+}

veilid-core/src/network_manager/bootstrap/mod.rs

@@ -0,0 +1,165 @@
+mod bootstrap_record;
+mod debug;
+mod dial_info_converter;
+mod direct_bootstrap;
+mod txt_bootstrap;
+
+use super::*;
+use futures_util::StreamExt as _;
+use stop_token::future::FutureExt as _;
+
+pub use bootstrap_record::*;
+pub use dial_info_converter::*;
+pub use txt_bootstrap::*;
+
+impl_veilid_log_facility!("net");
+
+impl NetworkManager {
+    //#[instrument(level = "trace", skip(self), err)]
+    pub fn bootstrap_with_peer(
+        &self,
+        crypto_kinds: Vec<CryptoKind>,
+        pi: Arc<PeerInfo>,
+        unord: &FuturesUnordered<PinBoxFutureStatic<Option<NodeRef>>>,
+    ) {
+        veilid_log!(self trace
+            "--- bootstrapping {} with {:?}",
+            pi.node_ids(),
+            pi.signed_node_info().node_info().dial_info_detail_list()
+        );
+
+        let routing_domain = pi.routing_domain();
+        let routing_table = self.routing_table();
+
+        let nr = match routing_table.register_node_with_peer_info(pi, true) {
+            Ok(nr) => nr,
+            Err(e) => {
+                veilid_log!(self error "failed to register bootstrap peer info: {}", e);
+                return;
+            }
+        };
+
+        // Add this our futures to process in parallel
+        for crypto_kind in crypto_kinds {
+            // Bootstrap this crypto kind
+            let nr = nr.unfiltered();
+            unord.push(Box::pin(
+                    async move {
+                        let network_manager = nr.network_manager();
+                        let routing_table = nr.routing_table();
+
+                        // Get what contact method would be used for contacting the bootstrap
+                        let bsdi = match network_manager
+                            .get_node_contact_method(nr.sequencing_filtered(Sequencing::PreferOrdered))
+                        {
+                            Ok(Some(ncm)) if ncm.is_direct() => ncm.direct_dial_info().unwrap(),
+                            Ok(v) => {
+                                veilid_log!(nr debug "invalid contact method for bootstrap, ignoring peer: {:?}", v);
+                                // let _ =
+                                //     network_manager
+                                //     .get_node_contact_method(nr.clone());
+                                return None;
+                            }
+                            Err(e) => {
+                                veilid_log!(nr warn "unable to bootstrap: {}", e);
+                                return None;
+                            }
+                        };
+
+                        // Need VALID signed peer info, so ask bootstrap to find_node of itself
+                        // which will ensure it has the bootstrap's signed peer info as part of the response
+                        let _ = routing_table.find_nodes_close_to_node_ref(crypto_kind, nr.sequencing_filtered(Sequencing::PreferOrdered), vec![]).await;
+
+                        // Ensure we got the signed peer info
+                        if !nr.signed_node_info_has_valid_signature(routing_domain) {
+                            veilid_log!(nr info "bootstrap server is not responding for dialinfo: {}", bsdi);
+
+                            // Try a different dialinfo next time
+                            network_manager.address_filter().set_dial_info_failed(bsdi);
+                            None
+                        } else {
+                            // otherwise this bootstrap is valid, lets ask it to find ourselves now
+                            // We should prefer nodes that have relaying, signaling, routing and validation of dial info
+                            // for our initial nodes in our routing table because we need these things in order to
+                            // properly attach to the network
+                            routing_table.reverse_find_node(crypto_kind, nr.clone(), true, CONNECTIVITY_CAPABILITIES.to_vec()).await;
+
+                            veilid_log!(nr info "bootstrap of {} successful via {}", crypto_kind, nr);
+                            Some(nr)
+                        }
+                    }
+                    .instrument(Span::current()),
+                ));
+        }
+    }
+
+    /// Takes in a list of bootstrap peer info, and attempts bootstrapping
+    /// A list of valid bootstrap peer noderefs is returned
+    #[instrument(level = "trace", skip(self), err)]
+    pub async fn bootstrap_with_peer_list(
+        &self,
+        bootstrap_peers: Vec<Arc<PeerInfo>>,
+        stop_token: StopToken,
+    ) -> EyreResult<Vec<NodeRef>> {
+        if bootstrap_peers.is_empty() {
+            veilid_log!(self debug "No peers suitable for bootstrap");
+            return Ok(vec![]);
+        }
+
+        veilid_log!(self debug "Bootstrap peers: {:?}", &bootstrap_peers);
+
+        // Get crypto kinds to bootstrap
+        let crypto_kinds = self.get_bootstrap_crypto_kinds();
+        veilid_log!(self debug "Bootstrap crypto kinds: {:?}", &crypto_kinds);
+
+        // Run all bootstrap operations concurrently
+        let mut unord = FuturesUnordered::<PinBoxFutureStatic<Option<NodeRef>>>::new();
+        for peer in bootstrap_peers {
+            // Validate bootstrap key for crypto kinds
+            let mut peer_has_crypto_kind = false;
+            for ck in crypto_kinds.iter().copied() {
+                if peer.node_ids().get(ck).is_some() {
+                    peer_has_crypto_kind = true;
+                }
+            }
+
+            if peer_has_crypto_kind {
+                veilid_log!(self info "Attempting bootstrap: {}", peer.node_ids());
+                self.bootstrap_with_peer(crypto_kinds.clone(), peer, &unord);
+            }
+        }
+
+        // Wait for all bootstrap operations to complete before we complete the singlefuture
+        let mut valid_bootstraps = vec![];
+        while let Ok(Some(res)) = unord.next().timeout_at(stop_token.clone()).await {
+            if let Some(valid_bootstrap) = res {
+                valid_bootstraps.push(valid_bootstrap);
+            }
+        }
+        Ok(valid_bootstraps)
+    }
+
+    /// Get counts by crypto kind and figure out which crypto kinds need bootstrapping
+    pub fn get_bootstrap_crypto_kinds(&self) -> Vec<CryptoKind> {
+        let routing_table = self.routing_table();
+
+        let live_entry_counts = routing_table.cached_live_entry_counts();
+
+        let mut crypto_kinds = Vec::new();
+        for rd in BOOTSTRAP_ROUTING_DOMAINS {
+            for crypto_kind in VALID_CRYPTO_KINDS {
+                // Do we need to bootstrap this crypto kind?
+                let eckey = (rd, crypto_kind);
+                let cnt = live_entry_counts
+                    .connectivity_capabilities
+                    .get(&eckey)
+                    .copied()
+                    .unwrap_or_default();
+                if cnt < MIN_BOOTSTRAP_CONNECTIVITY_PEERS {
+                    crypto_kinds.push(crypto_kind);
+                }
+            }
+        }
+        crypto_kinds
+    }
+}

veilid-core/src/network_manager/bootstrap/txt_bootstrap/mod.rs

@@ -0,0 +1,202 @@
+mod v0;
+mod v1;
+
+use super::*;
+
+pub use v0::*;
+pub use v1::*;
+
+const SUPPORTED_BOOTSTRAP_TXT_VERSIONS: [u8; 2] =
+    [BOOTSTRAP_TXT_VERSION_0, BOOTSTRAP_TXT_VERSION_1];
+
+impl_veilid_log_facility!("net");
+
+impl NetworkManager {
+    /// TXT bootstrap request
+    /// Sends a bootstrap request via DNS for TXT records and bootstraps with them
+    #[instrument(level = "trace", target = "net", err, skip(self))]
+    pub async fn txt_bootstrap(&self, hostname: String) -> EyreResult<Vec<Arc<PeerInfo>>> {
+        // Get the minimum bootstrap version we are supporting
+        // If no keys are available, allow v0.
+        // If bootstrap keys are specified, require at least v1.
+        let min_boot_version = self.config().with(|c| {
+            if c.network.routing_table.bootstrap_keys.is_empty() {
+                BOOTSTRAP_TXT_VERSION_0
+            } else {
+                BOOTSTRAP_TXT_VERSION_1
+            }
+        });
+
+        // Resolve bootstrap servers and recurse their TXT entries
+        // This only operates in the PublicInternet routing domain because DNS is inherently
+        // for PublicInternet use. Other routing domains may use other mechanisms
+        // such as LLMNR/MDNS/DNS-SD.
+        let txt_strings =
+            match pin_future!(self.resolve_bootstrap_txt_strings(hostname.clone())).await {
+                Ok(v) => v,
+                Err(e) => {
+                    veilid_log!(self debug "Bootstrap resolution failure: {}", e);
+                    return Err(e);
+                }
+            };
+
+        // Heuristic to determine which version to parse
+        // Take the first record and see if there is a '|' delimited string in it
+        // If so, the first field is a record version number.
+        // If no '|' delimited string is found, then this is a v0 hostname record
+        let mut opt_max_version: Option<u8> = None;
+
+        for txt_string in &txt_strings {
+            let v = match txt_string.split_once('|').map(|x| u8::from_str(x.0)) {
+                Some(Err(e)) => {
+                    // Parse error, skip it
+                    veilid_log!(self debug "malformed txt record in bootstrap response: {}\n{}", txt_string, e);
+                    continue;
+                }
+                Some(Ok(v)) => {
+                    // Version
+                    if SUPPORTED_BOOTSTRAP_TXT_VERSIONS.contains(&v) {
+                        v
+                    } else {
+                        veilid_log!(self debug "unsupported bootstrap record version: {}", txt_string);
+                        continue;
+                    }
+                }
+                None => {
+                    // No '|' means try version 0
+                    0u8
+                }
+            };
+
+            if v < min_boot_version {
+                veilid_log!(self debug "ignoring older bootstrap record version: {}", txt_string);
+                continue;
+            }
+
+            opt_max_version = opt_max_version.map(|x| x.max(v)).or(Some(v));
+        }
+        let Some(max_version) = opt_max_version else {
+            veilid_log!(self debug "no suitable txt record in bootstrap response");
+            return Ok(vec![]);
+        };
+
+        // Process the best version available
+        let bsrecs = match max_version {
+            BOOTSTRAP_TXT_VERSION_0 => {
+                // Resolve second-level hostname
+                let record_strings = self.resolve_bootstrap_v0(hostname, txt_strings).await?;
+
+                // Parse v0 records
+                let bsrecs = match self.parse_bootstrap_v0(&record_strings) {
+                    Ok(v) => v,
+                    Err(e) => {
+                        veilid_log!(self debug "Bootstrap v0 parsing failure: {}", e);
+                        return Err(e);
+                    }
+                };
+
+                veilid_log!(self debug "Bootstrap v0 resolution: {:#?}", bsrecs);
+
+                bsrecs
+            }
+            BOOTSTRAP_TXT_VERSION_1 => {
+                // Parse v1 records
+                let bsrecs = match self.parse_bootstrap_v1(&txt_strings) {
+                    Ok(v) => v,
+                    Err(e) => {
+                        veilid_log!(self debug "Bootstrap v1 parsing failure: {}", e);
+                        return Err(e);
+                    }
+                };
+
+                veilid_log!(self debug "Bootstrap v1 resolution: {:#?}", bsrecs);
+
+                bsrecs
+            }
+            _ => {
+                veilid_log!(self debug "unsupported bootstrap version");
+                return Ok(vec![]);
+            }
+        };
+
+        let routing_table = self.routing_table();
+
+        let peers: Vec<Arc<PeerInfo>> = bsrecs
+            .into_iter()
+            .filter_map(|bsrec| {
+                if routing_table.matches_own_node_id(bsrec.node_ids()) {
+                    veilid_log!(self debug "Ignoring own node in bootstrap list");
+                    None
+                } else {
+                    // Get crypto support from list of node ids
+                    let crypto_support = bsrec.node_ids().kinds();
+
+                    // Make unsigned SignedNodeInfo
+                    let sni = SignedNodeInfo::Direct(SignedDirectNodeInfo::with_no_signature(
+                        NodeInfo::new(
+                            NetworkClass::InboundCapable, // Bootstraps are always inbound capable
+                            ProtocolTypeSet::all(), // Bootstraps are always capable of all protocols
+                            AddressTypeSet::all(),  // Bootstraps are always IPV4 and IPV6 capable
+                            bsrec.envelope_support().to_vec(), // Envelope support is as specified in the bootstrap list
+                            crypto_support, // Crypto support is derived from list of node ids
+                            vec![],         // Bootstrap needs no capabilities
+                            bsrec.dial_info_details().to_vec(), // Dial info is as specified in the bootstrap list
+                        ),
+                    ));
+
+                    Some(Arc::new(PeerInfo::new(
+                        RoutingDomain::PublicInternet,
+                        bsrec.node_ids().clone(),
+                        sni,
+                    )))
+                }
+            })
+            .collect();
+
+        Ok(peers)
+    }
+
+    /// Bootstrap resolution from TXT into strings
+    /// This is cached as to minimize the number of outbound network requests on bootstrap servers
+    #[instrument(level = "trace", skip(self), ret, err)]
+    pub async fn resolve_bootstrap_txt_strings(&self, hostname: String) -> EyreResult<Vec<String>> {
+        // Lookup hostname in cache
+        let cur_ts = Timestamp::now();
+        if let Some(res) = self.inner.lock().txt_lookup_cache.get(&hostname) {
+            // Ensure timestamp has not expired
+            if cur_ts < res.0 {
+                // Return cached strings
+                let txt_strings = res.1.clone();
+                veilid_log!(self debug " Cached TXT: {:?} => {:?}", hostname, txt_strings);
+                return Ok(txt_strings);
+            }
+        }
+
+        // Not in cache or cache expired
+        veilid_log!(self debug "Resolving bootstrap TXT: {:?}", hostname);
+
+        // Get TXT record for bootstrap (bootstrap.veilid.net, bootstrap-v1.veilid.net, or similar)
+        let txt_strings = match intf::txt_lookup(&hostname).await {
+            Ok(v) => v,
+            Err(e) => {
+                veilid_log!(self warn
+                    "Network may be down. No bootstrap resolution for '{}': {}",
+                    hostname, e
+                );
+                return Ok(vec![]);
+            }
+        };
+        veilid_log!(self debug " TXT: {:?} => {:?}", hostname, txt_strings);
+
+        // Cache result if we have one
+        if !txt_strings.is_empty() {
+            let exp_ts = cur_ts + TXT_LOOKUP_EXPIRATION;
+            self.inner
+                .lock()
+                .txt_lookup_cache
+                .insert(hostname, (exp_ts, txt_strings.clone()));
+        }
+
+        Ok(txt_strings)
+    }
+}

veilid-core/src/network_manager/bootstrap/txt_bootstrap/v0.rs

@@ -0,0 +1,122 @@
+use super::*;
+use futures_util::StreamExt as _;
+
+pub const BOOTSTRAP_TXT_VERSION_0: u8 = 0;
+
+impl_veilid_log_facility!("net");
+
+impl NetworkManager {
+    /// Bootstrap resolution from TXT into strings
+    #[instrument(level = "trace", skip(self), ret, err)]
+    pub async fn resolve_bootstrap_v0(
+        &self,
+        hostname: String,
+        txt_strings: Vec<String>,
+    ) -> EyreResult<Vec<String>> {
+        veilid_log!(self debug "Resolving v0 bootstraps: {:?}", txt_strings);
+
+        // Resolve from bootstrap root to bootstrap hostnames
+        let mut bsnames = Vec::<String>::new();
+
+        for txt_string in txt_strings {
+            // Split the bootstrap name record by commas
+            for rec in txt_string.split(',') {
+                let rec = rec.trim();
+                // If the name specified is fully qualified, go with it
+                let bsname = if rec.ends_with('.') {
+                    rec.to_string()
+                }
+                // If the name is not fully qualified, prepend it to the bootstrap name
+                else {
+                    format!("{}.{}", rec, hostname)
+                };
+
+                // Add to the list of bootstrap name to look up
+                bsnames.push(bsname);
+            }
+        }
+
+        // Get bootstrap nodes from hostnames concurrently
+        let mut unord = FuturesUnordered::new();
+        for bsname in bsnames {
+            unord.push(
+                async move {
+                    // look up bootstrap node txt records
+                    let bsnirecords = match intf::txt_lookup(&bsname).await {
+                        Err(e) => {
+                            veilid_log!(self warn
+                                "Network may be down. Bootstrap node txt lookup failed for {}: {}",
+                                bsname, e
+                            );
+                            return None;
+                        }
+                        Ok(v) => v,
+                    };
+                    veilid_log!(self debug " TXT: {:?} => {:?}", bsname, bsnirecords);
+
+                    Some(bsnirecords)
+                }
+                .instrument(Span::current()),
+            );
+        }
+
+        let mut all_records: HashSet<String> = HashSet::new();
+        while let Some(bootstrap_records) = unord.next().await {
+            let Some(bootstrap_records) = bootstrap_records else {
+                continue;
+            };
+            for br in bootstrap_records {
+                all_records.insert(br);
+            }
+        }
+
+        let mut all_records_sorted = all_records.into_iter().collect::<Vec<_>>();
+        all_records_sorted.sort();
+
+        Ok(all_records_sorted)
+    }
+
+    /// Parse v0 bootstrap record strings into BootstrapRecord structs
+    #[instrument(level = "trace", skip(self), ret, err)]
+    pub fn parse_bootstrap_v0(
+        &self,
+        record_strings: &[String],
+    ) -> EyreResult<Vec<BootstrapRecord>> {
+        veilid_log!(self debug "Parsing v0 bootstraps: {:?}", record_strings);
+
+        // For each record string resolve into BootstrapRecord pairs
+        let dial_info_converter = BootstrapDialInfoConverter::default();
+
+        let mut bootstrap_records: Vec<BootstrapRecord> = Vec::new();
+        for record_string in record_strings {
+            let Some(bootstrap_record) =
+                BootstrapRecord::new_from_v0_str(self, &dial_info_converter, record_string)?
+            else {
+                continue;
+            };
+
+            bootstrap_records.push(bootstrap_record);
+        }
+
+        let mut merged_bootstrap_records: Vec<BootstrapRecord> = Vec::new();
+
+        for mut bsrec in bootstrap_records {
+            let mut mbi = 0;
+            while mbi < merged_bootstrap_records.len() {
+                let mbr = &mut merged_bootstrap_records[mbi];
+                if mbr.node_ids().contains_any(bsrec.node_ids()) {
+                    // Merge record, pop this one out
+                    let mbr = merged_bootstrap_records.remove(mbi);
+                    bsrec.merge(mbr);
+                } else {
+                    // No overlap, go to next record
+                    mbi += 1;
+                }
+            }
+            // Append merged record
+            merged_bootstrap_records.push(bsrec);
+        }
+
+        Ok(merged_bootstrap_records)
+    }
+}

veilid-core/src/network_manager/bootstrap/txt_bootstrap/v1.rs

@@ -0,0 +1,62 @@
+use super::*;
+
+pub const BOOTSTRAP_TXT_VERSION_1: u8 = 1;
+
+impl_veilid_log_facility!("net");
+
+impl NetworkManager {
+    /// Parse v1 bootstrap record strings into BootstrapRecord structs
+    #[instrument(level = "trace", skip(self), ret, err)]
+    pub fn parse_bootstrap_v1(
+        &self,
+        record_strings: &[String],
+    ) -> EyreResult<Vec<BootstrapRecord>> {
+        veilid_log!(self debug "Parsing v1 bootstraps: {:?}", record_strings);
+
+        let signing_keys = self
+            .config()
+            .with(|c| c.network.routing_table.bootstrap_keys.clone());
+        if signing_keys.is_empty() {
+            veilid_log!(self warn "No signing keys in config. Proceeding with UNVERIFIED bootstrap.");
+        }
+
+        // For each record string resolve into BootstrapRecord pairs
+        let dial_info_converter = BootstrapDialInfoConverter::default();
+
+        let mut bootstrap_records: Vec<BootstrapRecord> = Vec::new();
+        for record_string in record_strings {
+            let Some(bootstrap_record) = BootstrapRecord::new_from_v1_str(
+                self,
+                &dial_info_converter,
+                record_string,
+                &signing_keys,
+            )?
+            else {
+                continue;
+            };
+
+            bootstrap_records.push(bootstrap_record);
+        }
+
+        let mut merged_bootstrap_records: Vec<BootstrapRecord> = Vec::new();
+
+        for mut bsrec in bootstrap_records {
+            let mut mbi = 0;
+            while mbi < merged_bootstrap_records.len() {
+                let mbr = &mut merged_bootstrap_records[mbi];
+                if mbr.node_ids().contains_any(bsrec.node_ids()) {
+                    // Merge record, pop this one out
+                    let mbr = merged_bootstrap_records.remove(mbi);
+                    bsrec.merge(mbr);
+                } else {
+                    // No overlap, go to next record
+                    mbi += 1;
+                }
+            }
+            // Append merged record
+            merged_bootstrap_records.push(bsrec);
+        }
+
+        Ok(merged_bootstrap_records)
+    }
+}

veilid-core/src/network_manager/connection_manager.rs

@@ -480,10 +480,12 @@ impl ConnectionManager {
                 }
                 Err(e) => {
                     if retry_count == 0 {
-                        return Err(e).wrap_err(format!(
-                            "failed to connect: {:?} -> {:?}",
-                            preferred_local_address, dial_info
-                        ));
+                        bail!(
+                            "failed to connect: {:?} -> {:?}: {:#?}",
+                            preferred_local_address,
+                            dial_info,
+                            e
+                        );
                     }
                 }
             };

veilid-core/src/network_manager/direct_boot.rs

@@ -1,64 +0,0 @@
-use super::*;
-
-impl_veilid_log_facility!("net");
-
-impl NetworkManager {
-    // Direct bootstrap request handler (separate fallback mechanism from cheaper TXT bootstrap mechanism)
-    #[instrument(level = "trace", target = "net", skip(self), ret, err)]
-    pub async fn handle_boot_request(&self, flow: Flow) -> EyreResult<NetworkResult<()>> {
-        let routing_table = self.routing_table();
-
-        // Get a bunch of nodes with the various
-        let bootstrap_nodes = routing_table.find_bootstrap_nodes_filtered(2);
-
-        // Serialize out peer info
-        let bootstrap_peerinfo: Vec<Arc<PeerInfo>> = bootstrap_nodes
-            .iter()
-            .filter_map(|nr| nr.get_peer_info(RoutingDomain::PublicInternet))
-            .collect();
-        let json_bytes = serialize_json(bootstrap_peerinfo).as_bytes().to_vec();
-
-        veilid_log!(self trace "BOOT reponse: {}", String::from_utf8_lossy(&json_bytes));
-
-        // Reply with a chunk of signed routing table
-        let net = self.net();
-        match pin_future_closure!(net.send_data_to_existing_flow(flow, json_bytes)).await? {
-            SendDataToExistingFlowResult::Sent(_) => {
-                // Bootstrap reply was sent
-                Ok(NetworkResult::value(()))
-            }
-            SendDataToExistingFlowResult::NotSent(_) => Ok(NetworkResult::no_connection_other(
-                "bootstrap reply could not be sent",
-            )),
-        }
-    }
-
-    // Direct bootstrap request
-    #[instrument(level = "trace", target = "net", err, skip(self))]
-    pub async fn boot_request(&self, dial_info: DialInfo) -> EyreResult<Vec<Arc<PeerInfo>>> {
-        let timeout_ms = self.config().with(|c| c.network.rpc.timeout_ms);
-        // Send boot magic to requested peer address
-        let data = BOOT_MAGIC.to_vec();
-
-        let out_data: Vec<u8> = network_result_value_or_log!(self self
-            .net()
-            .send_recv_data_unbound_to_dial_info(dial_info, data, timeout_ms)
-            .await? => [ format!(": dial_info={}, data.len={}", dial_info, data.len()) ]
-        {
-            return Ok(Vec::new());
-        });
-
-        let bootstrap_peerinfo_str =
-            std::str::from_utf8(&out_data).wrap_err("bad utf8 in boot peerinfo")?;
-
-        let bootstrap_peerinfo: Vec<PeerInfo> = match deserialize_json(bootstrap_peerinfo_str) {
-            Ok(v) => v,
-            Err(e) => {
-                error!("{}", e);
-                return Err(e).wrap_err("failed to deserialize peerinfo");
-            }
-        };
-
-        Ok(bootstrap_peerinfo.into_iter().map(Arc::new).collect())
-    }
-}

veilid-core/src/network_manager/mod.rs

@@ -7,11 +7,11 @@ mod wasm;
 
 mod address_check;
 mod address_filter;
+mod bootstrap;
 mod connection_handle;
 mod connection_manager;
 mod connection_table;
 mod debug;
-mod direct_boot;
 mod network_connection;
 mod node_contact_method_cache;
 mod receipt_manager;
@@ -31,6 +31,7 @@ pub use network_connection::*;
 pub use receipt_manager::*;
 pub use stats::*;
 
+pub(crate) use bootstrap::*;
 pub(crate) use node_contact_method_cache::*;
 pub(crate) use types::*;
 
@@ -57,17 +58,29 @@ pub use wasm::{/* LOCAL_NETWORK_CAPABILITIES, */ MAX_CAPABILITIES, PUBLIC_INTERN
 
 impl_veilid_log_facility!("net");
 
-pub const MAX_MESSAGE_SIZE: usize = MAX_ENVELOPE_SIZE;
-pub const IPADDR_TABLE_SIZE: usize = 1024;
-pub const IPADDR_MAX_INACTIVE_DURATION_US: TimestampDuration =
-    TimestampDuration::new(300_000_000u64); // 5 minutes
-pub const ADDRESS_FILTER_TASK_INTERVAL_SECS: u32 = 60;
+/// Bootstrap v0 FOURCC
 pub const BOOT_MAGIC: &[u8; 4] = b"BOOT";
+/// Bootstrap v1 FOURCC
+pub const B01T_MAGIC: &[u8; 4] = b"B01T";
+/// Cache size for TXT lookups used by bootstrap
+pub const TXT_LOOKUP_CACHE_SIZE: usize = 256;
+/// Duration that TXT lookups are valid in the cache (5 minutes, <= the DNS record expiration timeout)
+pub const TXT_LOOKUP_EXPIRATION: TimestampDuration = TimestampDuration::new_secs(300);
+/// Maximum size for a message is the same as the maximum size for an Envelope
+pub const MAX_MESSAGE_SIZE: usize = MAX_ENVELOPE_SIZE;
+/// Statistics table size for tracking performance by IP address
+pub const IPADDR_TABLE_SIZE: usize = 1024;
+/// Eviction time for ip addresses from statistics tables (5 minutes)
+pub const IPADDR_MAX_INACTIVE_DURATION: TimestampDuration = TimestampDuration::new_secs(300);
+/// How frequently to process adddress filter background tasks
+pub const ADDRESS_FILTER_TASK_INTERVAL_SECS: u32 = 60;
+/// Delay between hole punch operations to improve likelihood of seqential state processing
 pub const HOLE_PUNCH_DELAY_MS: u32 = 100;
+/// Number of rpc relay operations that can be handles simultaneously
 pub const RELAY_WORKERS_PER_CORE: u32 = 16;
 
-// Things we get when we start up and go away when we shut down
-// Routing table is not in here because we want it to survive a network shutdown/startup restart
+/// Things we get when we start up and go away when we shut down
+/// Routing table is not in here because we want it to survive a network shutdown/startup restart
 #[derive(Clone)]
 struct NetworkComponents {
     net: Network,
@@ -186,6 +199,9 @@ struct NetworkManagerInner {
     peer_info_change_subscription: Option<EventBusSubscription>,
     socket_address_change_subscription: Option<EventBusSubscription>,
 
+    // TXT lookup cache
+    txt_lookup_cache: LruCache<String, (Timestamp, Vec<String>)>,
+
     // Relay workers
     relay_stop_source: Option<StopSource>,
     relay_send_channel: Option<flume::Sender<RelayWorkerRequest>>,
@@ -242,6 +258,7 @@ impl NetworkManager {
             address_check: None,
             peer_info_change_subscription: None,
             socket_address_change_subscription: None,
+            txt_lookup_cache: LruCache::new(TXT_LOOKUP_CACHE_SIZE),
             //
             relay_send_channel: None,
             relay_stop_source: None,
@@ -1007,7 +1024,11 @@ impl NetworkManager {
 
         // Is this a direct bootstrap request instead of an envelope?
         if data[0..4] == *BOOT_MAGIC {
-            network_result_value_or_log!(self pin_future!(self.handle_boot_request(flow)).await? => [ format!(": flow={:?}", flow) ] {});
+            network_result_value_or_log!(self pin_future!(self.handle_boot_v0_request(flow)).await? => [ format!(": v0 flow={:?}", flow) ] {});
+            return Ok(true);
+        }
+        if data[0..4] == *B01T_MAGIC {
+            network_result_value_or_log!(self pin_future!(self.handle_boot_v1_request(flow)).await? => [ format!(": v1 flow={:?}", flow) ] {});
             return Ok(true);
         }
 

veilid-core/src/network_manager/native/tasks/mod.rs

@@ -41,9 +41,10 @@ impl Network {
 
     // Determine if we need to check for public dialinfo
     fn wants_update_network_class_tick(&self) -> bool {
-        let public_internet_network_class = self
-            .routing_table()
-            .get_network_class(RoutingDomain::PublicInternet);
+        let routing_table = self.routing_table();
+
+        let public_internet_network_class =
+            routing_table.get_network_class(RoutingDomain::PublicInternet);
 
         let needs_update_network_class = self.needs_update_network_class();
         if needs_update_network_class
@@ -51,18 +52,17 @@ impl Network {
             || (public_internet_network_class == NetworkClass::OutboundOnly
                 && self.inner.lock().next_outbound_only_dial_info_check <= Timestamp::now())
         {
-            let routing_table = self.routing_table();
-            let rth = routing_table.get_routing_table_health();
+            let live_entry_counts = routing_table.cached_live_entry_counts();
 
-            // We want at least two live entries per crypto kind before we start doing this (bootstrap)
+            // Bootstrap needs to have gotten us connectivity nodes
             let mut has_at_least_two = true;
             for ck in VALID_CRYPTO_KINDS {
-                if rth
-                    .live_entry_counts
+                if live_entry_counts
+                    .connectivity_capabilities
                     .get(&(RoutingDomain::PublicInternet, ck))
                     .copied()
                     .unwrap_or_default()
-                    < 2
+                    < MIN_BOOTSTRAP_CONNECTIVITY_PEERS
                 {
                     has_at_least_two = false;
                     break;

veilid-core/src/network_manager/tasks/rolling_transfers.rs

@@ -30,7 +30,7 @@ impl NetworkManager {
                 );
 
                 // While we're here, lets see if this address has timed out
-                if cur_ts.saturating_sub(stats.last_seen_ts) >= IPADDR_MAX_INACTIVE_DURATION_US {
+                if cur_ts.saturating_sub(stats.last_seen_ts) >= IPADDR_MAX_INACTIVE_DURATION {
                     // it's dead, put it in the dead list
                     dead_addrs.insert(*addr);
                 }

veilid-core/src/network_manager/tests/mod.rs

@@ -1,3 +1,4 @@
+pub mod test_bootstrap;
 pub mod test_connection_table;
 pub mod test_signed_node_info;
 

veilid-core/src/network_manager/tests/test_bootstrap.rs

@@ -0,0 +1,93 @@
+use super::*;
+use crate::tests::mock_registry;
+
+fn make_mock_bootstrap_record(include_timestamp: bool) -> BootstrapRecord {
+    let mut node_ids = CryptoTypedGroup::new();
+    node_ids.add(
+        TypedKey::from_str("VLD0:f8G4Ckr1UR8YXnmAllwfvBEvXGgfYicZllb7jEpJeSU")
+            .expect("should parse key"),
+    );
+    let envelope_support = VALID_ENVELOPE_VERSIONS.to_vec();
+    let dial_info_details = vec![
+        DialInfoDetail {
+            class: DialInfoClass::Direct,
+            dial_info: DialInfo::try_ws(
+                SocketAddress::new(Address::IPV4(Ipv4Addr::UNSPECIFIED), 5150),
+                "ws://example.com:5150/ws".to_owned(),
+            )
+            .expect("should make ws dialinfo"),
+        },
+        DialInfoDetail {
+            class: DialInfoClass::Direct,
+            dial_info: DialInfo::try_wss(
+                SocketAddress::new(Address::IPV4(Ipv4Addr::UNSPECIFIED), 5150),
+                "wss://example.com:5150/wss".to_owned(),
+            )
+            .expect("should make wss dialinfo"),
+        },
+    ];
+    let opt_timestamp = if include_timestamp {
+        Some(Timestamp::now().as_u64() / 1_000_000u64)
+    } else {
+        None
+    };
+    BootstrapRecord::new(
+        node_ids,
+        envelope_support,
+        dial_info_details,
+        opt_timestamp,
+        vec![],
+    )
+}
+
+pub async fn test_bootstrap_v0() {
+    let registry = mock_registry::init("").await;
+    let network_manager = registry.network_manager();
+    let dial_info_converter = MockDialInfoConverter::default();
+
+    let bsrec = make_mock_bootstrap_record(false);
+    let v0str = bsrec
+        .to_v0_string(&dial_info_converter)
+        .await
+        .expect("should make string");
+    let bsrec2 = BootstrapRecord::new_from_v0_str(&network_manager, &dial_info_converter, &v0str)
+        .expect("should parse string")
+        .expect("should be valid record");
+    assert_eq!(bsrec, bsrec2);
+
+    mock_registry::terminate(registry).await;
+}
+
+pub async fn test_bootstrap_v1() {
+    let registry = mock_registry::init("").await;
+    let network_manager = registry.network_manager();
+    let dial_info_converter = MockDialInfoConverter::default();
+
+    let bsrec = make_mock_bootstrap_record(true);
+    let signing_key_pairs = [TypedKeyPair::from_str("VLD0:W7ENB-SUWpPA7usY8ORVQf_si5QmFbD1Uqa89Jg2Uc0:hbdjau5sr3rBNwN68XeWLg3rfXnXLaLqfbbqhELqV1E").expect("should parse keypair"),
+        TypedKeyPair::from_str("VLD0:v6XPfyOoCP_ZP-CWFNrf_pF_dpxsq74p2LW_Q5Q4yPQ:n-DhHtOU7KWQkdp5to8cpBa_u0RFt2IDZzXPqMTq4O0").expect("should parse keypair")];
+    let signing_keys = signing_key_pairs
+        .iter()
+        .map(|skp| TypedKey::new(skp.kind, skp.value.key))
+        .collect::<Vec<_>>();
+    let v1str = bsrec
+        .to_v1_string(&network_manager, &dial_info_converter, signing_key_pairs[0])
+        .await
+        .expect("should make string");
+    let bsrec2 = BootstrapRecord::new_from_v1_str(
+        &network_manager,
+        &dial_info_converter,
+        &v1str,
+        &signing_keys,
+    )
+    .expect("should parse string")
+    .expect("should be valid record");
+    assert_eq!(bsrec, bsrec2);
+
+    mock_registry::terminate(registry).await;
+}
+
+pub async fn test_all() {
+    test_bootstrap_v0().await;
+    test_bootstrap_v1().await;
+}

veilid-core/src/network_manager/types/dial_info/mod.rs

@@ -315,179 +315,6 @@ impl DialInfo {
         }
     }
 
-    pub fn try_vec_from_short<S: AsRef<str>, H: AsRef<str>>(
-        short: S,
-        hostname: H,
-    ) -> VeilidAPIResult<Vec<Self>> {
-        let short = short.as_ref();
-        let hostname = hostname.as_ref();
-
-        if short.len() < 2 {
-            apibail_parse_error!("invalid short url length", short);
-        }
-        let url = match &short[0..1] {
-            "U" => {
-                format!("udp://{}:{}", hostname, &short[1..])
-            }
-            "T" => {
-                format!("tcp://{}:{}", hostname, &short[1..])
-            }
-            "W" => {
-                format!("ws://{}:{}", hostname, &short[1..])
-            }
-            "S" => {
-                format!("wss://{}:{}", hostname, &short[1..])
-            }
-            _ => {
-                apibail_parse_error!("invalid short url type", short);
-            }
-        };
-        Self::try_vec_from_url(url)
-    }
-
-    pub fn try_vec_from_url<S: AsRef<str>>(url: S) -> VeilidAPIResult<Vec<Self>> {
-        let url = url.as_ref();
-        let split_url = SplitUrl::from_str(url)
-            .map_err(|e| VeilidAPIError::parse_error(format!("unable to split url: {}", e), url))?;
-
-        let port = match split_url.scheme.as_str() {
-            "udp" | "tcp" => split_url
-                .port
-                .ok_or_else(|| VeilidAPIError::parse_error("Missing port in udp url", url))?,
-            "ws" => split_url.port.unwrap_or(80u16),
-            "wss" => split_url.port.unwrap_or(443u16),
-            _ => {
-                apibail_parse_error!("Invalid dial info url scheme", split_url.scheme);
-            }
-        };
-
-        let socket_addrs = {
-            // Resolve if possible, WASM doesn't support resolution and doesn't need it to connect to the dialinfo
-            // This will not be used on signed dialinfo, only for bootstrapping, so we don't need to worry about
-            // the '0.0.0.0' address being propagated across the routing table
-            cfg_if::cfg_if! {
-                if #[cfg(all(target_arch = "wasm32", target_os = "unknown"))] {
-                    vec![SocketAddr::new(IpAddr::V4(Ipv4Addr::new(0,0,0,0)), port)]
-                } else {
-                    match split_url.host {
-                        SplitUrlHost::Hostname(_) => split_url
-                            .host_port(port)
-                            .to_socket_addrs()
-                            .map_err(|_| VeilidAPIError::parse_error("couldn't resolve hostname in url", url))?
-                            .collect(),
-                        SplitUrlHost::IpAddr(a) => vec![SocketAddr::new(a, port)],
-                    }
-                }
-            }
-        };
-
-        let mut out = Vec::new();
-        for sa in socket_addrs {
-            out.push(match split_url.scheme.as_str() {
-                "udp" => Self::udp_from_socketaddr(sa),
-                "tcp" => Self::tcp_from_socketaddr(sa),
-                "ws" => Self::try_ws(
-                    SocketAddress::from_socket_addr(sa).canonical(),
-                    url.to_string(),
-                )?,
-                "wss" => Self::try_wss(
-                    SocketAddress::from_socket_addr(sa).canonical(),
-                    url.to_string(),
-                )?,
-                _ => {
-                    unreachable!("Invalid dial info url scheme")
-                }
-            });
-        }
-        Ok(out)
-    }
-
-    pub async fn to_short(&self) -> (String, String) {
-        match self {
-            DialInfo::UDP(di) => (
-                format!("U{}", di.socket_address.port()),
-                intf::ptr_lookup(di.socket_address.ip_addr())
-                    .await
-                    .unwrap_or_else(|_| di.socket_address.to_string()),
-            ),
-            DialInfo::TCP(di) => (
-                format!("T{}", di.socket_address.port()),
-                intf::ptr_lookup(di.socket_address.ip_addr())
-                    .await
-                    .unwrap_or_else(|_| di.socket_address.to_string()),
-            ),
-            DialInfo::WS(di) => {
-                let mut split_url = SplitUrl::from_str(&format!("ws://{}", di.request)).unwrap();
-                if let SplitUrlHost::IpAddr(a) = split_url.host {
-                    if let Ok(host) = intf::ptr_lookup(a).await {
-                        split_url.host = SplitUrlHost::Hostname(host);
-                    }
-                }
-                (
-                    format!(
-                        "W{}{}",
-                        split_url.port.unwrap_or(80),
-                        split_url
-                            .path
-                            .map(|p| format!("/{}", p))
-                            .unwrap_or_default()
-                    ),
-                    split_url.host.to_string(),
-                )
-            }
-            DialInfo::WSS(di) => {
-                let mut split_url = SplitUrl::from_str(&format!("wss://{}", di.request)).unwrap();
-                if let SplitUrlHost::IpAddr(a) = split_url.host {
-                    if let Ok(host) = intf::ptr_lookup(a).await {
-                        split_url.host = SplitUrlHost::Hostname(host);
-                    }
-                }
-                (
-                    format!(
-                        "S{}{}",
-                        split_url.port.unwrap_or(443),
-                        split_url
-                            .path
-                            .map(|p| format!("/{}", p))
-                            .unwrap_or_default()
-                    ),
-                    split_url.host.to_string(),
-                )
-            }
-        }
-    }
-    #[expect(dead_code)]
-    pub async fn to_url(&self) -> String {
-        match self {
-            DialInfo::UDP(di) => intf::ptr_lookup(di.socket_address.ip_addr())
-                .await
-                .map(|h| format!("udp://{}:{}", h, di.socket_address.port()))
-                .unwrap_or_else(|_| format!("udp://{}", di.socket_address)),
-            DialInfo::TCP(di) => intf::ptr_lookup(di.socket_address.ip_addr())
-                .await
-                .map(|h| format!("tcp://{}:{}", h, di.socket_address.port()))
-                .unwrap_or_else(|_| format!("tcp://{}", di.socket_address)),
-            DialInfo::WS(di) => {
-                let mut split_url = SplitUrl::from_str(&format!("ws://{}", di.request)).unwrap();
-                if let SplitUrlHost::IpAddr(a) = split_url.host {
-                    if let Ok(host) = intf::ptr_lookup(a).await {
-                        split_url.host = SplitUrlHost::Hostname(host);
-                    }
-                }
-                split_url.to_string()
-            }
-            DialInfo::WSS(di) => {
-                let mut split_url = SplitUrl::from_str(&format!("wss://{}", di.request)).unwrap();
-                if let SplitUrlHost::IpAddr(a) = split_url.host {
-                    if let Ok(host) = intf::ptr_lookup(a).await {
-                        split_url.host = SplitUrlHost::Hostname(host);
-                    }
-                }
-                split_url.to_string()
-            }
-        }
-    }
-
     pub fn ordered_sequencing_sort(a: &DialInfo, b: &DialInfo) -> core::cmp::Ordering {
         let s = ProtocolType::ordered_sequencing_sort(a.protocol_type(), b.protocol_type());
         if s != core::cmp::Ordering::Equal {

veilid-core/src/routing_table/debug.rs

@@ -1,59 +1,6 @@
 use super::*;
-use routing_table::tasks::bootstrap::BOOTSTRAP_TXT_VERSION_0;
 
 impl RoutingTable {
-    pub async fn debug_info_txtrecord(&self) -> String {
-        let mut out = String::new();
-
-        let gdis = self.dial_info_details(RoutingDomain::PublicInternet);
-        if gdis.is_empty() {
-            out += "No TXT Record\n";
-        } else {
-            let mut short_urls = Vec::new();
-            let mut some_hostname = Option::<String>::None;
-            for gdi in gdis {
-                let (short_url, hostname) = gdi.dial_info.to_short().await;
-                if let Some(h) = &some_hostname {
-                    if h != &hostname {
-                        return format!(
-                            "Inconsistent hostnames for dial info: {} vs {}",
-                            some_hostname.unwrap(),
-                            hostname
-                        );
-                    }
-                } else {
-                    some_hostname = Some(hostname);
-                }
-
-                short_urls.push(short_url);
-            }
-            if some_hostname.is_none() || short_urls.is_empty() {
-                return "No dial info for bootstrap host".to_owned();
-            }
-            short_urls.sort();
-            short_urls.dedup();
-
-            let valid_envelope_versions = VALID_ENVELOPE_VERSIONS.map(|x| x.to_string()).join(",");
-            let node_ids = self
-                .node_ids()
-                .iter()
-                .map(|x| x.to_string())
-                .collect::<Vec<String>>()
-                .join(",");
-            out += "TXT Record:\n";
-            out += &format!(
-                "{}|{}|{}|{}|",
-                BOOTSTRAP_TXT_VERSION_0,
-                valid_envelope_versions,
-                node_ids,
-                some_hostname.unwrap()
-            );
-            out += &short_urls.join(",");
-            out += "\n";
-        }
-        out
-    }
-
     pub fn debug_info_nodeid(&self) -> String {
         let mut out = String::new();
         for nid in self.node_ids().iter() {
@@ -243,7 +190,7 @@ impl RoutingTable {
 
         out += &format!("Entries: {}\n", inner.bucket_entry_count());
         out += "   Live:\n";
-        for ec in inner.cached_entry_counts() {
+        for ec in inner.cached_live_entry_counts().any_capabilities.iter() {
             let routing_domain = ec.0 .0;
             let crypto_kind = ec.0 .1;
             let count = ec.1;

veilid-core/src/routing_table/mod.rs

@@ -35,8 +35,10 @@ impl_veilid_log_facility!("rtab");
 
 //////////////////////////////////////////////////////////////////////////
 
-/// How many nodes in our routing table we require for a functional PublicInternet RoutingDomain
-pub const MIN_PUBLIC_INTERNET_ROUTING_DOMAIN_NODE_COUNT: usize = 4;
+/// Minimum number of nodes we need, per crypto kind, per routing domain, or we trigger a bootstrap
+pub const MIN_BOOTSTRAP_CONNECTIVITY_PEERS: usize = 4;
+/// Set of routing domains that use the bootstrap mechanism
+pub const BOOTSTRAP_ROUTING_DOMAINS: [RoutingDomain; 1] = [RoutingDomain::PublicInternet];
 
 /// How frequently we tick the relay management routine
 pub const RELAY_MANAGEMENT_INTERVAL_SECS: u32 = 1;
@@ -87,7 +89,7 @@ pub struct RoutingTableHealth {
     /// Number of dead (always unresponsive) entries in the routing table
     pub dead_entry_count: usize,
     /// Number of live (responsive) entries in the routing table per RoutingDomain and CryptoKind
-    pub live_entry_counts: BTreeMap<(RoutingDomain, CryptoKind), usize>,
+    pub live_entry_counts: LiveEntryCounts,
     /// If PublicInternet network class is valid yet
     pub public_internet_ready: bool,
     /// If LocalNetwork network class is valid yet
@@ -599,6 +601,11 @@ impl RoutingTable {
         self.inner.read().get_current_peer_info(routing_domain)
     }
 
+    /// Return a list of the current valid bootstrap peers in a particular routing domain
+    pub fn get_bootstrap_peers(&self, routing_domain: RoutingDomain) -> Vec<NodeRef> {
+        self.inner.read().get_bootstrap_peers(routing_domain)
+    }
+
     /// Return the domain's currently registered network class
     #[cfg_attr(all(target_arch = "wasm32", target_os = "unknown"), expect(dead_code))]
     pub fn get_network_class(&self, routing_domain: RoutingDomain) -> NetworkClass {
@@ -712,6 +719,10 @@ impl RoutingTable {
         self.inner.read().get_routing_table_health()
     }
 
+    pub fn cached_live_entry_counts(&self) -> Arc<LiveEntryCounts> {
+        self.inner.read().cached_live_entry_counts()
+    }
+
     #[instrument(level = "trace", skip_all)]
     pub fn get_recent_peers(&self) -> Vec<(TypedKey, RecentPeersEntry)> {
         let mut recent_peers = Vec::new();
@@ -868,116 +879,6 @@ impl RoutingTable {
         )
     }
 
-    /// Retrieve up to N of each type of protocol capable nodes for a single crypto kind
-    fn find_bootstrap_nodes_filtered_per_crypto_kind(
-        &self,
-        crypto_kind: CryptoKind,
-        max_per_type: usize,
-    ) -> Vec<NodeRef> {
-        let protocol_types = [
-            ProtocolType::UDP,
-            ProtocolType::TCP,
-            ProtocolType::WS,
-            ProtocolType::WSS,
-        ];
-
-        let protocol_types_len = protocol_types.len();
-        let mut nodes_proto_v4 = [0usize, 0usize, 0usize, 0usize];
-        let mut nodes_proto_v6 = [0usize, 0usize, 0usize, 0usize];
-
-        let filter = Box::new(
-            move |rti: &RoutingTableInner, entry: Option<Arc<BucketEntry>>| {
-                let entry = entry.unwrap();
-                entry.with(rti, |_rti, e| {
-                    // skip nodes on our local network here
-                    if e.has_node_info(RoutingDomain::LocalNetwork.into()) {
-                        return false;
-                    }
-
-                    // Ensure crypto kind is supported
-                    if !e.crypto_kinds().contains(&crypto_kind) {
-                        return false;
-                    }
-
-                    // Only nodes with direct publicinternet node info
-                    let Some(signed_node_info) = e.signed_node_info(RoutingDomain::PublicInternet)
-                    else {
-                        return false;
-                    };
-                    let SignedNodeInfo::Direct(signed_direct_node_info) = signed_node_info else {
-                        return false;
-                    };
-                    let node_info = signed_direct_node_info.node_info();
-
-                    // Bootstraps must have -only- inbound capable network class
-                    if !matches!(node_info.network_class(), NetworkClass::InboundCapable) {
-                        return false;
-                    }
-
-                    // Check for direct dialinfo and a good mix of protocol and address types
-                    let mut keep = false;
-                    for did in node_info.dial_info_detail_list() {
-                        // Bootstraps must have -only- direct dial info
-                        if !matches!(did.class, DialInfoClass::Direct) {
-                            return false;
-                        }
-                        if matches!(did.dial_info.address_type(), AddressType::IPV4) {
-                            for (n, protocol_type) in protocol_types.iter().enumerate() {
-                                if nodes_proto_v4[n] < max_per_type
-                                    && did.dial_info.protocol_type() == *protocol_type
-                                {
-                                    nodes_proto_v4[n] += 1;
-                                    keep = true;
-                                }
-                            }
-                        } else if matches!(did.dial_info.address_type(), AddressType::IPV6) {
-                            for (n, protocol_type) in protocol_types.iter().enumerate() {
-                                if nodes_proto_v6[n] < max_per_type
-                                    && did.dial_info.protocol_type() == *protocol_type
-                                {
-                                    nodes_proto_v6[n] += 1;
-                                    keep = true;
-                                }
-                            }
-                        }
-                    }
-                    keep
-                })
-            },
-        ) as RoutingTableEntryFilter;
-
-        let filters = VecDeque::from([filter]);
-
-        self.find_preferred_fastest_nodes(
-            protocol_types_len * 2 * max_per_type,
-            filters,
-            |_rti, entry: Option<Arc<BucketEntry>>| {
-                NodeRef::new(self.registry(), entry.unwrap().clone())
-            },
-        )
-    }
-
-    /// Retrieve up to N of each type of protocol capable nodes for all crypto kinds
-    pub fn find_bootstrap_nodes_filtered(&self, max_per_type: usize) -> Vec<NodeRef> {
-        let mut out =
-            self.find_bootstrap_nodes_filtered_per_crypto_kind(VALID_CRYPTO_KINDS[0], max_per_type);
-
-        // Merge list of nodes so we don't have duplicates
-        for crypto_kind in &VALID_CRYPTO_KINDS[1..] {
-            let nrs =
-                self.find_bootstrap_nodes_filtered_per_crypto_kind(*crypto_kind, max_per_type);
-            'nrloop: for nr in nrs {
-                for nro in &out {
-                    if nro.same_entry(&nr) {
-                        continue 'nrloop;
-                    }
-                }
-                out.push(nr);
-            }
-        }
-        out
-    }
-
     pub fn find_preferred_fastest_nodes<'a, T, O>(
         &self,
         node_count: usize,

veilid-core/src/routing_table/routing_table_inner/mod.rs

@@ -12,6 +12,16 @@ pub const RECENT_PEERS_TABLE_SIZE: usize = 64;
 // Critical sections
 pub const LOCK_TAG_TICK: &str = "TICK";
 
+/// Keeping track of how many entries we have of each type we care about
+#[derive(Debug, Clone, Default, Eq, PartialEq)]
+pub struct LiveEntryCounts {
+    /// A rough count of the entries in the table per routing domain and crypto kind with any capabilities
+    pub any_capabilities: EntryCounts,
+    /// A rough count of the entries in the table per routing domain and crypto kind with CONNECTIVITY_CAPABILITIES
+    pub connectivity_capabilities: EntryCounts,
+    /// A rough count of the entries in the table per routing domain and crypto kind with DISTANCE_METRIC_CAPABILITIES
+    pub distance_metric_capabilities: EntryCounts,
+}
 pub type EntryCounts = BTreeMap<(RoutingDomain, CryptoKind), usize>;
 
 #[derive(Debug)]
@@ -32,8 +42,8 @@ pub struct RoutingTableInner {
     pub(super) buckets: BTreeMap<CryptoKind, Vec<Bucket>>,
     /// A weak set of all the entries we have in the buckets for faster iteration
     pub(super) all_entries: PtrWeakHashSet<Weak<BucketEntry>>,
-    /// A rough count of the entries in the table per routing domain and crypto kind
-    pub(super) live_entry_count: EntryCounts,
+    /// Summary of how many entries we have of capability combinations we care about
+    pub(super) live_entry_counts: Arc<LiveEntryCounts>,
     /// The public internet routing domain
     pub(super) public_internet_routing_domain: PublicInternetRoutingDomainDetail,
     /// The dial info we use on the local network
@@ -67,7 +77,7 @@ impl RoutingTableInner {
             ),
             local_network_routing_domain: LocalNetworkRoutingDomainDetail::new(registry.clone()),
             all_entries: PtrWeakHashSet::new(),
-            live_entry_count: BTreeMap::new(),
+            live_entry_counts: Default::default(),
             self_latency_stats_accounting: LatencyStatsAccounting::new(),
             self_transfer_stats_accounting: TransferStatsAccounting::new(),
             self_transfer_stats: TransferStatsDownUp::default(),
@@ -281,6 +291,11 @@ impl RoutingTableInner {
         self.with_routing_domain(routing_domain, |rdd| rdd.get_peer_info(self))
     }
 
+    /// Return a list of the current valid bootstrap peers in a particular routing domain
+    pub fn get_bootstrap_peers(&self, routing_domain: RoutingDomain) -> Vec<NodeRef> {
+        self.with_routing_domain(routing_domain, |rdd| rdd.get_bootstrap_peers())
+    }
+
     /// Return the domain's currently registered network class
     pub fn get_network_class(&self, routing_domain: RoutingDomain) -> NetworkClass {
         self.with_routing_domain(routing_domain, |rdd| rdd.network_class())
@@ -391,10 +406,11 @@ impl RoutingTableInner {
 
     /// Build the counts of entries per routing domain and crypto kind and cache them
     /// Only considers entries that have valid signed node info
-    pub fn refresh_cached_entry_counts(&mut self) -> EntryCounts {
-        self.live_entry_count.clear();
+    pub fn refresh_cached_live_entry_counts(&mut self) -> Arc<LiveEntryCounts> {
+        let mut live_entry_counts = LiveEntryCounts::default();
+
         let cur_ts = Timestamp::now();
-        self.with_entries_mut(cur_ts, BucketEntryState::Unreliable, |rti, entry| {
+        self.with_entries_mut(cur_ts, BucketEntryState::Unreliable, |_rti, entry| {
             entry.with_inner(|e| {
                 // Tally per routing domain and crypto kind
                 for rd in RoutingDomain::all() {
@@ -403,10 +419,25 @@ impl RoutingTableInner {
                         if sni.has_any_signature() {
                             // Tally
                             for crypto_kind in e.crypto_kinds() {
-                                rti.live_entry_count
+                                live_entry_counts
+                                    .any_capabilities
                                     .entry((rd, crypto_kind))
                                     .and_modify(|x| *x += 1)
                                     .or_insert(1);
+                                if e.has_all_capabilities(rd, CONNECTIVITY_CAPABILITIES) {
+                                    live_entry_counts
+                                        .connectivity_capabilities
+                                        .entry((rd, crypto_kind))
+                                        .and_modify(|x| *x += 1)
+                                        .or_insert(1);
+                                }
+                                if e.has_all_capabilities(rd, DISTANCE_METRIC_CAPABILITIES) {
+                                    live_entry_counts
+                                        .distance_metric_capabilities
+                                        .entry((rd, crypto_kind))
+                                        .and_modify(|x| *x += 1)
+                                        .or_insert(1);
+                                }
                             }
                         }
                     }
@@ -414,13 +445,14 @@ impl RoutingTableInner {
             });
             Option::<()>::None
         });
-        self.live_entry_count.clone()
+        self.live_entry_counts = Arc::new(live_entry_counts);
+        self.live_entry_counts.clone()
     }
 
     /// Return the last cached entry counts
     /// Only considers entries that have valid signed node info
-    pub fn cached_entry_counts(&self) -> EntryCounts {
-        self.live_entry_count.clone()
+    pub fn cached_live_entry_counts(&self) -> Arc<LiveEntryCounts> {
+        self.live_entry_counts.clone()
     }
 
     /// Count entries that match some criteria
@@ -1050,7 +1082,7 @@ impl RoutingTableInner {
             .get_published_peer_info(RoutingDomain::LocalNetwork)
             .is_some();
 
-        let live_entry_counts = self.cached_entry_counts();
+        let live_entry_counts = self.cached_live_entry_counts().as_ref().clone();
 
         RoutingTableHealth {
             reliable_entry_count,

veilid-core/src/routing_table/routing_table_inner/routing_domains/local_network/mod.rs

@@ -105,11 +105,20 @@ impl RoutingDomainDetail for LocalNetworkRoutingDomainDetail {
     fn get_peer_info(&self, rti: &RoutingTableInner) -> Arc<PeerInfo> {
         self.common.get_peer_info(rti)
     }
-
     fn get_published_peer_info(&self) -> Option<Arc<PeerInfo>> {
         (*self.published_peer_info.lock()).clone()
     }
 
+    fn get_bootstrap_peers(&self) -> Vec<NodeRef> {
+        self.common.get_bootstrap_peers()
+    }
+    fn clear_bootstrap_peers(&self) {
+        self.common.clear_bootstrap_peers();
+    }
+    fn add_bootstrap_peer(&self, bootstrap_peer: NodeRef) {
+        self.common.add_bootstrap_peer(bootstrap_peer)
+    }
+
     fn can_contain_address(&self, address: Address) -> bool {
         let ip = address.ip_addr();
         for localnet in &self.local_networks {

veilid-core/src/routing_table/routing_table_inner/routing_domains/mod.rs

@@ -58,6 +58,12 @@ pub trait RoutingDomainDetail {
     fn set_relay_node_last_keepalive(&mut self, ts: Option<Timestamp>);
     // Set last relay optimized time
     fn set_relay_node_last_optimized(&mut self, ts: Option<Timestamp>);
+
+    // Bootstrap peers
+    #[expect(dead_code)]
+    fn get_bootstrap_peers(&self) -> Vec<NodeRef>;
+    fn clear_bootstrap_peers(&self);
+    fn add_bootstrap_peer(&self, bootstrap_peer: NodeRef);
 }
 
 trait RoutingDomainDetailCommonAccessors: RoutingDomainDetail {
@@ -129,6 +135,7 @@ struct RoutingDomainDetailCommon {
     cached_peer_info: Mutex<Option<Arc<PeerInfo>>>,
     relay_node_last_keepalive: Option<Timestamp>,
     relay_node_last_optimized: Option<Timestamp>,
+    bootstrap_peers: Mutex<Vec<NodeRef>>,
 }
 
 impl RoutingDomainDetailCommon {
@@ -145,6 +152,7 @@ impl RoutingDomainDetailCommon {
             cached_peer_info: Mutex::new(Default::default()),
             relay_node_last_keepalive: Default::default(),
             relay_node_last_optimized: Default::default(),
+            bootstrap_peers: Mutex::new(Default::default()),
         }
     }
 
@@ -184,6 +192,19 @@ impl RoutingDomainDetailCommon {
         self.capabilities.clone()
     }
 
+    pub fn get_bootstrap_peers(&self) -> Vec<NodeRef> {
+        self.bootstrap_peers.lock().clone()
+    }
+
+    pub fn clear_bootstrap_peers(&self) {
+        self.bootstrap_peers.lock().clear()
+    }
+
+    pub fn add_bootstrap_peer(&self, bootstrap_peer: NodeRef) {
+        let mut bootstrap_peers = self.bootstrap_peers.lock();
+        bootstrap_peers.push(bootstrap_peer);
+    }
+
     pub fn requires_relay(&self, compatible_address_types: AddressTypeSet) -> Option<RelayKind> {
         match self.network_class() {
             NetworkClass::InboundCapable => {

veilid-core/src/routing_table/routing_table_inner/routing_domains/public_internet/mod.rs

@@ -87,11 +87,20 @@ impl RoutingDomainDetail for PublicInternetRoutingDomainDetail {
     fn get_peer_info(&self, rti: &RoutingTableInner) -> Arc<PeerInfo> {
         self.common.get_peer_info(rti)
     }
-
     fn get_published_peer_info(&self) -> Option<Arc<PeerInfo>> {
         (*self.published_peer_info.lock()).clone()
     }
 
+    fn get_bootstrap_peers(&self) -> Vec<NodeRef> {
+        self.common.get_bootstrap_peers()
+    }
+    fn clear_bootstrap_peers(&self) {
+        self.common.clear_bootstrap_peers();
+    }
+    fn add_bootstrap_peer(&self, bootstrap_peer: NodeRef) {
+        self.common.add_bootstrap_peer(bootstrap_peer)
+    }
+
     ////////////////////////////////////////////////
 
     fn can_contain_address(&self, address: Address) -> bool {
@@ -142,6 +151,7 @@ impl RoutingDomainDetail for PublicInternetRoutingDomainDetail {
             } else {
                 veilid_log!(self debug "[PublicInternet] Unpublishing because current peer info is invalid");
             }
+
             *ppi_lock = opt_new_peer_info.clone();
 
             opt_new_peer_info

veilid-core/src/routing_table/tasks/bootstrap.rs

@@ -5,368 +5,7 @@ use stop_token::future::FutureExt as StopFutureExt;
 
 impl_veilid_log_facility!("rtab");
 
-pub const BOOTSTRAP_TXT_VERSION_0: u8 = 0;
-pub const MIN_BOOTSTRAP_PEERS: usize = 4;
-
-#[derive(Clone, Debug)]
-pub struct BootstrapRecord {
-    node_ids: TypedKeyGroup,
-    envelope_support: Vec<u8>,
-    dial_info_details: Vec<DialInfoDetail>,
-}
-impl BootstrapRecord {
-    pub fn merge(&mut self, other: BootstrapRecord) {
-        self.node_ids.add_all(&other.node_ids);
-        for x in other.envelope_support {
-            if !self.envelope_support.contains(&x) {
-                self.envelope_support.push(x);
-                self.envelope_support.sort();
-            }
-        }
-        for did in other.dial_info_details {
-            if !self.dial_info_details.contains(&did) {
-                self.dial_info_details.push(did);
-            }
-        }
-    }
-}
-
 impl RoutingTable {
-    /// Process bootstrap version 0
-    fn process_bootstrap_records_v0(
-        &self,
-        records: Vec<String>,
-    ) -> EyreResult<Option<BootstrapRecord>> {
-        // Bootstrap TXT Record Format Version 0:
-        // txt_version|envelope_support|node_ids|hostname|dialinfoshort*
-        //
-        // Split bootstrap node record by '|' and then lists by ','. Example:
-        // 0|0|VLD0:7lxDEabK_qgjbe38RtBa3IZLrud84P6NhGP-pRTZzdQ|bootstrap-1.dev.veilid.net|T5150,U5150,W5150/ws
-
-        if records.len() != 5 {
-            bail!("invalid number of fields in bootstrap v0 txt record");
-        }
-
-        // Envelope support
-        let mut envelope_support = Vec::new();
-        for ess in records[1].split(',') {
-            let ess = ess.trim();
-            let es = match ess.parse::<u8>() {
-                Ok(v) => v,
-                Err(e) => {
-                    bail!(
-                        "invalid envelope version specified in bootstrap node txt record: {}",
-                        e
-                    );
-                }
-            };
-            envelope_support.push(es);
-        }
-        envelope_support.sort();
-        envelope_support.dedup();
-
-        // Node Id
-        let mut node_ids = TypedKeyGroup::new();
-        for node_id_str in records[2].split(',') {
-            let node_id_str = node_id_str.trim();
-            let node_id = match TypedKey::from_str(node_id_str) {
-                Ok(v) => v,
-                Err(e) => {
-                    bail!(
-                        "Invalid node id in bootstrap node record {}: {}",
-                        node_id_str,
-                        e
-                    );
-                }
-            };
-            node_ids.add(node_id);
-        }
-
-        // If this is our own node id, then we skip it for bootstrap, in case we are a bootstrap node
-        if self.matches_own_node_id(&node_ids) {
-            return Ok(None);
-        }
-
-        // Hostname
-        let hostname_str = records[3].trim();
-
-        // Resolve each record and store in node dial infos list
-        let mut dial_info_details = Vec::new();
-        for rec in records[4].split(',') {
-            let rec = rec.trim();
-            let dial_infos = match DialInfo::try_vec_from_short(rec, hostname_str) {
-                Ok(dis) => dis,
-                Err(e) => {
-                    veilid_log!(self warn "Couldn't resolve bootstrap node dial info {}: {}", rec, e);
-                    continue;
-                }
-            };
-
-            for di in dial_infos {
-                dial_info_details.push(DialInfoDetail {
-                    dial_info: di,
-                    class: DialInfoClass::Direct,
-                });
-            }
-        }
-
-        Ok(Some(BootstrapRecord {
-            node_ids,
-            envelope_support,
-            dial_info_details,
-        }))
-    }
-
-    // Bootstrap lookup process
-    #[instrument(level = "trace", skip(self), ret, err)]
-    pub async fn resolve_bootstrap(
-        &self,
-        bootstrap: Vec<String>,
-    ) -> EyreResult<Vec<BootstrapRecord>> {
-        // Resolve from bootstrap root to bootstrap hostnames
-        let mut bsnames = Vec::<String>::new();
-        for bh in bootstrap {
-            // Get TXT record for bootstrap (bootstrap.veilid.net, or similar)
-            let records = match intf::txt_lookup(&bh).await {
-                Ok(v) => v,
-                Err(e) => {
-                    veilid_log!(self warn
-                        "Network may be down. No bootstrap resolution for '{}': {}",
-                        bh, e
-                    );
-                    continue;
-                }
-            };
-            for record in records {
-                // Split the bootstrap name record by commas
-                for rec in record.split(',') {
-                    let rec = rec.trim();
-                    // If the name specified is fully qualified, go with it
-                    let bsname = if rec.ends_with('.') {
-                        rec.to_string()
-                    }
-                    // If the name is not fully qualified, prepend it to the bootstrap name
-                    else {
-                        format!("{}.{}", rec, bh)
-                    };
-
-                    // Add to the list of bootstrap name to look up
-                    bsnames.push(bsname);
-                }
-            }
-        }
-
-        // Get bootstrap nodes from hostnames concurrently
-        let mut unord = FuturesUnordered::new();
-        for bsname in bsnames {
-            unord.push(
-                async move {
-                    // look up bootstrap node txt records
-                    let bsnirecords = match intf::txt_lookup(&bsname).await {
-                        Err(e) => {
-                            veilid_log!(self warn
-                                "Network may be down. Bootstrap node txt lookup failed for {}: {}",
-                                bsname, e
-                            );
-                            return None;
-                        }
-                        Ok(v) => v,
-                    };
-                    // for each record resolve into key/bootstraprecord pairs
-                    let mut bootstrap_records: Vec<BootstrapRecord> = Vec::new();
-                    for bsnirecord in bsnirecords {
-                        // All formats split on '|' character
-                        let records: Vec<String> = bsnirecord
-                            .trim()
-                            .split('|')
-                            .map(|x| x.trim().to_owned())
-                            .collect();
-
-                        // Bootstrap TXT record version
-                        let txt_version: u8 = match records[0].parse::<u8>() {
-                            Ok(v) => v,
-                            Err(e) => {
-                                veilid_log!(self warn
-                                "invalid txt_version specified in bootstrap node txt record: {}",
-                                e
-                            );
-                                continue;
-                            }
-                        };
-                        let bootstrap_record = match txt_version {
-                            BOOTSTRAP_TXT_VERSION_0 => {
-                                match self.process_bootstrap_records_v0(records) {
-                                    Err(e) => {
-                                        veilid_log!(self error
-                                            "couldn't process v0 bootstrap records from {}: {}",
-                                            bsname, e
-                                        );
-                                        continue;
-                                    }
-                                    Ok(Some(v)) => v,
-                                    Ok(None) => {
-                                        // skipping
-                                        continue;
-                                    }
-                                }
-                            }
-                            _ => {
-                                veilid_log!(self warn "unsupported bootstrap txt record version");
-                                continue;
-                            }
-                        };
-
-                        bootstrap_records.push(bootstrap_record);
-                    }
-                    Some(bootstrap_records)
-                }
-                .instrument(Span::current()),
-            );
-        }
-
-        let mut merged_bootstrap_records: Vec<BootstrapRecord> = Vec::new();
-        while let Some(bootstrap_records) = unord.next().await {
-            let Some(bootstrap_records) = bootstrap_records else {
-                continue;
-            };
-            for mut bsrec in bootstrap_records {
-                let mut mbi = 0;
-                while mbi < merged_bootstrap_records.len() {
-                    let mbr = &mut merged_bootstrap_records[mbi];
-                    if mbr.node_ids.contains_any(&bsrec.node_ids) {
-                        // Merge record, pop this one out
-                        let mbr = merged_bootstrap_records.remove(mbi);
-                        bsrec.merge(mbr);
-                    } else {
-                        // No overlap, go to next record
-                        mbi += 1;
-                    }
-                }
-                // Append merged record
-                merged_bootstrap_records.push(bsrec);
-            }
-        }
-
-        // ensure dial infos are sorted
-        for mbr in &mut merged_bootstrap_records {
-            mbr.dial_info_details.sort();
-        }
-
-        Ok(merged_bootstrap_records)
-    }
-
-    //#[instrument(level = "trace", skip(self), err)]
-    pub fn bootstrap_with_peer(
-        &self,
-        crypto_kinds: Vec<CryptoKind>,
-        pi: Arc<PeerInfo>,
-        unord: &FuturesUnordered<PinBoxFutureStatic<()>>,
-    ) {
-        veilid_log!(self trace
-            "--- bootstrapping {} with {:?}",
-            pi.node_ids(),
-            pi.signed_node_info().node_info().dial_info_detail_list()
-        );
-
-        let routing_domain = pi.routing_domain();
-
-        let nr = match self.register_node_with_peer_info(pi, true) {
-            Ok(nr) => nr,
-            Err(e) => {
-                veilid_log!(self error "failed to register bootstrap peer info: {}", e);
-                return;
-            }
-        };
-
-        // Add this our futures to process in parallel
-        for crypto_kind in crypto_kinds {
-            // Bootstrap this crypto kind
-            let nr = nr.unfiltered();
-            unord.push(Box::pin(
-                async move {
-                    let network_manager = nr.network_manager();
-                    let routing_table = nr.routing_table();
-
-                    // Get what contact method would be used for contacting the bootstrap
-                    let bsdi = match network_manager
-                        .get_node_contact_method(nr.sequencing_filtered(Sequencing::PreferOrdered))
-                    {
-                        Ok(Some(ncm)) if ncm.is_direct() => ncm.direct_dial_info().unwrap(),
-                        Ok(v) => {
-                            veilid_log!(nr debug "invalid contact method for bootstrap, ignoring peer: {:?}", v);
-                            // let _ =
-                            //     network_manager
-                            //     .get_node_contact_method(nr.clone());
-                            return;
-                        }
-                        Err(e) => {
-                            veilid_log!(nr warn "unable to bootstrap: {}", e);
-                            return;
-                        }
-                    };
-
-                    // Need VALID signed peer info, so ask bootstrap to find_node of itself
-                    // which will ensure it has the bootstrap's signed peer info as part of the response
-                    let _ = routing_table.find_nodes_close_to_node_ref(crypto_kind, nr.sequencing_filtered(Sequencing::PreferOrdered), vec![]).await;
-
-                    // Ensure we got the signed peer info
-                    if !nr.signed_node_info_has_valid_signature(routing_domain) {
-                        veilid_log!(nr warn "bootstrap server is not responding");
-                        veilid_log!(nr debug "bootstrap server is not responding for dialinfo: {}", bsdi);
-
-                        // Try a different dialinfo next time
-                        network_manager.address_filter().set_dial_info_failed(bsdi);
-                    } else {
-                        veilid_log!(nr info "bootstrap of {} successful via {}", crypto_kind, nr);
-
-                        // otherwise this bootstrap is valid, lets ask it to find ourselves now
-                        routing_table.reverse_find_node(crypto_kind, nr, true, vec![]).await
-                    }
-                }
-                .instrument(Span::current()),
-            ));
-        }
-    }
-
-    #[instrument(level = "trace", skip(self), err)]
-    pub async fn bootstrap_with_peer_list(
-        &self,
-        peers: Vec<Arc<PeerInfo>>,
-        stop_token: StopToken,
-    ) -> EyreResult<()> {
-        veilid_log!(self debug "  bootstrap peers: {:?}", &peers);
-
-        // Get crypto kinds to bootstrap
-        let crypto_kinds = self.get_bootstrap_crypto_kinds();
-
-        veilid_log!(self debug "  bootstrap crypto kinds: {:?}", &crypto_kinds);
-
-        // Run all bootstrap operations concurrently
-        let mut unord = FuturesUnordered::<PinBoxFutureStatic<()>>::new();
-        for peer in peers {
-            self.bootstrap_with_peer(crypto_kinds.clone(), peer, &unord);
-        }
-
-        // Wait for all bootstrap operations to complete before we complete the singlefuture
-        while let Ok(Some(_)) = unord.next().timeout_at(stop_token.clone()).await {}
-        Ok(())
-    }
-
-    // Get counts by crypto kind and figure out which crypto kinds need bootstrapping
-    fn get_bootstrap_crypto_kinds(&self) -> Vec<CryptoKind> {
-        let entry_count = self.inner.read().cached_entry_counts();
-        let mut crypto_kinds = Vec::new();
-        for crypto_kind in VALID_CRYPTO_KINDS {
-            // Do we need to bootstrap this crypto kind?
-            let eckey = (RoutingDomain::PublicInternet, crypto_kind);
-            let cnt = entry_count.get(&eckey).copied().unwrap_or_default();
-            if cnt < MIN_BOOTSTRAP_PEERS {
-                crypto_kinds.push(crypto_kind);
-            }
-        }
-        crypto_kinds
-    }
-
     #[instrument(level = "trace", skip_all, err)]
     pub async fn bootstrap_task_routine(
         &self,
@@ -374,86 +13,153 @@ impl RoutingTable {
         _last_ts: Timestamp,
         _cur_ts: Timestamp,
     ) -> EyreResult<()> {
-        let bootstrap = self
+        let bootstraps = self
             .config()
             .with(|c| c.network.routing_table.bootstrap.clone());
 
         // Don't bother if bootstraps aren't configured
-        if bootstrap.is_empty() {
+        if bootstraps.is_empty() {
             return Ok(());
         }
 
         veilid_log!(self debug "--- bootstrap_task");
 
         // See if we are specifying a direct dialinfo for bootstrap, if so use the direct mechanism
+        // Otherwise treat them as txt names for the normal dns-based bootstrap mechanism
+        let dial_info_converter = BootstrapDialInfoConverter::default();
+
         let mut bootstrap_dialinfos = Vec::<DialInfo>::new();
-        for b in &bootstrap {
-            if let Ok(bootstrap_di_vec) = DialInfo::try_vec_from_url(b) {
+        let mut bootstrap_txt_names = Vec::<String>::new();
+        for bootstrap in bootstraps {
+            if let Ok(bootstrap_di_vec) = dial_info_converter.try_vec_from_url(&bootstrap) {
                 for bootstrap_di in bootstrap_di_vec {
                     bootstrap_dialinfos.push(bootstrap_di);
                 }
+            } else {
+                bootstrap_txt_names.push(bootstrap);
+            }
+        }
+
+        // Process bootstraps into peers lists
+        let network_manager = self.network_manager();
+        let mut unord = FuturesUnordered::<PinBoxFuture<EyreResult<Vec<Arc<PeerInfo>>>>>::new();
+
+        // Get a peer list from bootstraps to process
+        for bsdi in bootstrap_dialinfos {
+            // Direct bootstrap
+            unord.push(pin_dyn_future!(async {
+                let bsdi = bsdi;
+                veilid_log!(self debug "Direct bootstrap with: {}", bsdi);
+                pin_future!(network_manager.direct_bootstrap(bsdi)).await
+            }));
+        }
+        for bstn in bootstrap_txt_names {
+            // TXT bootstrap
+            unord.push(pin_dyn_future!(async {
+                let bstn = bstn;
+                veilid_log!(self debug "TXT bootstrap with: {}", bstn);
+                pin_future!(network_manager.txt_bootstrap(bstn)).await
+            }));
+        }
+
+        let mut bootstrapped_peer_id_set = HashSet::<TypedKey>::new();
+        let mut bootstrapped_peers = vec![];
+        loop {
+            match unord.next().timeout_at(stop_token.clone()).await {
+                Ok(Some(res)) => match res {
+                    Ok(peers) => {
+                        for peer in peers {
+                            let peer_node_ids = peer.node_ids();
+                            if !peer_node_ids
+                                .iter()
+                                .any(|x| bootstrapped_peer_id_set.contains(x))
+                            {
+                                if self.matches_own_node_id(peer_node_ids) {
+                                    veilid_log!(self debug "Ignoring own node in bootstrap response");
+                                } else {
+                                    for nid in peer.node_ids().iter().copied() {
+                                        bootstrapped_peer_id_set.insert(nid);
+                                    }
+                                    bootstrapped_peers.push(peer);
+                                }
+                            }
+                        }
+                    }
+                    Err(e) => {
+                        veilid_log!(self debug "Bootstrap error: {}", e);
+                    }
+                },
+                Ok(None) => {
+                    // Done
+                    break;
+                }
+                Err(_) => {
+                    // Cancelled
+                    return Ok(());
+                }
             }
         }
 
-        // Get a peer list from bootstrap to process
-        let peers = if !bootstrap_dialinfos.is_empty() {
-            // Direct bootstrap
-            let network_manager = self.network_manager();
+        // Get list of routing domains to bootstrap
+        // NOTE: someday we may want to boot other domains than PublicInternet
+        // so we are writing this code as generically as possible
+        let routing_domains: RoutingDomainSet = bootstrapped_peers
+            .iter()
+            .map(|pi| pi.routing_domain())
+            .collect();
 
-            let mut peer_map = HashMap::<TypedKeyGroup, Arc<PeerInfo>>::new();
-            for bootstrap_di in bootstrap_dialinfos {
-                veilid_log!(self debug "direct bootstrap with: {}", bootstrap_di);
-                let peers = pin_future!(network_manager.boot_request(bootstrap_di)).await?;
-                for peer in peers {
-                    if !peer_map.contains_key(peer.node_ids()) {
-                        peer_map.insert(peer.node_ids().clone(), peer);
+        // For each routing domain, bootstrap it if any node in the bootstrap list
+        // from TXT or BOOT mechanism has peer info in the routing domain
+        // When a routing domain is bootstrapped, its current list of bootstrap nodes is cleared
+        // and replaced with the latest peer info we are bootstrapping with.
+        //
+        // We do not use these stored bootstrap peers for anything right now but keeping track of them
+        // will be useful for eventual tasks like configuration distribution and automatic updates.
+        for rd in routing_domains {
+            self.inner.read().with_routing_domain(rd, |rdd| {
+                rdd.clear_bootstrap_peers();
+            });
+        }
+
+        let valid_bootstraps = network_manager
+            .bootstrap_with_peer_list(bootstrapped_peers.clone(), stop_token)
+            .await?;
+        if valid_bootstraps.is_empty() {
+            veilid_log!(self debug "No external bootstrap peers");
+        } else {
+            veilid_log!(self debug "External bootstrap peers: {:#?}", valid_bootstraps);
+
+            for rd in routing_domains {
+                // Get all node ids that have a peerinfo in this routing domain
+                let mut rd_peer_ids = BTreeSet::new();
+                for peer in bootstrapped_peers.iter() {
+                    if peer.routing_domain() == rd {
+                        for nid in peer.node_ids().iter().copied() {
+                            rd_peer_ids.insert(nid);
+                        }
                     }
                 }
+
+                // Add valid bootstrap peers to routing domain
+                self.inner.read().with_routing_domain(rd, |rdd| {
+                    for bootstrap_peer in valid_bootstraps.iter().cloned() {
+                        let mut add = false;
+                        if let Some(pi) = bootstrap_peer.get_peer_info(rd) {
+                            for nid in pi.node_ids().iter() {
+                                if rd_peer_ids.contains(nid) {
+                                    add = true;
+                                    break;
+                                }
+                            }
+                        }
+                        if add {
+                            rdd.add_bootstrap_peer(bootstrap_peer);
+                        }
+                    }
+                });
             }
-            peer_map.into_values().collect()
-        } else {
-            // If not direct, resolve bootstrap servers and recurse their TXT entries
-            let bsrecs = match pin_future!(self
-                .resolve_bootstrap(bootstrap)
-                .timeout_at(stop_token.clone()))
-            .await
-            {
-                Ok(v) => v?,
-                Err(_) => {
-                    // Stop requested
-                    return Ok(());
-                }
-            };
-            let peers: Vec<Arc<PeerInfo>> = bsrecs
-                .into_iter()
-                .map(|bsrec| {
-                    // Get crypto support from list of node ids
-                    let crypto_support = bsrec.node_ids.kinds();
+        }
 
-                    // Make unsigned SignedNodeInfo
-                    let sni = SignedNodeInfo::Direct(SignedDirectNodeInfo::with_no_signature(
-                        NodeInfo::new(
-                            NetworkClass::InboundCapable, // Bootstraps are always inbound capable
-                            ProtocolTypeSet::all(), // Bootstraps are always capable of all protocols
-                            AddressTypeSet::all(),  // Bootstraps are always IPV4 and IPV6 capable
-                            bsrec.envelope_support, // Envelope support is as specified in the bootstrap list
-                            crypto_support, // Crypto support is derived from list of node ids
-                            vec![],         // Bootstrap needs no capabilities
-                            bsrec.dial_info_details, // Dial info is as specified in the bootstrap list
-                        ),
-                    ));
-
-                    Arc::new(PeerInfo::new(
-                        RoutingDomain::PublicInternet,
-                        bsrec.node_ids,
-                        sni,
-                    ))
-                })
-                .collect();
-
-            peers
-        };
-
-        self.bootstrap_with_peer_list(peers, stop_token).await
+        Ok(())
     }
 }

veilid-core/src/routing_table/tasks/closest_peers_refresh.rs

@@ -9,6 +9,7 @@ use stop_token::future::FutureExt as StopFutureExt;
 impl RoutingTable {
     /// Ask our closest peers to give us more peers close to ourselves. This will
     /// assist with the DHT and other algorithms that utilize the distance metric.
+    /// This only finds nodes in the PublicInternet domain.
     #[instrument(level = "trace", skip(self), err)]
     pub async fn closest_peers_refresh_task_routine(
         &self,

veilid-core/src/routing_table/tasks/mod.rs

@@ -137,9 +137,9 @@ impl RoutingTable {
         }
 
         // Refresh entry counts
-        let entry_counts = {
+        let live_entry_counts = {
             let mut inner = self.inner.write();
-            inner.refresh_cached_entry_counts()
+            inner.refresh_cached_live_entry_counts()
         };
 
         // Only do the rest if the network has started
@@ -147,27 +147,47 @@ impl RoutingTable {
             return Ok(());
         }
 
-        let min_peer_count = self
-            .config()
-            .with(|c| c.network.dht.min_peer_count as usize);
-
-        // Figure out which tables need bootstrap or peer minimum refresh
+        // Figure out if we need bootstrap
         let mut needs_bootstrap = false;
-        let mut needs_peer_minimum_refresh = false;
-        for ck in VALID_CRYPTO_KINDS {
-            let eckey = (RoutingDomain::PublicInternet, ck);
-            let cnt = entry_counts.get(&eckey).copied().unwrap_or_default();
-            if cnt < MIN_PUBLIC_INTERNET_ROUTING_DOMAIN_NODE_COUNT {
-                needs_bootstrap = true;
-            } else if cnt < min_peer_count {
-                needs_peer_minimum_refresh = true;
+        for rd in BOOTSTRAP_ROUTING_DOMAINS {
+            for ck in VALID_CRYPTO_KINDS {
+                let eckey = (rd, ck);
+                let cnt = live_entry_counts
+                    .connectivity_capabilities
+                    .get(&eckey)
+                    .copied()
+                    .unwrap_or_default();
+                if cnt < MIN_BOOTSTRAP_CONNECTIVITY_PEERS {
+                    needs_bootstrap = true;
+                }
+            }
+            if needs_bootstrap {
+                self.bootstrap_task.tick().await?;
             }
         }
-        if needs_bootstrap {
-            self.bootstrap_task.tick().await?;
-        }
-        if needs_peer_minimum_refresh {
-            self.peer_minimum_refresh_task.tick().await?;
+
+        // Figure out if we need peer minimum refresh
+        // XXX: eventually this should probably be done per-routingdomain
+        let mut needs_peer_minimum_refresh = false;
+        if !needs_bootstrap {
+            let min_peer_count = self
+                .config()
+                .with(|c| c.network.dht.min_peer_count as usize);
+
+            for ck in VALID_CRYPTO_KINDS {
+                let eckey = (RoutingDomain::PublicInternet, ck);
+                let cnt = live_entry_counts
+                    .connectivity_capabilities
+                    .get(&eckey)
+                    .copied()
+                    .unwrap_or_default();
+                if cnt < min_peer_count {
+                    needs_peer_minimum_refresh = true;
+                }
+            }
+            if needs_peer_minimum_refresh {
+                self.peer_minimum_refresh_task.tick().await?;
+            }
         }
 
         // Ping validate some nodes to groom the table
@@ -181,7 +201,7 @@ impl RoutingTable {
         // Run the relay management task
         self.relay_management_task.tick().await?;
 
-        // Get more nodes if we need to
+        // Get more nodes close to our node id
         if !needs_bootstrap && !needs_peer_minimum_refresh {
             // Run closest peers refresh task
             self.closest_peers_refresh_task.tick().await?;

veilid-core/src/routing_table/tasks/peer_minimum_refresh.rs

@@ -10,7 +10,8 @@ impl RoutingTable {
     // mechanism for LocalNetwork suffices for locating all the local network
     // peers that are available. This, however, may query other LocalNetwork
     // nodes for their PublicInternet peers, which is a very fast way to get
-    // a new node online.
+    // a new node online. This finds nodes that have connectivity capabilities
+    // specifically, as those are required for most nodes to get online.
     #[instrument(level = "trace", skip(self), err)]
     pub async fn peer_minimum_refresh_task_routine(
         &self,
@@ -19,7 +20,7 @@ impl RoutingTable {
         _cur_ts: Timestamp,
     ) -> EyreResult<()> {
         // Get counts by crypto kind
-        let entry_count = self.inner.read().cached_entry_counts();
+        let live_entry_counts = self.inner.read().cached_live_entry_counts();
 
         let (min_peer_count, min_peer_refresh_time_ms) = self.config().with(|c| {
             (
@@ -37,7 +38,11 @@ impl RoutingTable {
         for crypto_kind in VALID_CRYPTO_KINDS {
             // Do we need to peer minimum refresh this crypto kind?
             let eckey = (RoutingDomain::PublicInternet, crypto_kind);
-            let cnt = entry_count.get(&eckey).copied().unwrap_or_default();
+            let cnt = live_entry_counts
+                .connectivity_capabilities
+                .get(&eckey)
+                .copied()
+                .unwrap_or_default();
             if cnt == 0 || cnt > min_peer_count {
                 // If we have enough nodes, skip it
                 // If we have zero nodes, bootstrap will get it
@@ -54,6 +59,13 @@ impl RoutingTable {
                         if !compatible_crypto {
                             return false;
                         }
+                        // Keep only the entries with connectivity capabilities
+                        if !e.has_all_capabilities(
+                            RoutingDomain::PublicInternet,
+                            CONNECTIVITY_CAPABILITIES,
+                        ) {
+                            return false;
+                        }
                         // Keep only the entries we haven't talked to in the min_peer_refresh_time
                         if let Some(last_q_ts) = e.peer_stats().rpc_stats.last_question_ts {
                             if cur_ts.saturating_sub(last_q_ts.as_u64())
@@ -78,8 +90,16 @@ impl RoutingTable {
 
             for nr in noderefs {
                 ord.push_back(
-                    async move { self.reverse_find_node(crypto_kind, nr, false, vec![]).await }
-                        .instrument(Span::current()),
+                    async move {
+                        self.reverse_find_node(
+                            crypto_kind,
+                            nr,
+                            false,
+                            CONNECTIVITY_CAPABILITIES.to_vec(),
+                        )
+                        .await
+                    }
+                    .instrument(Span::current()),
                 );
             }
         }

veilid-core/src/routing_table/types/node_info.rs

@@ -14,6 +14,8 @@ pub const CAP_APPMESSAGE: Capability = FourCC(*b"APPM");
 pub const CAP_BLOCKSTORE: Capability = FourCC(*b"BLOC");
 
 pub const DISTANCE_METRIC_CAPABILITIES: &[Capability] = &[CAP_DHT, CAP_DHT_WATCH];
+pub const CONNECTIVITY_CAPABILITIES: &[Capability] =
+    &[CAP_RELAY, CAP_SIGNAL, CAP_ROUTE, CAP_VALIDATE_DIAL_INFO];
 
 #[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
 pub struct NodeInfo {

veilid-core/src/tests/common/test_veilid_config.rs

@@ -208,10 +208,17 @@ pub fn config_callback(key: String) -> ConfigCallbackReturn {
         "network.routing_table.node_id_secret" => Ok(Box::new(TypedSecretGroup::new())),
         // "network.routing_table.bootstrap" => Ok(Box::new(Vec::<String>::new())),
         #[cfg(not(all(target_arch = "wasm32", target_os = "unknown")))]
-        "network.routing_table.bootstrap" => Ok(Box::new(vec!["bootstrap.veilid.net".to_string()])),
+        "network.routing_table.bootstrap" => {
+            Ok(Box::new(vec!["bootstrap-v1.veilid.net".to_string()]))
+        }
         #[cfg(all(target_arch = "wasm32", target_os = "unknown"))]
         "network.routing_table.bootstrap" => Ok(Box::new(vec![
-            "ws://bootstrap.veilid.net:5150/ws".to_string(),
+            "ws://bootstrap-v1.veilid.net:5150/ws".to_string(),
+        ])),
+        "network.routing_table.bootstrap_keys" => Ok(Box::new(vec![
+            TypedKey::from_str("VLD0:Vj0lKDdUQXmQ5Ol1SZdlvXkBHUccBcQvGLN9vbLSI7k").unwrap(),
+            TypedKey::from_str("VLD0:QeQJorqbXtC7v3OlynCZ_W3m76wGNeB5NTF81ypqHAo").unwrap(),
+            TypedKey::from_str("VLD0:QNdcl-0OiFfYVj9331XVR6IqZ49NG-E18d5P7lwi4TA").unwrap(),
         ])),
         "network.routing_table.limit_over_attached" => Ok(Box::new(64u32)),
         "network.routing_table.limit_fully_attached" => Ok(Box::new(32u32)),
@@ -353,12 +360,20 @@ pub fn test_config() {
     #[cfg(not(all(target_arch = "wasm32", target_os = "unknown")))]
     assert_eq!(
         inner.network.routing_table.bootstrap,
-        vec!["bootstrap.veilid.net"],
+        vec!["bootstrap-v1.veilid.net"],
     );
     #[cfg(all(target_arch = "wasm32", target_os = "unknown"))]
     assert_eq!(
         inner.network.routing_table.bootstrap,
-        vec!["ws://bootstrap.veilid.net:5150/ws"],
+        vec!["ws://bootstrap-v1.veilid.net:5150/ws"],
+    );
+    assert_eq!(
+        inner.network.routing_table.bootstrap_keys,
+        vec![
+            TypedKey::from_str("VLD0:Vj0lKDdUQXmQ5Ol1SZdlvXkBHUccBcQvGLN9vbLSI7k").unwrap(),
+            TypedKey::from_str("VLD0:QeQJorqbXtC7v3OlynCZ_W3m76wGNeB5NTF81ypqHAo").unwrap(),
+            TypedKey::from_str("VLD0:QNdcl-0OiFfYVj9331XVR6IqZ49NG-E18d5P7lwi4TA").unwrap(),
+        ],
     );
     assert_eq!(inner.network.routing_table.limit_over_attached, 64u32);
     assert_eq!(inner.network.routing_table.limit_fully_attached, 32u32);

veilid-core/src/tests/native/mod.rs

@@ -28,8 +28,10 @@ pub async fn run_all_tests() {
     test_envelope_receipt::test_all().await;
     info!("TEST: veilid_api::tests::test_serialize_json");
     veilid_api::tests::test_serialize_json::test_all().await;
-    info!("TEST: routing_table::test_serialize_routing_table");
+    info!("TEST: routing_table::tests::test_serialize_routing_table");
     routing_table::tests::test_serialize_routing_table::test_all().await;
+    info!("TEST: network_manager::tests::test_bootstrap");
+    network_manager::tests::test_bootstrap::test_all().await;
     // info!("TEST: test_dht");
     // test_dht::test_all().await;
 
@@ -131,6 +133,8 @@ cfg_if! {
 
         run_test!(routing_table, test_serialize_routing_table);
 
+        run_test!(network_manager, test_bootstrap);
+
         // run_test!(test_dht);
     }
 }

veilid-core/src/veilid_api/debug.rs

@@ -237,6 +237,9 @@ fn get_public_key(text: &str) -> Option<PublicKey> {
 fn get_keypair(text: &str) -> Option<KeyPair> {
     KeyPair::from_str(text).ok()
 }
+fn get_typedkeypair(text: &str) -> Option<TypedKeyPair> {
+    TypedKeyPair::from_str(text).ok()
+}
 
 fn get_crypto_system_version<'a>(
     crypto: &'a Crypto,
@@ -644,10 +647,20 @@ impl VeilidAPI {
         Ok(routing_table.debug_info_peerinfo(routing_domain, published))
     }
 
-    async fn debug_txtrecord(&self, _args: String) -> VeilidAPIResult<String> {
+    async fn debug_txtrecord(&self, args: String) -> VeilidAPIResult<String> {
         // Dump routing table txt record
-        let routing_table = self.core_context()?.routing_table();
-        Ok(routing_table.debug_info_txtrecord().await)
+        let args: Vec<String> = args.split_whitespace().map(|s| s.to_owned()).collect();
+
+        let signing_key_pair = get_debug_argument_at(
+            &args,
+            0,
+            "debug_txtrecord",
+            "signing_key_pair",
+            get_typedkeypair,
+        )?;
+
+        let network_manager = self.core_context()?.network_manager();
+        Ok(network_manager.debug_info_txtrecord(signing_key_pair).await)
     }
 
     fn debug_keypair(&self, args: String) -> VeilidAPIResult<String> {

veilid-core/src/veilid_api/tests/fixtures.rs

@@ -151,6 +151,10 @@ pub fn fix_veilidconfig() -> VeilidConfig {
                 node_id: TypedKeyGroup::new(),
                 node_id_secret: TypedSecretGroup::new(),
                 bootstrap: vec!["boots".to_string()],
+                bootstrap_keys: vec![TypedKey::from_str(
+                    "VLD0:qrxwD1-aM9xiUw4IAPVXE_4qgoIfyR4Y6MEPyaDl_GQ",
+                )
+                .unwrap()],
                 limit_over_attached: 1,
                 limit_fully_attached: 2,
                 limit_attached_strong: 3,

veilid-core/src/veilid_config.rs

@@ -500,6 +500,8 @@ pub struct VeilidConfigRoutingTable {
     #[schemars(with = "Vec<String>")]
     pub node_id_secret: TypedSecretGroup,
     pub bootstrap: Vec<String>,
+    #[schemars(with = "Vec<String>")]
+    pub bootstrap_keys: Vec<TypedKey>,
     pub limit_over_attached: u32,
     pub limit_fully_attached: u32,
     pub limit_attached_strong: u32,
@@ -513,16 +515,25 @@ impl Default for VeilidConfigRoutingTable {
     fn default() -> Self {
         cfg_if::cfg_if! {
             if #[cfg(all(target_arch = "wasm32", target_os = "unknown"))] {
-                let bootstrap = vec!["ws://bootstrap.veilid.net:5150/ws".to_string()];
+                let bootstrap = vec!["ws://bootstrap-v1.veilid.net:5150/ws".to_string()];
             } else {
-                let bootstrap = vec!["bootstrap.veilid.net".to_string()];
+                let bootstrap = vec!["bootstrap-v1.veilid.net".to_string()];
             }
         }
+        let bootstrap_keys = vec![
+            // Primary Veilid Foundation bootstrap signing key
+            TypedKey::from_str("VLD0:Vj0lKDdUQXmQ5Ol1SZdlvXkBHUccBcQvGLN9vbLSI7k").unwrap(),
+            // Secondary Veilid Foundation bootstrap signing key
+            TypedKey::from_str("VLD0:QeQJorqbXtC7v3OlynCZ_W3m76wGNeB5NTF81ypqHAo").unwrap(),
+            // Backup Veilid Foundation bootstrap signing key
+            TypedKey::from_str("VLD0:QNdcl-0OiFfYVj9331XVR6IqZ49NG-E18d5P7lwi4TA").unwrap(),
+        ];
 
         Self {
             node_id: TypedKeyGroup::default(),
             node_id_secret: TypedSecretGroup::default(),
             bootstrap,
+            bootstrap_keys,
             limit_over_attached: 64,
             limit_fully_attached: 32,
             limit_attached_strong: 16,
@@ -967,6 +978,7 @@ impl VeilidStartupOptions {
         get_config!(inner.network.routing_table.node_id);
         get_config!(inner.network.routing_table.node_id_secret);
         get_config!(inner.network.routing_table.bootstrap);
+        get_config!(inner.network.routing_table.bootstrap_keys);
         get_config!(inner.network.routing_table.limit_over_attached);
         get_config!(inner.network.routing_table.limit_fully_attached);
         get_config!(inner.network.routing_table.limit_attached_strong);

veilid-flutter/example/lib/app.dart

@@ -1,3 +1,6 @@
+// Allow environment variables
+// ignore_for_file: avoid_redundant_argument_values, do_not_use_environment
+
 import 'dart:async';
 import 'dart:convert';
 
@@ -109,21 +112,21 @@ class _MyAppState extends State<MyApp> with UiLoggy {
       var config = await getDefaultVeilidConfig(
           isWeb: kIsWeb,
           programName: 'Veilid Plugin Example',
-          // ignore: avoid_redundant_argument_values, do_not_use_environment
           bootstrap: const String.fromEnvironment('BOOTSTRAP'),
-          // ignore: avoid_redundant_argument_values, do_not_use_environment
+          bootstrapKeys: const bool.hasEnvironment('BOOTSTRAP_KEYS')
+              ? const String.fromEnvironment('BOOTSTRAP_KEYS')
+              : const bool.hasEnvironment('BOOTSTRAP')
+                  ? ''
+                  : null,
           networkKeyPassword: const String.fromEnvironment('NETWORK_KEY'));
-      // ignore: do_not_use_environment
       if (const String.fromEnvironment('DELETE_TABLE_STORE') == '1') {
         config = config.copyWith(
             tableStore: config.tableStore.copyWith(delete: true));
       }
-      // ignore: do_not_use_environment
       if (const String.fromEnvironment('DELETE_PROTECTED_STORE') == '1') {
         config = config.copyWith(
             protectedStore: config.protectedStore.copyWith(delete: true));
       }
-      // ignore: do_not_use_environment
       if (const String.fromEnvironment('DELETE_BLOCK_STORE') == '1') {
         config = config.copyWith(
             blockStore: config.blockStore.copyWith(delete: true));
@@ -164,7 +167,7 @@ class _MyAppState extends State<MyApp> with UiLoggy {
               color: materialBackgroundColor.shade100,
               boxShadow: [
                 BoxShadow(
-                  color: Colors.black.withOpacity(0.15),
+                  color: Colors.black.withAlpha(38),
                   spreadRadius: 4,
                   blurRadius: 4,
                 )

veilid-flutter/lib/default_config.dart

@@ -9,6 +9,7 @@ Future<VeilidConfig> getDefaultVeilidConfig({
   required bool isWeb,
   required String programName,
   String bootstrap = '',
+  String? bootstrapKeys,
   String namespace = '',
   String deviceEncryptionKeyPassword = '',
   String? newDeviceEncryptionKeyPassword,
@@ -40,10 +41,15 @@ Future<VeilidConfig> getDefaultVeilidConfig({
         newDeviceEncryptionKeyPassword: newDeviceEncryptionKeyPassword,
       ),
       network: defaultConfig.network.copyWith(
-          networkKeyPassword: networkKeyPassword,
-          routingTable: defaultConfig.network.routingTable.copyWith(
-              bootstrap: bootstrap.isNotEmpty
-                  ? bootstrap.split(',')
-                  : defaultConfig.network.routingTable.bootstrap),
-          dht: defaultConfig.network.dht.copyWith()));
+        networkKeyPassword: networkKeyPassword,
+        routingTable: defaultConfig.network.routingTable.copyWith(
+            bootstrap: bootstrap.isNotEmpty
+                ? bootstrap.split(',')
+                : defaultConfig.network.routingTable.bootstrap,
+            bootstrapKeys: bootstrapKeys != null
+                ? bootstrapKeys.isNotEmpty
+                    ? bootstrapKeys.split(',').map(TypedKey.fromString).toList()
+                    : []
+                : defaultConfig.network.routingTable.bootstrapKeys),
+      ));
 }

veilid-flutter/lib/routing_context.freezed.dart

@@ -310,7 +310,7 @@ abstract mixin class $DHTSchemaMemberCopyWith<$Res> {
           DHTSchemaMember value, $Res Function(DHTSchemaMember) _then) =
       _$DHTSchemaMemberCopyWithImpl;
   @useResult
-  $Res call({FixedEncodedString43 mKey, int mCnt});
+  $Res call({PublicKey mKey, int mCnt});
 }
 
 /// @nodoc
@@ -331,9 +331,9 @@ class _$DHTSchemaMemberCopyWithImpl<$Res>
   }) {
     return _then(_self.copyWith(
       mKey: null == mKey
-          ? _self.mKey!
+          ? _self.mKey
           : mKey // ignore: cast_nullable_to_non_nullable
-              as FixedEncodedString43,
+              as PublicKey,
       mCnt: null == mCnt
           ? _self.mCnt
           : mCnt // ignore: cast_nullable_to_non_nullable
@@ -351,7 +351,7 @@ class _DHTSchemaMember implements DHTSchemaMember {
       _$DHTSchemaMemberFromJson(json);
 
   @override
-  final FixedEncodedString43 mKey;
+  final PublicKey mKey;
   @override
   final int mCnt;
 
@@ -397,7 +397,7 @@ abstract mixin class _$DHTSchemaMemberCopyWith<$Res>
       __$DHTSchemaMemberCopyWithImpl;
   @override
   @useResult
-  $Res call({FixedEncodedString43 mKey, int mCnt});
+  $Res call({PublicKey mKey, int mCnt});
 }
 
 /// @nodoc
@@ -420,7 +420,7 @@ class __$DHTSchemaMemberCopyWithImpl<$Res>
       mKey: null == mKey
           ? _self.mKey
           : mKey // ignore: cast_nullable_to_non_nullable
-              as FixedEncodedString43,
+              as PublicKey,
       mCnt: null == mCnt
           ? _self.mCnt
           : mCnt // ignore: cast_nullable_to_non_nullable
@@ -476,10 +476,10 @@ abstract mixin class $DHTRecordDescriptorCopyWith<$Res> {
       _$DHTRecordDescriptorCopyWithImpl;
   @useResult
   $Res call(
-      {Typed<FixedEncodedString43> key,
-      FixedEncodedString43 owner,
+      {TypedKey key,
+      PublicKey owner,
       DHTSchema schema,
-      FixedEncodedString43? ownerSecret});
+      PublicKey? ownerSecret});
 
   $DHTSchemaCopyWith<$Res> get schema;
 }
@@ -504,21 +504,21 @@ class _$DHTRecordDescriptorCopyWithImpl<$Res>
   }) {
     return _then(_self.copyWith(
       key: null == key
-          ? _self.key!
+          ? _self.key
           : key // ignore: cast_nullable_to_non_nullable
-              as Typed<FixedEncodedString43>,
+              as TypedKey,
       owner: null == owner
-          ? _self.owner!
+          ? _self.owner
           : owner // ignore: cast_nullable_to_non_nullable
-              as FixedEncodedString43,
+              as PublicKey,
       schema: null == schema
           ? _self.schema
           : schema // ignore: cast_nullable_to_non_nullable
               as DHTSchema,
       ownerSecret: freezed == ownerSecret
-          ? _self.ownerSecret!
+          ? _self.ownerSecret
           : ownerSecret // ignore: cast_nullable_to_non_nullable
-              as FixedEncodedString43?,
+              as PublicKey?,
     ));
   }
 
@@ -545,13 +545,13 @@ class _DHTRecordDescriptor implements DHTRecordDescriptor {
       _$DHTRecordDescriptorFromJson(json);
 
   @override
-  final Typed<FixedEncodedString43> key;
+  final TypedKey key;
   @override
-  final FixedEncodedString43 owner;
+  final PublicKey owner;
   @override
   final DHTSchema schema;
   @override
-  final FixedEncodedString43? ownerSecret;
+  final PublicKey? ownerSecret;
 
   /// Create a copy of DHTRecordDescriptor
   /// with the given fields replaced by the non-null parameter values.
@@ -600,10 +600,10 @@ abstract mixin class _$DHTRecordDescriptorCopyWith<$Res>
   @override
   @useResult
   $Res call(
-      {Typed<FixedEncodedString43> key,
-      FixedEncodedString43 owner,
+      {TypedKey key,
+      PublicKey owner,
       DHTSchema schema,
-      FixedEncodedString43? ownerSecret});
+      PublicKey? ownerSecret});
 
   @override
   $DHTSchemaCopyWith<$Res> get schema;
@@ -631,11 +631,11 @@ class __$DHTRecordDescriptorCopyWithImpl<$Res>
       key: null == key
           ? _self.key
           : key // ignore: cast_nullable_to_non_nullable
-              as Typed<FixedEncodedString43>,
+              as TypedKey,
       owner: null == owner
           ? _self.owner
           : owner // ignore: cast_nullable_to_non_nullable
-              as FixedEncodedString43,
+              as PublicKey,
       schema: null == schema
           ? _self.schema
           : schema // ignore: cast_nullable_to_non_nullable
@@ -643,7 +643,7 @@ class __$DHTRecordDescriptorCopyWithImpl<$Res>
       ownerSecret: freezed == ownerSecret
           ? _self.ownerSecret
           : ownerSecret // ignore: cast_nullable_to_non_nullable
-              as FixedEncodedString43?,
+              as PublicKey?,
     ));
   }
 
@@ -704,7 +704,7 @@ abstract mixin class $ValueDataCopyWith<$Res> {
   $Res call(
       {int seq,
       @Uint8ListJsonConverter.jsIsArray() Uint8List data,
-      FixedEncodedString43 writer});
+      PublicKey writer});
 }
 
 /// @nodoc
@@ -733,9 +733,9 @@ class _$ValueDataCopyWithImpl<$Res> implements $ValueDataCopyWith<$Res> {
           : data // ignore: cast_nullable_to_non_nullable
               as Uint8List,
       writer: null == writer
-          ? _self.writer!
+          ? _self.writer
           : writer // ignore: cast_nullable_to_non_nullable
-              as FixedEncodedString43,
+              as PublicKey,
     ));
   }
 }
@@ -757,7 +757,7 @@ class _ValueData implements ValueData {
   @Uint8ListJsonConverter.jsIsArray()
   final Uint8List data;
   @override
-  final FixedEncodedString43 writer;
+  final PublicKey writer;
 
   /// Create a copy of ValueData
   /// with the given fields replaced by the non-null parameter values.
@@ -806,7 +806,7 @@ abstract mixin class _$ValueDataCopyWith<$Res>
   $Res call(
       {int seq,
       @Uint8ListJsonConverter.jsIsArray() Uint8List data,
-      FixedEncodedString43 writer});
+      PublicKey writer});
 }
 
 /// @nodoc
@@ -837,7 +837,7 @@ class __$ValueDataCopyWithImpl<$Res> implements _$ValueDataCopyWith<$Res> {
       writer: null == writer
           ? _self.writer
           : writer // ignore: cast_nullable_to_non_nullable
-              as FixedEncodedString43,
+              as PublicKey,
     ));
   }
 }

veilid-flutter/lib/veilid_config.dart

@@ -335,6 +335,7 @@ sealed class VeilidConfigRoutingTable with _$VeilidConfigRoutingTable {
     required List<TypedKey> nodeId,
     required List<TypedSecret> nodeIdSecret,
     required List<String> bootstrap,
+    required List<TypedKey> bootstrapKeys,
     required int limitOverAttached,
     required int limitFullyAttached,
     required int limitAttachedStrong,

veilid-flutter/lib/veilid_config.freezed.dart

@@ -5520,6 +5520,7 @@ mixin _$VeilidConfigRoutingTable implements DiagnosticableTreeMixin {
   List<TypedKey> get nodeId;
   List<TypedSecret> get nodeIdSecret;
   List<String> get bootstrap;
+  List<TypedKey> get bootstrapKeys;
   int get limitOverAttached;
   int get limitFullyAttached;
   int get limitAttachedStrong;
@@ -5544,6 +5545,7 @@ mixin _$VeilidConfigRoutingTable implements DiagnosticableTreeMixin {
       ..add(DiagnosticsProperty('nodeId', nodeId))
       ..add(DiagnosticsProperty('nodeIdSecret', nodeIdSecret))
       ..add(DiagnosticsProperty('bootstrap', bootstrap))
+      ..add(DiagnosticsProperty('bootstrapKeys', bootstrapKeys))
       ..add(DiagnosticsProperty('limitOverAttached', limitOverAttached))
       ..add(DiagnosticsProperty('limitFullyAttached', limitFullyAttached))
       ..add(DiagnosticsProperty('limitAttachedStrong', limitAttachedStrong))
@@ -5560,6 +5562,8 @@ mixin _$VeilidConfigRoutingTable implements DiagnosticableTreeMixin {
             const DeepCollectionEquality()
                 .equals(other.nodeIdSecret, nodeIdSecret) &&
             const DeepCollectionEquality().equals(other.bootstrap, bootstrap) &&
+            const DeepCollectionEquality()
+                .equals(other.bootstrapKeys, bootstrapKeys) &&
             (identical(other.limitOverAttached, limitOverAttached) ||
                 other.limitOverAttached == limitOverAttached) &&
             (identical(other.limitFullyAttached, limitFullyAttached) ||
@@ -5579,6 +5583,7 @@ mixin _$VeilidConfigRoutingTable implements DiagnosticableTreeMixin {
       const DeepCollectionEquality().hash(nodeId),
       const DeepCollectionEquality().hash(nodeIdSecret),
       const DeepCollectionEquality().hash(bootstrap),
+      const DeepCollectionEquality().hash(bootstrapKeys),
       limitOverAttached,
       limitFullyAttached,
       limitAttachedStrong,
@@ -5587,7 +5592,7 @@ mixin _$VeilidConfigRoutingTable implements DiagnosticableTreeMixin {
 
   @override
   String toString({DiagnosticLevel minLevel = DiagnosticLevel.info}) {
-    return 'VeilidConfigRoutingTable(nodeId: $nodeId, nodeIdSecret: $nodeIdSecret, bootstrap: $bootstrap, limitOverAttached: $limitOverAttached, limitFullyAttached: $limitFullyAttached, limitAttachedStrong: $limitAttachedStrong, limitAttachedGood: $limitAttachedGood, limitAttachedWeak: $limitAttachedWeak)';
+    return 'VeilidConfigRoutingTable(nodeId: $nodeId, nodeIdSecret: $nodeIdSecret, bootstrap: $bootstrap, bootstrapKeys: $bootstrapKeys, limitOverAttached: $limitOverAttached, limitFullyAttached: $limitFullyAttached, limitAttachedStrong: $limitAttachedStrong, limitAttachedGood: $limitAttachedGood, limitAttachedWeak: $limitAttachedWeak)';
   }
 }
 
@@ -5598,9 +5603,10 @@ abstract mixin class $VeilidConfigRoutingTableCopyWith<$Res> {
       _$VeilidConfigRoutingTableCopyWithImpl;
   @useResult
   $Res call(
-      {List<Typed<FixedEncodedString43>> nodeId,
-      List<Typed<FixedEncodedString43>> nodeIdSecret,
+      {List<TypedKey> nodeId,
+      List<TypedSecret> nodeIdSecret,
       List<String> bootstrap,
+      List<TypedKey> bootstrapKeys,
       int limitOverAttached,
       int limitFullyAttached,
       int limitAttachedStrong,
@@ -5624,6 +5630,7 @@ class _$VeilidConfigRoutingTableCopyWithImpl<$Res>
     Object? nodeId = null,
     Object? nodeIdSecret = null,
     Object? bootstrap = null,
+    Object? bootstrapKeys = null,
     Object? limitOverAttached = null,
     Object? limitFullyAttached = null,
     Object? limitAttachedStrong = null,
@@ -5632,17 +5639,21 @@ class _$VeilidConfigRoutingTableCopyWithImpl<$Res>
   }) {
     return _then(_self.copyWith(
       nodeId: null == nodeId
-          ? _self.nodeId!
+          ? _self.nodeId
           : nodeId // ignore: cast_nullable_to_non_nullable
-              as List<Typed<FixedEncodedString43>>,
+              as List<TypedKey>,
       nodeIdSecret: null == nodeIdSecret
-          ? _self.nodeIdSecret!
+          ? _self.nodeIdSecret
           : nodeIdSecret // ignore: cast_nullable_to_non_nullable
-              as List<Typed<FixedEncodedString43>>,
+              as List<TypedSecret>,
       bootstrap: null == bootstrap
           ? _self.bootstrap
           : bootstrap // ignore: cast_nullable_to_non_nullable
               as List<String>,
+      bootstrapKeys: null == bootstrapKeys
+          ? _self.bootstrapKeys
+          : bootstrapKeys // ignore: cast_nullable_to_non_nullable
+              as List<TypedKey>,
       limitOverAttached: null == limitOverAttached
           ? _self.limitOverAttached
           : limitOverAttached // ignore: cast_nullable_to_non_nullable
@@ -5673,9 +5684,10 @@ class _VeilidConfigRoutingTable
     with DiagnosticableTreeMixin
     implements VeilidConfigRoutingTable {
   const _VeilidConfigRoutingTable(
-      {required final List<Typed<FixedEncodedString43>> nodeId,
-      required final List<Typed<FixedEncodedString43>> nodeIdSecret,
+      {required final List<TypedKey> nodeId,
+      required final List<TypedSecret> nodeIdSecret,
       required final List<String> bootstrap,
+      required final List<TypedKey> bootstrapKeys,
       required this.limitOverAttached,
       required this.limitFullyAttached,
       required this.limitAttachedStrong,
@@ -5683,21 +5695,22 @@ class _VeilidConfigRoutingTable
       required this.limitAttachedWeak})
       : _nodeId = nodeId,
         _nodeIdSecret = nodeIdSecret,
-        _bootstrap = bootstrap;
+        _bootstrap = bootstrap,
+        _bootstrapKeys = bootstrapKeys;
   factory _VeilidConfigRoutingTable.fromJson(Map<String, dynamic> json) =>
       _$VeilidConfigRoutingTableFromJson(json);
 
-  final List<Typed<FixedEncodedString43>> _nodeId;
+  final List<TypedKey> _nodeId;
   @override
-  List<Typed<FixedEncodedString43>> get nodeId {
+  List<TypedKey> get nodeId {
     if (_nodeId is EqualUnmodifiableListView) return _nodeId;
     // ignore: implicit_dynamic_type
     return EqualUnmodifiableListView(_nodeId);
   }
 
-  final List<Typed<FixedEncodedString43>> _nodeIdSecret;
+  final List<TypedSecret> _nodeIdSecret;
   @override
-  List<Typed<FixedEncodedString43>> get nodeIdSecret {
+  List<TypedSecret> get nodeIdSecret {
     if (_nodeIdSecret is EqualUnmodifiableListView) return _nodeIdSecret;
     // ignore: implicit_dynamic_type
     return EqualUnmodifiableListView(_nodeIdSecret);
@@ -5711,6 +5724,14 @@ class _VeilidConfigRoutingTable
     return EqualUnmodifiableListView(_bootstrap);
   }
 
+  final List<TypedKey> _bootstrapKeys;
+  @override
+  List<TypedKey> get bootstrapKeys {
+    if (_bootstrapKeys is EqualUnmodifiableListView) return _bootstrapKeys;
+    // ignore: implicit_dynamic_type
+    return EqualUnmodifiableListView(_bootstrapKeys);
+  }
+
   @override
   final int limitOverAttached;
   @override
@@ -5745,6 +5766,7 @@ class _VeilidConfigRoutingTable
       ..add(DiagnosticsProperty('nodeId', nodeId))
       ..add(DiagnosticsProperty('nodeIdSecret', nodeIdSecret))
       ..add(DiagnosticsProperty('bootstrap', bootstrap))
+      ..add(DiagnosticsProperty('bootstrapKeys', bootstrapKeys))
       ..add(DiagnosticsProperty('limitOverAttached', limitOverAttached))
       ..add(DiagnosticsProperty('limitFullyAttached', limitFullyAttached))
       ..add(DiagnosticsProperty('limitAttachedStrong', limitAttachedStrong))
@@ -5762,6 +5784,8 @@ class _VeilidConfigRoutingTable
                 .equals(other._nodeIdSecret, _nodeIdSecret) &&
             const DeepCollectionEquality()
                 .equals(other._bootstrap, _bootstrap) &&
+            const DeepCollectionEquality()
+                .equals(other._bootstrapKeys, _bootstrapKeys) &&
             (identical(other.limitOverAttached, limitOverAttached) ||
                 other.limitOverAttached == limitOverAttached) &&
             (identical(other.limitFullyAttached, limitFullyAttached) ||
@@ -5781,6 +5805,7 @@ class _VeilidConfigRoutingTable
       const DeepCollectionEquality().hash(_nodeId),
       const DeepCollectionEquality().hash(_nodeIdSecret),
       const DeepCollectionEquality().hash(_bootstrap),
+      const DeepCollectionEquality().hash(_bootstrapKeys),
       limitOverAttached,
       limitFullyAttached,
       limitAttachedStrong,
@@ -5789,7 +5814,7 @@ class _VeilidConfigRoutingTable
 
   @override
   String toString({DiagnosticLevel minLevel = DiagnosticLevel.info}) {
-    return 'VeilidConfigRoutingTable(nodeId: $nodeId, nodeIdSecret: $nodeIdSecret, bootstrap: $bootstrap, limitOverAttached: $limitOverAttached, limitFullyAttached: $limitFullyAttached, limitAttachedStrong: $limitAttachedStrong, limitAttachedGood: $limitAttachedGood, limitAttachedWeak: $limitAttachedWeak)';
+    return 'VeilidConfigRoutingTable(nodeId: $nodeId, nodeIdSecret: $nodeIdSecret, bootstrap: $bootstrap, bootstrapKeys: $bootstrapKeys, limitOverAttached: $limitOverAttached, limitFullyAttached: $limitFullyAttached, limitAttachedStrong: $limitAttachedStrong, limitAttachedGood: $limitAttachedGood, limitAttachedWeak: $limitAttachedWeak)';
   }
 }
 
@@ -5802,9 +5827,10 @@ abstract mixin class _$VeilidConfigRoutingTableCopyWith<$Res>
   @override
   @useResult
   $Res call(
-      {List<Typed<FixedEncodedString43>> nodeId,
-      List<Typed<FixedEncodedString43>> nodeIdSecret,
+      {List<TypedKey> nodeId,
+      List<TypedSecret> nodeIdSecret,
       List<String> bootstrap,
+      List<TypedKey> bootstrapKeys,
       int limitOverAttached,
       int limitFullyAttached,
       int limitAttachedStrong,
@@ -5828,6 +5854,7 @@ class __$VeilidConfigRoutingTableCopyWithImpl<$Res>
     Object? nodeId = null,
     Object? nodeIdSecret = null,
     Object? bootstrap = null,
+    Object? bootstrapKeys = null,
     Object? limitOverAttached = null,
     Object? limitFullyAttached = null,
     Object? limitAttachedStrong = null,
@@ -5838,15 +5865,19 @@ class __$VeilidConfigRoutingTableCopyWithImpl<$Res>
       nodeId: null == nodeId
           ? _self._nodeId
           : nodeId // ignore: cast_nullable_to_non_nullable
-              as List<Typed<FixedEncodedString43>>,
+              as List<TypedKey>,
       nodeIdSecret: null == nodeIdSecret
           ? _self._nodeIdSecret
           : nodeIdSecret // ignore: cast_nullable_to_non_nullable
-              as List<Typed<FixedEncodedString43>>,
+              as List<TypedSecret>,
       bootstrap: null == bootstrap
           ? _self._bootstrap
           : bootstrap // ignore: cast_nullable_to_non_nullable
               as List<String>,
+      bootstrapKeys: null == bootstrapKeys
+          ? _self._bootstrapKeys
+          : bootstrapKeys // ignore: cast_nullable_to_non_nullable
+              as List<TypedKey>,
       limitOverAttached: null == limitOverAttached
           ? _self.limitOverAttached
           : limitOverAttached // ignore: cast_nullable_to_non_nullable

veilid-flutter/lib/veilid_config.g.dart

@@ -420,6 +420,9 @@ _VeilidConfigRoutingTable _$VeilidConfigRoutingTableFromJson(
           .toList(),
       bootstrap:
           (json['bootstrap'] as List<dynamic>).map((e) => e as String).toList(),
+      bootstrapKeys: (json['bootstrap_keys'] as List<dynamic>)
+          .map(Typed<FixedEncodedString43>.fromJson)
+          .toList(),
       limitOverAttached: (json['limit_over_attached'] as num).toInt(),
       limitFullyAttached: (json['limit_fully_attached'] as num).toInt(),
       limitAttachedStrong: (json['limit_attached_strong'] as num).toInt(),
@@ -433,6 +436,7 @@ Map<String, dynamic> _$VeilidConfigRoutingTableToJson(
       'node_id': instance.nodeId.map((e) => e.toJson()).toList(),
       'node_id_secret': instance.nodeIdSecret.map((e) => e.toJson()).toList(),
       'bootstrap': instance.bootstrap,
+      'bootstrap_keys': instance.bootstrapKeys.map((e) => e.toJson()).toList(),
       'limit_over_attached': instance.limitOverAttached,
       'limit_fully_attached': instance.limitFullyAttached,
       'limit_attached_strong': instance.limitAttachedStrong,

veilid-flutter/lib/veilid_state.freezed.dart

@@ -2379,10 +2379,7 @@ abstract mixin class $PeerTableDataCopyWith<$Res> {
           PeerTableData value, $Res Function(PeerTableData) _then) =
       _$PeerTableDataCopyWithImpl;
   @useResult
-  $Res call(
-      {List<Typed<FixedEncodedString43>> nodeIds,
-      String peerAddress,
-      PeerStats peerStats});
+  $Res call({List<TypedKey> nodeIds, String peerAddress, PeerStats peerStats});
 
   $PeerStatsCopyWith<$Res> get peerStats;
 }
@@ -2406,9 +2403,9 @@ class _$PeerTableDataCopyWithImpl<$Res>
   }) {
     return _then(_self.copyWith(
       nodeIds: null == nodeIds
-          ? _self.nodeIds!
+          ? _self.nodeIds
           : nodeIds // ignore: cast_nullable_to_non_nullable
-              as List<Typed<FixedEncodedString43>>,
+              as List<TypedKey>,
       peerAddress: null == peerAddress
           ? _self.peerAddress
           : peerAddress // ignore: cast_nullable_to_non_nullable
@@ -2435,16 +2432,16 @@ class _$PeerTableDataCopyWithImpl<$Res>
 @JsonSerializable()
 class _PeerTableData implements PeerTableData {
   const _PeerTableData(
-      {required final List<Typed<FixedEncodedString43>> nodeIds,
+      {required final List<TypedKey> nodeIds,
       required this.peerAddress,
       required this.peerStats})
       : _nodeIds = nodeIds;
   factory _PeerTableData.fromJson(Map<String, dynamic> json) =>
       _$PeerTableDataFromJson(json);
 
-  final List<Typed<FixedEncodedString43>> _nodeIds;
+  final List<TypedKey> _nodeIds;
   @override
-  List<Typed<FixedEncodedString43>> get nodeIds {
+  List<TypedKey> get nodeIds {
     if (_nodeIds is EqualUnmodifiableListView) return _nodeIds;
     // ignore: implicit_dynamic_type
     return EqualUnmodifiableListView(_nodeIds);
@@ -2501,10 +2498,7 @@ abstract mixin class _$PeerTableDataCopyWith<$Res>
       __$PeerTableDataCopyWithImpl;
   @override
   @useResult
-  $Res call(
-      {List<Typed<FixedEncodedString43>> nodeIds,
-      String peerAddress,
-      PeerStats peerStats});
+  $Res call({List<TypedKey> nodeIds, String peerAddress, PeerStats peerStats});
 
   @override
   $PeerStatsCopyWith<$Res> get peerStats;
@@ -2531,7 +2525,7 @@ class __$PeerTableDataCopyWithImpl<$Res>
       nodeIds: null == nodeIds
           ? _self._nodeIds
           : nodeIds // ignore: cast_nullable_to_non_nullable
-              as List<Typed<FixedEncodedString43>>,
+              as List<TypedKey>,
       peerAddress: null == peerAddress
           ? _self.peerAddress
           : peerAddress // ignore: cast_nullable_to_non_nullable
@@ -2715,7 +2709,7 @@ class VeilidAppMessage implements VeilidUpdate {
 
   @Uint8ListJsonConverter.jsIsArray()
   final Uint8List message;
-  final Typed<FixedEncodedString43>? sender;
+  final TypedKey? sender;
   final String? routeId;
 
   @JsonKey(name: 'kind')
@@ -2765,7 +2759,7 @@ abstract mixin class $VeilidAppMessageCopyWith<$Res>
   @useResult
   $Res call(
       {@Uint8ListJsonConverter.jsIsArray() Uint8List message,
-      Typed<FixedEncodedString43>? sender,
+      TypedKey? sender,
       String? routeId});
 }
 
@@ -2793,7 +2787,7 @@ class _$VeilidAppMessageCopyWithImpl<$Res>
       sender: freezed == sender
           ? _self.sender
           : sender // ignore: cast_nullable_to_non_nullable
-              as Typed<FixedEncodedString43>?,
+              as TypedKey?,
       routeId: freezed == routeId
           ? _self.routeId
           : routeId // ignore: cast_nullable_to_non_nullable
@@ -2818,7 +2812,7 @@ class VeilidAppCall implements VeilidUpdate {
   @Uint8ListJsonConverter.jsIsArray()
   final Uint8List message;
   final String callId;
-  final Typed<FixedEncodedString43>? sender;
+  final TypedKey? sender;
   final String? routeId;
 
   @JsonKey(name: 'kind')
@@ -2870,7 +2864,7 @@ abstract mixin class $VeilidAppCallCopyWith<$Res>
   $Res call(
       {@Uint8ListJsonConverter.jsIsArray() Uint8List message,
       String callId,
-      Typed<FixedEncodedString43>? sender,
+      TypedKey? sender,
       String? routeId});
 }
 
@@ -2903,7 +2897,7 @@ class _$VeilidAppCallCopyWithImpl<$Res>
       sender: freezed == sender
           ? _self.sender
           : sender // ignore: cast_nullable_to_non_nullable
-              as Typed<FixedEncodedString43>?,
+              as TypedKey?,
       routeId: freezed == routeId
           ? _self.routeId
           : routeId // ignore: cast_nullable_to_non_nullable
@@ -3358,7 +3352,7 @@ class VeilidUpdateValueChange implements VeilidUpdate {
   factory VeilidUpdateValueChange.fromJson(Map<String, dynamic> json) =>
       _$VeilidUpdateValueChangeFromJson(json);
 
-  final Typed<FixedEncodedString43> key;
+  final TypedKey key;
   final List<ValueSubkeyRange> _subkeys;
   List<ValueSubkeyRange> get subkeys {
     if (_subkeys is EqualUnmodifiableListView) return _subkeys;
@@ -3417,7 +3411,7 @@ abstract mixin class $VeilidUpdateValueChangeCopyWith<$Res>
       _$VeilidUpdateValueChangeCopyWithImpl;
   @useResult
   $Res call(
-      {Typed<FixedEncodedString43> key,
+      {TypedKey key,
       List<ValueSubkeyRange> subkeys,
       int count,
       ValueData? value});
@@ -3446,7 +3440,7 @@ class _$VeilidUpdateValueChangeCopyWithImpl<$Res>
       key: null == key
           ? _self.key
           : key // ignore: cast_nullable_to_non_nullable
-              as Typed<FixedEncodedString43>,
+              as TypedKey,
       subkeys: null == subkeys
           ? _self._subkeys
           : subkeys // ignore: cast_nullable_to_non_nullable

veilid-flutter/packages/veilid_test/lib/src/veilid_fixture.dart

@@ -1,9 +1,11 @@
+// Allow environment variables
+// ignore_for_file: do_not_use_environment, avoid_redundant_argument_values
+
 import 'dart:async';
 
 import 'package:async_tools/async_tools.dart';
 import 'package:veilid/veilid.dart';
 
-// ignore: do_not_use_environment
 bool kIsWeb = const bool.fromEnvironment('dart.library.js_util');
 
 abstract class VeilidFixture {
@@ -32,17 +34,14 @@ class DefaultVeilidFixture implements VeilidFixture {
     _updateStreamController = StreamController.broadcast();
 
     final ignoreLogTargetsStr =
-        // ignore: do_not_use_environment
         const String.fromEnvironment('IGNORE_LOG_TARGETS').trim();
     final ignoreLogTargets = ignoreLogTargetsStr.isEmpty
         ? <String>[]
         : ignoreLogTargetsStr.split(',').map((e) => e.trim()).toList();
 
     final logLevel = VeilidConfigLogLevel.fromJson(
-        // ignore: do_not_use_environment
         const String.fromEnvironment('LOG_LEVEL', defaultValue: 'info'));
 
-    // ignore: do_not_use_environment
     final flamePathStr = const String.fromEnvironment('FLAME').trim();
 
     final Map<String, dynamic> platformConfigJson;
@@ -91,9 +90,12 @@ class DefaultVeilidFixture implements VeilidFixture {
     var config = await getDefaultVeilidConfig(
       isWeb: kIsWeb,
       programName: programName,
-      // ignore: avoid_redundant_argument_values, do_not_use_environment
       bootstrap: const String.fromEnvironment('BOOTSTRAP'),
-      // ignore: avoid_redundant_argument_values, do_not_use_environment
+      bootstrapKeys: const bool.hasEnvironment('BOOTSTRAP_KEYS')
+          ? const String.fromEnvironment('BOOTSTRAP_KEYS')
+          : const bool.hasEnvironment('BOOTSTRAP')
+              ? ''
+              : null,
       networkKeyPassword: const String.fromEnvironment('NETWORK_KEY'),
     );
 

veilid-python/veilid/config.py

@@ -71,6 +71,7 @@ class VeilidConfigRoutingTable(ConfigBase):
     node_id: list[TypedKey]
     node_id_secret: list[TypedSecret]
     bootstrap: list[str]
+    bootstrap_keys: list[TypedKey]
     limit_over_attached: int
     limit_fully_attached: int
     limit_attached_strong: int

veilid-python/veilid/schema/RecvMessage.json

@@ -4475,6 +4475,7 @@
       "type": "object",
       "required": [
         "bootstrap",
+        "bootstrap_keys",
         "limit_attached_good",
         "limit_attached_strong",
         "limit_attached_weak",
@@ -4490,6 +4491,12 @@
             "type": "string"
           }
         },
+        "bootstrap_keys": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        },
         "limit_attached_good": {
           "type": "integer",
           "format": "uint32",

veilid-server/src/main.rs

@@ -114,7 +114,7 @@ pub struct CmdlineArgs {
     /// Set the node ids and secret keys
     ///
     /// Specify node ids in typed key set format ('\[VLD0:xxxx,VLD1:xxxx\]') on the command line, a prompt appears to enter the secret key set interactively.
-    #[arg(long, value_name = "key_set")]
+    #[arg(long, value_name = "NODE_IDS")]
     set_node_id: Option<String>,
 
     /// Delete the entire contents of the protected store (DANGER, NO UNDO!)
@@ -134,8 +134,8 @@ pub struct CmdlineArgs {
     dump_config: bool,
 
     /// Prints the bootstrap TXT record for this node and then quits
-    #[arg(long)]
-    dump_txt_record: bool,
+    #[arg(long, value_name = "BOOTSTRAP_SIGNING_KEYPAIR")]
+    dump_txt_record: Option<String>,
 
     /// Emits a JSON-Schema for a named type
     #[arg(long, value_name = "schema_name")]
@@ -145,6 +145,10 @@ pub struct CmdlineArgs {
     #[arg(long, value_name = "BOOTSTRAP_LIST")]
     bootstrap: Option<String>,
 
+    /// Specify a list of bootstrap node ids to use
+    #[arg(long, value_name = "BOOTSTRAP_NODE_IDS_LIST")]
+    bootstrap_keys: Option<String>,
+
     /// Panic on ctrl-c instead of graceful shutdown
     #[arg(long)]
     panic: bool,
@@ -299,7 +303,7 @@ fn main() -> EyreResult<()> {
     if let Some(network_key) = args.network_key {
         settingsrw.core.network.network_key_password = Some(network_key);
     }
-    if args.dump_txt_record {
+    if args.dump_txt_record.is_some() {
         // Turn off terminal logging so we can be interactive
         settingsrw.logging.terminal.enabled = false;
     }
@@ -336,9 +340,42 @@ fn main() -> EyreResult<()> {
                 bootstrap_list.push(x);
             }
         }
-        settingsrw.core.network.routing_table.bootstrap = bootstrap_list;
+        if bootstrap_list != settingsrw.core.network.routing_table.bootstrap {
+            settingsrw.core.network.routing_table.bootstrap = bootstrap_list;
+            settingsrw.core.network.routing_table.bootstrap_keys = vec![];
+        }
     };
 
+    if let Some(bootstrap_keys) = args.bootstrap_keys {
+        println!("Overriding bootstrap keys with: ");
+        let mut bootstrap_keys_list: Vec<veilid_core::TypedKey> = Vec::new();
+        for x in bootstrap_keys.split(',') {
+            let x = x.trim();
+            let key = match veilid_core::TypedKey::from_str(x) {
+                Ok(v) => v,
+                Err(e) => {
+                    bail!("Failed to parse bootstrap key: {}\n{}", e, x)
+                }
+            };
+
+            println!("    {}", key);
+            bootstrap_keys_list.push(key);
+        }
+        settingsrw.core.network.routing_table.bootstrap_keys = bootstrap_keys_list;
+    };
+
+    if settingsrw
+        .core
+        .network
+        .routing_table
+        .bootstrap_keys
+        .is_empty()
+    {
+        println!(
+            "Bootstrap verification is disabled. Add bootstrap keys to your config to enable it."
+        );
+    }
+
     #[cfg(feature = "rt-tokio")]
     if args.console {
         settingsrw.logging.console.enabled = true;
@@ -426,8 +463,15 @@ fn main() -> EyreResult<()> {
             "Node Id and Secret set successfully",
             "Failed to set Node Id and Secret",
         )
-    } else if args.dump_txt_record {
-        (ServerMode::DumpTXTRecord, "", "Failed to dump txt record")
+    } else if let Some(skpstr) = args.dump_txt_record.as_ref() {
+        (
+            ServerMode::DumpTXTRecord(
+                veilid_core::TypedKeyPair::from_str(skpstr)
+                    .expect("should be valid typed key pair"),
+            ),
+            "",
+            "Failed to dump bootstrap TXT record",
+        )
     } else {
         (ServerMode::Normal, "", "")
     };

veilid-server/src/server.rs

@@ -16,7 +16,7 @@ use veilid_core::tools::*;
 pub enum ServerMode {
     Normal,
     ShutdownImmediate,
-    DumpTXTRecord,
+    DumpTXTRecord(veilid_core::TypedKeyPair),
 }
 
 lazy_static! {
@@ -163,7 +163,7 @@ pub async fn run_veilid_server_subnode(
     }
 
     // Process dump-txt-record
-    if matches!(server_mode, ServerMode::DumpTXTRecord) {
+    if let ServerMode::DumpTXTRecord(keypair) = server_mode {
         let start_time = Instant::now();
         while Instant::now().duration_since(start_time) < Duration::from_secs(10) {
             match veilid_api.get_state().await {
@@ -179,7 +179,7 @@ pub async fn run_veilid_server_subnode(
             }
             sleep(100).await;
         }
-        match veilid_api.debug("txtrecord".to_string()).await {
+        match veilid_api.debug(format!("txtrecord {}", keypair)).await {
             Ok(v) => {
                 print!("{}", v);
             }

veilid-server/src/settings.rs

@@ -142,7 +142,8 @@ core:
         routing_table:
             node_id: null
             node_id_secret: null
-            bootstrap: ['bootstrap.veilid.net']
+            bootstrap: ['bootstrap-v1.veilid.net']
+            bootstrap_keys: ['VLD0:Vj0lKDdUQXmQ5Ol1SZdlvXkBHUccBcQvGLN9vbLSI7k','VLD0:QeQJorqbXtC7v3OlynCZ_W3m76wGNeB5NTF81ypqHAo','VLD0:QNdcl-0OiFfYVj9331XVR6IqZ49NG-E18d5P7lwi4TA']
             limit_over_attached: 64
             limit_fully_attached: 32
             limit_attached_strong: 16
@@ -700,6 +701,7 @@ pub struct RoutingTable {
     pub node_id: Option<veilid_core::TypedKeyGroup>,
     pub node_id_secret: Option<veilid_core::TypedSecretGroup>,
     pub bootstrap: Vec<String>,
+    pub bootstrap_keys: Vec<veilid_core::TypedKey>,
     pub limit_over_attached: u32,
     pub limit_fully_attached: u32,
     pub limit_attached_strong: u32,
@@ -1158,6 +1160,7 @@ impl Settings {
         set_config_value!(inner.core.network.routing_table.node_id, value);
         set_config_value!(inner.core.network.routing_table.node_id_secret, value);
         set_config_value!(inner.core.network.routing_table.bootstrap, value);
+        set_config_value!(inner.core.network.routing_table.bootstrap_keys, value);
         set_config_value!(inner.core.network.routing_table.limit_over_attached, value);
         set_config_value!(inner.core.network.routing_table.limit_fully_attached, value);
         set_config_value!(
@@ -1353,6 +1356,9 @@ impl Settings {
                 "network.routing_table.bootstrap" => {
                     Ok(Box::new(inner.core.network.routing_table.bootstrap.clone()))
                 }
+                "network.routing_table.bootstrap_keys" => Ok(Box::new(
+                    inner.core.network.routing_table.bootstrap_keys.clone(),
+                )),
                 "network.routing_table.limit_over_attached" => Ok(Box::new(
                     inner.core.network.routing_table.limit_over_attached,
                 )),
@@ -1826,7 +1832,15 @@ mod tests {
         //
         assert_eq!(
             s.core.network.routing_table.bootstrap,
-            vec!["bootstrap.veilid.net".to_owned()]
+            vec!["bootstrap-v1.veilid.net".to_owned()]
+        );
+        assert_eq!(
+            s.core.network.routing_table.bootstrap_keys,
+            vec![
+                TypedKey::from_str("VLD0:Vj0lKDdUQXmQ5Ol1SZdlvXkBHUccBcQvGLN9vbLSI7k").unwrap(),
+                TypedKey::from_str("VLD0:QeQJorqbXtC7v3OlynCZ_W3m76wGNeB5NTF81ypqHAo").unwrap(),
+                TypedKey::from_str("VLD0:QNdcl-0OiFfYVj9331XVR6IqZ49NG-E18d5P7lwi4TA").unwrap(),
+            ]
         );
         //
         assert_eq!(s.core.network.rpc.concurrency, 0);

veilid-wasm/README.md

@@ -9,7 +9,7 @@ Running Veilid in the browser via WebAssembly has some limitations:
 ### Browser-based limitations
 
 1. TCP/UDP sockets are unavailable in the browser. This limits WASM nodes to communicating using WebSockets.
-1. Lookup of DNS records is unavailable in the browser, which means bootstrapping via TXT record also will not work. WASM nodes will need to connect to the bootstrap server directly via WebSockets, using this URL format: `ws://bootstrap.veilid.net:5150/ws` in the `network.routing_table.bootstrap[]` section of the veilid config.
+1. Lookup of DNS records is unavailable in the browser, which means bootstrapping via TXT record also will not work. WASM nodes will need to connect to the bootstrap server directly via WebSockets, using this URL format: `ws://bootstrap-v1.veilid.net:5150/ws` in the `network.routing_table.bootstrap[]` section of the veilid config.
 1. Do not set up any nodes with a core.network.protocol.wss.url IP address such as wss://12.34.56.78:5150/ws to support SSL. Even though a Certificate Authority (trusted by browsers) will give you an SSL certificate for an IP address, this is unsupported by Veilid as of v0.2.3. Any wss:// URL containing an IP address causes an RPC error in veilid-core and your node will lose communication with other nodes.
 1. Since a WASM node running in the browser can't open ports, WASM nodes select another node to act as its Inbound Relay, so other nodes can react out to it and open a WS connection.
 1. Because of browser security policy regarding WebSockets: