150 Commits

Author SHA1 Message Date
j-berman
8b12a335c6 fcmp++: implement iterative audit_tree function
- Recursion goes too deep
2024-08-08 19:26:08 -07:00
j-berman
d4847f649e Rename everything from fcmp* to fcmp_pp 2024-08-08 14:10:34 -07:00
j-berman
10c6c12b18 fcmp++: compilation fixes + misc. cleanup
- Removed call to hash_init_point in constructor
- Replaced global static CURVE_TREES_V1 with a smart pointer
- Don't need to link Rust static lib when including curve_trees.h
- leaves table doesn't need dupsort flags, all leaves should be
unique by key
- rename fcmp -> fcmp_pp
- return when 0 leaves passed into trim_tree
2024-08-08 13:31:21 -07:00
j-berman
b90cee8bab Store {O,C} for each leaf tuple instead of {O.x,I.x,C.x}
- Can derive {O.x,I.x,C.x} from {O,C}
- Note: this slows down tests since they do the derivation both
on insertion into the tree, and when auditing the tree
- At the hard fork, we don't need to store {O,C} in the
output_amounts table anymore since that table will no longer be
useful
2024-08-02 10:28:13 -07:00
j-berman
34eafa85f3 Store points in the tree in compressed encoding (32 bytes) 2024-08-01 10:04:51 -07:00
j-berman
729e31df74 include fcmp/curve_trees.h in db_lmdb.h 2024-07-29 03:40:44 -07:00
j-berman
c383087955 Instantiate m_curve_trees on BlockchainLMDB class in c'tor 2024-07-29 03:40:44 -07:00
j-berman
420b4b6a78 Resolve cross-compile errors
uint64_t -> size_t where value is expected bounded to small value
2024-07-29 03:40:44 -07:00
j-berman
b585a7f408 Remove copy in get_tree_extension and better named funcs 2024-07-29 03:40:44 -07:00
j-berman
db12610d94 Remove leaves from locked leaves table upon insertion to tree 2024-07-29 03:40:44 -07:00
j-berman
93795b4c9d Match output unlock time (fix off by 1) 2024-07-29 03:40:44 -07:00
j-berman
634e12e9ad Guarantee insertion order into the tree using global output ID
- Leaves enter the tree in the block they unlock, in the order
they appear in the chain
2024-07-29 03:40:44 -07:00
j-berman
306488b690 Implemented growing the tree on sync + lots of cleaning
- validate output and commitment in tuple conversion function
- function to get_unlock_height from height in chain + unlock_time
- tx_outs_to_leaf_tuples function
- cleaned up trim impl (reduced num params in instructions and
conditional complexity)
- renamed locked_outputs table to locked_leaves (clearer tie to
merkle tree)
- size_t -> uint64_t for db compatibility across 32-bit and 64-bit
machines
- added hash_grow tests
2024-07-29 03:40:44 -07:00
j-berman
8a89c20f3b lmdb migration to init curve trees tree from existing outputs
still rough
2024-07-29 03:40:44 -07:00
j-berman
f50ad5baac trim_tree db impl + db test + some housekeeping 2024-07-29 03:40:44 -07:00
j-berman
36f1e1965f Fix grow_tree, restructure it, and clean the approach
The initial impl didn't capture the following edge case:

  - Tree has 3 (or more) layers + 1 leaf layeri
  - Leaf layer last chunk IS full
  - Layer 0 last chunk is NOT full
  - Layer 1 last chunk is NOT full
  - Layer 2 last chunk IS NOT full

In this case, when updating layer 1, we need to use layer 0's old
last hash to update layer 1's old last hash. Same for Layer 2.

The solution is to use logic that checks the *prev* layer when
updating a layer to determine if the old last hash from the prev
layer is needed.

This commit restructures the grow_tree impl to account for this
and simplifies the approach as follows:

1. Read the tree to get num leaf tuples + last hashes in each layer

2. Get the tree extension using the above values + new leaf tuples
2a. Prior to updating the leaf layer, call the function
get_update_leaf_layer_metadata. This function uses existing totals
in the leaf layer, the new total of leaf tuples,  and tree params
to calculate how the layer after the leaf layer should be updated.
2b. For each subsequent layer, call the function
get_update_layer_metadata. This function uses the existing totals
in the *prev* layer, the new total of children in the *prev* layer,
and tree params to calculate how the layer should be updated.

3. Grow the tree using the tree extension.

This approach isolates update logic and actual hashing into neat
structured functions, rather than mix the two. This makes the code
easier to follow without needing to keep so much in your head at
one time.
2024-07-29 03:40:44 -07:00
j-berman
6045357f6a implement db->audit_tree, and flesh out db test to init/extend tree 2024-07-29 03:40:44 -07:00
j-berman
1ffde095c8 Implement get_tree_last_chunks in db, can now extend tree in db 2024-07-29 03:40:44 -07:00
j-berman
ab7c74136b Simplify edge case handling in hash_layer
- When retrieving last chunks, set next_start_child_chunk_index
so can know the correct start index without needing to modify
the offset
- Other smaller cleanup
2024-07-29 03:40:44 -07:00
j-berman
af9b74fcc7 start LMDB grow_tree 2024-07-29 03:40:44 -07:00
j-berman
e1c03f4d5a rough fcmp++ tree impl (lots of work remaining to clean it up and fix) 2024-07-29 03:40:44 -07:00
luigi1111
040bccc22f
Merge pull request #9338
4a376d6 copyright: fix vector.h typo (hinto.janaiyo)
d61e8e9 workflows: delete copyright.yml (plowsof)
341771a copyright: bump to 2024 (copyCat)
2024-07-16 18:42:49 -04:00
copyCat
341771ac3e copyright: bump to 2024 2024-05-21 16:29:33 +00:00
jeffro256
262ef473db
BlockchainDB: remove lock/unlock methods 2024-04-11 12:12:35 -05:00
jeffro256
b0bf49a65a
blockchain_db: add k-anonymity to txid fetching
Read more about k-anonymity [here](https://en.wikipedia.org/wiki/K-anonymity). We implement this feature in the monero daemon for transactions
by providing a "Txid Template", which is simply a txid with all but `num_matching_bits` bits zeroed out, and the number `num_matching_bits`. We add an operation to `BlockchainLMDB` called
`get_txids_loose` which takes a txid template and returns all txids in the database (chain and mempool) that satisfy that template. Thus, a client can
ask about a specific transaction from a daemon without revealing the exact transaction they are inquiring about. The client can control the statistical
chance that other TXIDs (besides the one in question) match the txid template sent to the daemon up to a power of 2. For example, if a client sets their `num_matching_bits`
to 5, then statistically any txid has a 1/(2^5) chance to match. With `num_matching_bits`=10, there is a 1/(2^10) chance, so on and so forth.

Co-authored-by: ACK-J <60232273+ACK-J@users.noreply.github.com>
2023-08-01 17:25:25 -05:00
mj-xmr
5b4fea72cf Copyright: Update to 2023
Co-authored-by: plowsof <plowsof@protonmail.com>
extra files
2023-01-16 13:00:18 +01:00
Howard Chu
b096e16699
Revert "db_lmdb: test for mmap support at init time"
This reverts commit bd96536637724413173271e8d5df1777f7879c29.

The check interferes with raw device/partition support.
2022-06-03 16:46:35 +01:00
mj-xmr
da9aa1f7f8
Copyright: Update to 2022 2022-03-04 06:59:20 +01:00
Howard Chu
5f54dea4ec
LMDB: fix deadlock in resized detection 2021-09-22 15:47:04 +01:00
moneromooo-monero
1eb14af1a3
rpc: limit the number of txes for get_blocks.bin 2020-12-29 04:31:58 +00:00
moneromooo-monero
55363c5941
Avoid some temporary strings when reading off the database 2020-08-17 14:02:27 +00:00
Alexander Blair
6d6c691a0f
Merge pull request #6536
bd9653663 db_lmdb: test for mmap support at init time (moneromooo-monero)
2020-07-19 03:37:15 -07:00
Alexander Blair
36d50d93f2
Merge pull request #6534
7bd66b01b daemon: guard against rare 'difficulty drift' bug with checkpoints and recalculation (stoffu)
2020-07-19 03:36:39 -07:00
stoffu
7bd66b01bf
daemon: guard against rare 'difficulty drift' bug with checkpoints and recalculation
On startup, it checks against the difficulty checkpoints, and if any mismatch is found, recalculates all the blocks with wrong difficulties. Additionally, once a week it recalculates difficulties of blocks after the last difficulty checkpoint.
2020-06-09 10:40:51 +09:00
moneromooo-monero
bd96536637
db_lmdb: test for mmap support at init time
It'll make it clearer when a DB init failure is due to being
on a filesystem which does not support mmap
2020-05-16 00:20:22 +00:00
SomaticFanatic
5ef0607da6 Update copyright year to 2020
Update copyright year to 2020
2020-05-06 22:36:54 -04:00
moneromooo-monero
8958b4e7aa
blockchain_db: faster fetching of consecutive txes
Useful for wallet refresh or node sync
2020-02-27 15:05:34 +00:00
Alexander Blair
8eedc8a390
Merge pull request #6140
08635a08 blockchain: speedup fetching pruned contiguous tx blobs (moneromooo-monero)
2020-01-25 16:55:18 -08:00
moneromooo-monero
08635a0875
blockchain: speedup fetching pruned contiguous tx blobs
About twice as fast, very roughly
2019-11-15 13:23:30 +00:00
Lee Clagett
5d7ae2d279 Adding support for hidden (anonymity) txpool 2019-11-02 20:36:03 +00:00
moneromooo-monero
06b8f29992
blockchain: keep alternative blocks in LMDB
Alternative blocks are cleared on startup unless --keep-alt-blocks
is passed on the command line
2019-05-08 17:36:52 +00:00
moneromooo-monero
5e673c03fe
blockchain_db: fix db txn ending too early
The db txn in add_block ending caused the entire overarching
batch txn to stop.
Also add a new guard class so a db txn can be stopped in the
face of exceptions.

Also use a read only db txn in init when the db itself is
read only, and do not save the max tx size in that case.
2019-04-14 08:35:38 +00:00
moneromooo-monero
089c7637a6
cryptonote: rework block blob size sanity check
Use the actual block weight limit, assuming that weight is always
greater or equal to size
2019-04-05 09:35:19 +00:00
moneromooo-monero
91f4c7f45f
Make difficulty 128 bit instead of 64 bit
Based on Boolberry work by:
  jahrsg <jahr@jahr.me>
  cr.zoidberg <crypto.zoidberg@gmail.com>
2019-03-24 21:03:19 +00:00
Riccardo Spagni
5ac46c5310
Merge pull request #5256
4b21d38d blockchain: speed up getting N blocks weights/long term weights (moneromooo-monero)
2019-03-21 14:48:40 +02:00
moneromooo-monero
4b21d38dfd
blockchain: speed up getting N blocks weights/long term weights 2019-03-08 12:04:14 +00:00
binaryFate
1f2930ce0b Update 2019 copyright 2019-03-05 22:05:34 +01:00
moneromooo-monero
79b4e9f377
save some database calls when getting top block hash and height 2019-03-05 11:58:05 +00:00
moneromooo-monero
b044d03a51
Avoid repeated (de)serialization when syncing 2019-03-05 11:57:55 +00:00
moneromooo-monero
b8787f4302
ArticMine's new block weight algorithm
This curbs runaway growth while still allowing substantial
spikes in block weight

Original specification from ArticMine:

here is the scaling proposal
Define: LongTermBlockWeight
Before fork:
LongTermBlockWeight = BlockWeight
At or after fork:
LongTermBlockWeight = min(BlockWeight, 1.4*LongTermEffectiveMedianBlockWeight)
Note: To avoid possible consensus issues over rounding the LongTermBlockWeight for a given block should be calculated to the nearest byte, and stored as a integer in the block itself. The stored LongTermBlockWeight is then used for future calculations of the LongTermEffectiveMedianBlockWeight and not recalculated each time.
Define:   LongTermEffectiveMedianBlockWeight
LongTermEffectiveMedianBlockWeight = max(300000, MedianOverPrevious100000Blocks(LongTermBlockWeight))
Change Definition of EffectiveMedianBlockWeight
From (current definition)
EffectiveMedianBlockWeight  = max(300000, MedianOverPrevious100Blocks(BlockWeight))
To (proposed definition)
EffectiveMedianBlockWeight  = min(max(300000, MedianOverPrevious100Blocks(BlockWeight)), 50*LongTermEffectiveMedianBlockWeight)
Notes:
1) There are no other changes to the existing penalty formula, median calculation, fees etc.
2) There is the requirement to store the LongTermBlockWeight of a block unencrypted in the block itself. This  is to avoid possible consensus issues over rounding and also to prevent the calculations from becoming unwieldy as we move away from the fork.
3) When the  EffectiveMedianBlockWeight cap is reached it is still possible to mine blocks up to 2x the EffectiveMedianBlockWeight by paying the corresponding penalty.

Note: the long term block weight is stored in the database, but not in the actual block itself,
since it requires recalculating anyway for verification.
2019-03-04 09:33:58 +00:00