Commit Graph

47 Commits

Author SHA1 Message Date
Riccardo Spagni
5a26676932
Merge pull request #343
e20a4dd blockchain: fix testnet syncing (to not use blocks.dat) (moneromooo-monero)
2015-07-18 22:59:02 +02:00
moneromooo-monero
e20a4ddc76
blockchain: fix testnet syncing (to not use blocks.dat)
These are mainnet blocks, and would cause syncing on testnet to
reject all incoming blocks.
2015-07-18 10:25:22 +01:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
70ae2ee711 Fixed threadpool bug when running on single core systems.
*Thanks to freshman for reporting bug.
2015-07-17 20:02:29 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
94ea3e8ed2 Removed on_idle() calls to Blockchain::store_blockchain() for lmdb.
Added option to cache tx-input verification results.
2015-07-15 23:20:25 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
e5d2680094 ** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)
Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-15 23:20:16 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
1f83444d3d Update blockchain.cpp
Fix compilation error
2015-07-15 23:20:15 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
da1d3c01de
Experimental BDB workaround optimizations 2015-07-15 21:13:42 -07:00
Riccardo Spagni
e01d32e52d
cleaning up, removing redundant files, renaming, fixing incorrect licenses 2015-05-31 13:40:18 +02:00
Thomas Winget
7b14d4a17f
Steps toward multiple dbs available -- working
There will need to be some more refactoring for these changes to be
considered complete/correct, but for now it's working.

new daemon cli argument "--db-type", works for LMDB and BerkeleyDB.

A good deal of refactoring is also present in this commit, namely
Blockchain no longer instantiates BlockchainDB, but rather is passed a
pointer to an already-instantiated BlockchainDB on init().
2015-03-25 12:09:44 -04:00
warptangent
275cbd4348
Add support for database open with flags
Add support to:
  - BlockchainDB, BlockchainLMDB
  - blockchain_import utility to open LMDB database with one or more
    LMDB flags.

Sample use:
  $ blockchain_import --database lmdb#nosync
  $ blockchain_import --database lmdb#nosync,nometasync
2015-03-16 00:26:59 -07:00
Thomas Winget
eee3ee7073
BlockchainDB implementations have names now
In order to make things more general, BlockchainDB now has get_db_name()
which should return a string with the "name" of that type of db.
This "name" will be the subfolder name that holds that db type's files
within the monero folder.

Small bugfix: blockchain_converter was not correctly appending this in
the prior hard-coded-string implementation of the subfolder data
directory concept.
2015-03-13 21:39:27 -04:00
Thomas Winget
5eab480cb1
Moved BlockchainDB into its own src/ subfolder
Ostensibly janitorial work, but should be more relevant later down the
line.  Things that depend on core cryptonote things (i.e.
    cryptonote_core) don't necessarily depend on BlockchainDB and thus
have no need to have BlockchainDB baked in with them.
2015-03-06 15:20:45 -05:00
warptangent
ce71abd0fe
Move LMDB storage to subfolder 2015-02-23 00:33:37 -08:00
warptangent
59305d3137
Blockchain: match original function declaration from blockchain_storage 2015-02-23 00:33:35 -08:00
warptangent
b88ab643ca
Fix Blockchain::get_tail_id() to set parameter to last block number instead of height
This reflects the behavior of blockchain_storage::get_tail_id().

Fixes #27 so that RPC method getlastblockheader works.
2015-02-22 10:41:41 -08:00
warptangent
8bd1983cdc
Blockchain: reflect log updates from blockchain_storage
See commit 4ba680f294
2015-02-01 19:30:20 -08:00
warptangent
7f9b070165
Blockchain: reflect log and assert updates from blockchain_storage
See commit cf5a8b1d6c
2015-02-01 19:30:14 -08:00
warptangent
70342ecada
Blockchain: reflect log level of blockchain_storage
Update to match LOG_PRINT_RED_Lx statements.
See commit cf5a8b1d6c
2015-02-01 19:29:18 -08:00
warptangent
c8d27fb38d
Blockchain: reflect assert behavior of blockchain_storage for get_tx_outputs_gindexs() 2015-02-01 19:29:03 -08:00
warptangent
d00ee784db
Update recently added log statement to fix possible null dereference
This would have been triggered if function was called without fourth
parameter and ring signature check failed.
2015-02-01 19:28:58 -08:00
Thomas Winget
acd4c369e4
Should fix std::min issues related to size_t 2015-01-19 17:39:38 -05:00
warptangent
800d9b9247
Remove code previously made unused and marked unused 2015-01-14 13:41:57 -08:00
warptangent
0840c2fd7e
Fix height assertion in Blockchain::handle_alternative_block()
It expects the total number of blocks of main chain, not last block id
(off-by-one error).

This again behaves like the same height assertion done in original
implementation in blockchain_storage::handle_alternative_block().

This allows a reorganization to proceed after an alternative block has
been added.
2015-01-11 21:23:02 -08:00
warptangent
63051bea1c
Fix comparison between main and alternate chain's cumulative
difficulty.

This fixes the continual reorganization between a main and alternate
chain, using the same two latest blocks from each.

The check that cumulative difficulty of the alternate chain is bigger
than main's was not using main's last block, but incorrectly using the
passed-in block's previous block. main_chain_cumulative_difficulty was
being used in two different ways. This has been split up to keep use
of main_chain_cumulative_difficulty consistent.
2015-01-11 21:23:02 -08:00
warptangent
909ea81067
Remove a have_block() check so alternate block can be processed
Remove have_block() check from Blockchain::handle_block_to_main_chain().
Add logging to have_block().

This allows blockchain reorganization to proceed further.

have_block() check here causes an error after a blockchain reorganize
begins with error: "Attempting to add block to main chain, but it's
already either there or in an alternate chain."

While reorganizing to become the main chain, a block in the
alternative chain would be refused due to have_block() rightfully
finding it in the alternative chain. The reorganization would end in
rollback, restoring to previous blockchain.

Original implementation didn't call it here, and it doesn't appear
necessary to be called from here in this implementation either. When
needed, it appears it's called prior to handle_block_to_main_chain().
2015-01-11 21:23:02 -08:00
warptangent
1701c26750
Use block index when obtaining block's difficulty for log statement
Use last block id, not number of blocks (off-by-one error).

Fixes error at start of blockchain reorganization: "Attempt to get
cumulative difficulty from height <XXXXXX> failed -- difficulty not in
db"
2015-01-11 19:57:46 -08:00
warptangent
4eba21fd48
Fix transfers to support mixins
Implement BlockchainLMDB::get_output_global_index()
- returns global output index for a given amount and amount output
  index.

Add information to debug statement for failed ring signature check
within Blockchain::check_tx_inputs()

Fixes bitmonerod RPC call "/getrandom_outs.bin" to return correct
output keys, used in creating a transaction with mixins.

TODO: get_output_global_index() could be refactored with part of
get_output_tx_and_index() as the latter uses the former's
functionality.  Keep track of LMDB read transaction.
2015-01-09 16:34:18 -08:00
warptangent
d045dfa7ce
Fix transfers (without mixins)
Fix Blockchain::get_tx_outputs_gindexs() to return amount output
indices.

Implement BlockchainLMDB::get_tx_amount_output_indices() and call it
from the function instead of BlockchainLMDB::get_tx_output_indices()

Previously, Blockchain::get_tx_outputs_gindexs() was instead returning
global output indices, which are internal to LMDB databases.

Allows bitmonerod RPC /get_o_indexes.bin to return the amount output
indices as expected.

Allows simplewallet refresh to set correct amount output indices for
incoming transfers. simplewallet can now construct and send valid
transactions (currently only without mixins).

This is a fix that doesn't require altering the structure of the
current LMDB databases.

TODO:

This can be done more efficiently by adding another LMDB database
(key-value table).

It's not used during regular transaction validation by bitmonerod. I
think it's currently used only or mainly by simplewallet for just its
own incoming transactions. So the current behavior is not a primary
bottleneck.

Currently, it's using the "output_amounts" database, walking through a
given amount's list of values, comparing each one to a given global
output index. The iteration number of the match is the desired result:
the amount output index. This is done for each global output index of
the transaction.

A tx's amount output indices can be stored in various other ways
allowing for faster lookup. Since a tx is only written once, there are
no special future write requirements for its list of indices.
2015-01-09 16:34:12 -08:00
Thomas Winget
14555eefd5
Fixes segfault in Blockchain::handle_alternative_block
This commit should fix the segfault in
Blockchain::handle_alternative_block, and also updates a few comments
that were either incorrect or incomplete.
2015-01-09 05:56:51 -05:00
warptangent
c5c100c69b Obtain tx hash and tx output index from amount and output offset
Fixes problem of obtaining incorrect outputs used for tx input.

Reverts to earlier intended behavior that was fixed in previous
commit's split of get_output_tx_and_index into two functions.
2015-01-04 19:39:43 -08:00
Thomas Winget
c50cd95674 Fixes a bug with getting output metadata from BlockchainDB
Thanks to moneromooo-monero for spotting the bug.
2015-01-04 19:39:43 -08:00
moneromooo-monero
1860658eec blockchain: do not append "testnet" to the data directory
It is already there (unless overridden via command line).
2015-01-04 19:39:42 -08:00
moneromooo-monero
198368b2e1 blockchain: fix wallet syncing from scratch
When the wallet syncs from the first block, it is fine to start
at the genesis block.
2015-01-04 19:39:42 -08:00
moneromooo-monero
b7270ab60e blockchain: add consts where appropriate 2015-01-04 19:39:41 -08:00
Thomas Winget
215e63b79f extraneous semicolon in Blockchain::complete_timestamps_vector
credit here:
https://bitcointalk.org/index.php?topic=583449.msg9562845#msg9562845
2015-01-04 19:39:40 -08:00
Thomas Winget
6c8b8acfe4 more blockchain height-related fixes, syncing other nodes code this time 2015-01-04 19:39:40 -08:00
Thomas Winget
26a7db38eb add new checkpointing behavior to Blockchain class 2015-01-04 19:31:20 -08:00
Thomas Winget
71b18d7166 moar bug fixes, removed debug prints 2015-01-04 19:31:19 -08:00
Thomas Winget
0915913111 BlockchainLMDB seems to be working*!
* - Well, mostly.  Haven't let it sync too far just yet.  Currently
trying to figure out the best way to deal with LMDB/mmap virtual memory
pages.
2015-01-04 19:31:19 -08:00
Thomas Winget
1a546e3222 some bug fixes, but still needs work
There are quite a few debug prints in this commit that will need removed
later, but for posterity (in case someone wants to debug this while I'm
away), I left them in.

Currently errors when syncing on the first block that has a "real"
transaction.  Seems to not be able to validate the ring signature, but I
can't for the life of me figure out what's going wrong.
2015-01-04 19:31:19 -08:00
Thomas Winget
006afe2172 Minor bug fixes and debug prints
Blockchain and BlockchainLMDB classes now have a debug print at the
beginning of each function at log level 2.  These can be removed at any
time, but for now are quite useful.

Blockchain runs, and adds the genesis block just fine, but for some
reason isn't getting new blocks.
2015-01-04 19:31:19 -08:00
Thomas Winget
90f402e258 minor fixes to Blockchain.cpp 2015-01-04 19:31:19 -08:00
Thomas Winget
74a1a89e27
Integrate BlockchainDB into cryptonote_core
Probably needs more looking at -- lot of things were done...in a rushed
sort of way.  That said, it all builds and *should* be at least
testable.

update for rebase (warptangent 2015-01-04)
  fix conflicts with upstream CMakeLists.txt files

  src/CMakeLists.txt (remove edits from original commit)
  tests/CMakeLists.txt (remove edits from original commit)
  src/cryptonote_core/CMakeLists.txt (edit)
  - use blockchain db .cpp and .h files
  - add LMDB_LIBRARIES
2015-01-04 19:29:51 -08:00
Thomas Winget
bc44bc19f4 Initial commit of BlockchainDB tests, other misc
miscellaneous changes to BlockchainDB/blockchain as well, namely
replacing instances of std::list with std::vector
2015-01-04 18:41:44 -08:00
Thomas Winget
07733f98c0 update new blockchain to build with new changes
Still need to add in the new checkpointing functionality, as well as
touch up a few things, but is okay for now.
2015-01-04 18:41:44 -08:00
Thomas Winget
1ffbeb2d2e stupid past me, fixing typos and shit... 2015-01-04 18:41:44 -08:00
Thomas Winget
aba548cbf7 import of BlockchainDB files
tried rebasing, tree-filter, and many other things.  at this point,
the history of these files previous to this can live on
in my bc2 branch, as I'm importing them as-is to here.
2015-01-04 18:41:43 -08:00