LMDB implementation code copy/paste/modified into the Berkeley DB
implementation. Need to test if it builds, then if it works, and so on,
but the code is all there.
Add support to:
- BlockchainDB, BlockchainLMDB
- blockchain_import utility to open LMDB database with one or more
LMDB flags.
Sample use:
$ blockchain_import --database lmdb#nosync
$ blockchain_import --database lmdb#nosync,nometasync
This imports to the blockchain database from an exported blockchain
file.
It can be used to bootstrap a new database or to add blocks to an
existing one.
Supports:
- both the in-memory and LMDB implementations
- optional: batching, verification, testnet
See help for usage.
Based on work by tomerkon.
See https://github.com/tomerkon
src/cryptonote_core/bootfileloader.{h,cpp}
Add log level support.
Add testnet support.
Add command-line options:
--help
--data-dir
--testnet-data-dir
--testnet
--log-level
--batch
--batch-size
--block-number
See help for usage. Run at log level 1 to see profiling stats.
In order to make things more general, BlockchainDB now has get_db_name()
which should return a string with the "name" of that type of db.
This "name" will be the subfolder name that holds that db type's files
within the monero folder.
Small bugfix: blockchain_converter was not correctly appending this in
the prior hard-coded-string implementation of the subfolder data
directory concept.
Ostensibly janitorial work, but should be more relevant later down the
line. Things that depend on core cryptonote things (i.e.
cryptonote_core) don't necessarily depend on BlockchainDB and thus
have no need to have BlockchainDB baked in with them.
+toc -doc -drmonero
Fixed the windows path, and improved logging and data
(for graph) logging, fixed some locks and added more checks.
Still there is a locking error,
not added by my patches, but present in master version
(locking of map/list of peers).
This was changed because sometimes the daemon does not complete its exit
routine with this method, but as it does correctly wind most things down
even if it gets stuck I've changed it back.
The RPC calls the daemon executable uses to talk to the running daemon
instance have mostly been added back in. Rate limiting has not been
added in upstream, but is on its way in a separate effort, so those
calls are still NOPed out.
many RPC functions added by the daemonize changes
(and related changes on the upstream dev branch that were not merged)
were commented out (apart from return). Other than that, this *should*
work...at any rate, it builds, and that's something.
new update of the pr with network limits
more debug options:
discarding downloaded blocks all or after given height.
trying to trigger the locking errors.
debug levels polished/tuned to sane values.
debug/logging improved.
warning: this pr should be correct code, but it could make
an existing (in master version) locking error appear more often.
it's a race on the list (map) of peers, e.g. between closing/deleting
them versus working on them in net-limit sleep in sending chunk.
the bug is not in this code/this pr, but in the master version.
the locking problem of master will be fixed in other pr.
problem is ub, and in practice is seems to usually cause program abort
(tested on debian stable with updated gcc). see --help for option
to add sleep to trigger the error faster.
Update of the PR with network limits
works very well for all speeds
(but remember that low download speed can stop upload
because we then slow down downloading of blockchain
requests too)
more debug options
fixed pedantic warnings in our code
should work again on Mac OS X and FreeBSD
fixed warning about size_t
tested on Debian, Ubuntu, Windows(testing now)
TCP options and ToS (QoS) flag
FIXED peer number limit
FIXED some spikes in ingress/download
FIXED problems when other up and down limit
commands and options for network limiting
works very well e.g. for 50 KiB/sec up and down
ToS (QoS) flag
peer number limit
TODO some spikes in ingress/download
TODO problems when other up and down limit
added "otshell utils" - simple logging (with colors, text files channels)
Usage:
default is lmdb for blockchain branch:
$ make release
same as:
$ DATABASE=lmdb make release
for original in-memory implementation:
$ DATABASE=memory make release
Use copied value of seed node index during thread creation, not
reference.
- fixes segfault
Use boost:🧵:try_join_until() instead of an atomic flag result
variable for each thread.
Add and handle interrupt for thread timeout.
- fixes segfault where a thread exceeds requested timeout and tries to
assign results to a referenced, but now out-of-scope, variable in
the main thread.
It expects the total number of blocks of main chain, not last block id
(off-by-one error).
This again behaves like the same height assertion done in original
implementation in blockchain_storage::handle_alternative_block().
This allows a reorganization to proceed after an alternative block has
been added.
difficulty.
This fixes the continual reorganization between a main and alternate
chain, using the same two latest blocks from each.
The check that cumulative difficulty of the alternate chain is bigger
than main's was not using main's last block, but incorrectly using the
passed-in block's previous block. main_chain_cumulative_difficulty was
being used in two different ways. This has been split up to keep use
of main_chain_cumulative_difficulty consistent.
Remove have_block() check from Blockchain::handle_block_to_main_chain().
Add logging to have_block().
This allows blockchain reorganization to proceed further.
have_block() check here causes an error after a blockchain reorganize
begins with error: "Attempting to add block to main chain, but it's
already either there or in an alternate chain."
While reorganizing to become the main chain, a block in the
alternative chain would be refused due to have_block() rightfully
finding it in the alternative chain. The reorganization would end in
rollback, restoring to previous blockchain.
Original implementation didn't call it here, and it doesn't appear
necessary to be called from here in this implementation either. When
needed, it appears it's called prior to handle_block_to_main_chain().
Complete method BlockchainLMDB::remove_output()
- use output index as the key for:
m_output_indices, m_output_txs, m_output_keys
- call new method BlockchainLMDB::remove_amount_output_index()
Add method to remove amount output index.
- BlockchainLMDB::remove_amount_output_index()
- for m_output_amounts
This also fixes the segfault when blockchain reorganization is
attempted.
Use last block id, not number of blocks (off-by-one error).
Fixes error at start of blockchain reorganization: "Attempt to get
cumulative difficulty from height <XXXXXX> failed -- difficulty not in
db"
Implement BlockchainLMDB::get_output_global_index()
- returns global output index for a given amount and amount output
index.
Add information to debug statement for failed ring signature check
within Blockchain::check_tx_inputs()
Fixes bitmonerod RPC call "/getrandom_outs.bin" to return correct
output keys, used in creating a transaction with mixins.
TODO: get_output_global_index() could be refactored with part of
get_output_tx_and_index() as the latter uses the former's
functionality. Keep track of LMDB read transaction.
Fix Blockchain::get_tx_outputs_gindexs() to return amount output
indices.
Implement BlockchainLMDB::get_tx_amount_output_indices() and call it
from the function instead of BlockchainLMDB::get_tx_output_indices()
Previously, Blockchain::get_tx_outputs_gindexs() was instead returning
global output indices, which are internal to LMDB databases.
Allows bitmonerod RPC /get_o_indexes.bin to return the amount output
indices as expected.
Allows simplewallet refresh to set correct amount output indices for
incoming transfers. simplewallet can now construct and send valid
transactions (currently only without mixins).
This is a fix that doesn't require altering the structure of the
current LMDB databases.
TODO:
This can be done more efficiently by adding another LMDB database
(key-value table).
It's not used during regular transaction validation by bitmonerod. I
think it's currently used only or mainly by simplewallet for just its
own incoming transactions. So the current behavior is not a primary
bottleneck.
Currently, it's using the "output_amounts" database, walking through a
given amount's list of values, comparing each one to a given global
output index. The iteration number of the match is the desired result:
the amount output index. This is done for each global output index of
the transaction.
A tx's amount output indices can be stored in various other ways
allowing for faster lookup. Since a tx is only written once, there are
no special future write requirements for its list of indices.
As it is useful for functions calling BlockchainDB functions to know
whether an exception is expected (attempting to get a block that doesn't
exist and counting it missing if not, to save time checking if it
does, for example), the inline functions throw{0,1} need to keep the
exception type information.
Slight comment update due to copy/paste failure.
1b46226 std::atomic_flag has no copy/move constructor, can't have a vector (Thomas Winget)
df53c0a small typo in previous commit (Thomas Winget)
4a53898 DNS seed timeout and fallback (Thomas Winget)
Fixes problem of obtaining incorrect outputs used for tx input.
Reverts to earlier intended behavior that was fixed in previous
commit's split of get_output_tx_and_index into two functions.
Ideally, the log would go in the exception's ctor, but
two log levels are used, so I'd need to specify the level
in the ctor, which isn't great as it's not really related
to the exception.
hard-coded config folder, hard-coded BlockchainDB subclass.
Needs finessing, but should be testable this way.
update for rebase (warptangent 2015-01-04)
fix conflicts with upstream CMakeLists.txt files
src/CMakeLists.txt (edit original commit)
src/blockchain_converter/CMakeLists.txt (add)
There are quite a few debug prints in this commit that will need removed
later, but for posterity (in case someone wants to debug this while I'm
away), I left them in.
Currently errors when syncing on the first block that has a "real"
transaction. Seems to not be able to validate the ring signature, but I
can't for the life of me figure out what's going wrong.
Blockchain and BlockchainLMDB classes now have a debug print at the
beginning of each function at log level 2. These can be removed at any
time, but for now are quite useful.
Blockchain runs, and adds the genesis block just fine, but for some
reason isn't getting new blocks.
Probably needs more looking at -- lot of things were done...in a rushed
sort of way. That said, it all builds and *should* be at least
testable.
update for rebase (warptangent 2015-01-04)
fix conflicts with upstream CMakeLists.txt files
src/CMakeLists.txt (remove edits from original commit)
tests/CMakeLists.txt (remove edits from original commit)
src/cryptonote_core/CMakeLists.txt (edit)
- use blockchain db .cpp and .h files
- add LMDB_LIBRARIES
All of the functionality for the LMDB implementation of BlockchainDB is
implemented, but only what is in tests/unit_tests/BlockchainDB.cpp has
been tested. This is basically add a block, see if you can get the
block and a tx from the block. More tests should be added at some
point.
Still needs testing (and need to write a few more unit tests), but
everything should be there. Lots of unfortunate duplication,
but...well, I can't see a way around it using LMDB.
A couple of other minor changes in this commit, only slightly relevant.
tried rebasing, tree-filter, and many other things. at this point,
the history of these files previous to this can live on
in my bc2 branch, as I'm importing them as-is to here.
simplewallet run without a wallet path argument should prompt again if
an invalid path was entered.
Validity here currently means the string isn't empty.
Allow pre-JSON wallet format to load without depending on existing bin
file.
Don't write bin file while inside keys rewrite, so bin file write
behavior here matches that of regular wallet load.
Fix for simplewallet loading a wallet with a keys file but no bin file.
- this situation previously required a user to restart simplewallet
before it would refresh its blockchain from the server.
f9822c4 wallet JSON update for non-deterministic wallet data (warptangent)
4c6230d Checking and handling for deterministic vs non-deterministic wallet (warptangent)
1beedb9 Extract check for deterministic keys to wallet2::is_deterministic() (warptangent)
359ede3 indentation (warptangent)
2290eff replace lines with call to recently added print_seed() (warptangent)
wallet2::store_keys() and wallet2::load_keys() should only use the JSON
attribute "seed_language" when applicable. That is only for
deterministic wallets.
- store_keys() don't add JSON attribute "seed_language" if
seed_language is empty
- load_keys() don't call set_seed_language if JSON attribute
"seed_language" not present
simple_wallet::seed()
- Check that wallet is deterministic.
simple_wallet::new_wallet()
- Prompt for seed language only if it's a non-deterministic wallet,
along with previous conditions.
simple_wallet::open_wallet()
- Fixed check for deterministic wallet (flag based on command line
non-deterministic argument was used before, but it's inapplicable to
opening an existing wallet).
- As with deterministic wallet, non-deterministic also included to be
rewritten to new JSON format file. That's what's done for newly
generated non-deterministic wallets, so old versions should be
updated to same format.
3300ae5 remove unused display variable (warptangent)
0e0e557 return true on success (warptangent)
2e11eb1 deterministic wallet use of twelve words fixed (warptangent)
21a3c46 ensure that keccak is called on view spend key, not a possibly pre-sc_reduce32 version of it - for deriving view secret key (warptangent)
f1eaf88 Prints seed after wallet upgrade. Removed iostream include. (Oran Juice)
70971be Doxygen comments (Oran Juice)
031ca23 Rewrites to old wallet file correctly (Oran Juice)
1f833dc Doxygen comments in (Oran Juice)
0bd88ff Writes seed language while generating wallet. Wallet open fix. (Oran Juice)
09a659e Stores seed language in wallet file. added rapidjson. Yet to test backward compatibility (Oran Juice)
dde7897 Disable legacy fees for now (iamsmooth)
cc74b43 Remove DEFAULT_FEE, add temporary acceptance of too-small per-kb fee >= 0.1, denominations based on DEFAULT_DUST_THRESHOLD, document fee arg to create_transactions as unused, se DEFAULT_DUST_THRESHOLD for wallet dust collection instead of calcualted tx fee (iamsmooth)
ce71c01 cmake: work around a bug with implicit link directories (Ben Boeckel)
de4fc40 mingw: copy required libraries to the build tree (Ben Boeckel)
ec54e2f cmake: place binaries together in the build tree (Ben Boeckel)
18c56ab msys: don't use LTO (Ben Boeckel)
5680c9c msys: factor out -Werror on msys (Ben Boeckel)
4751542 msys: look in msys' directory for files (Ben Boeckel)
d855fe4 miniupnpc: bump the _POSIX_C_SOURCE feature macro (Ben Boeckel)
c696492 unbound: fix getaddrinfo detection for 32-bit windows (Ben Boeckel)
e377687 cmake: Windows and static builds need this (Ben Boeckel)
01895dd cmake: fix up link lines (Ben Boeckel)
4b6515c unbound: fix type checking (Ben Boeckel)
d43a20f unbound: plumb the libdir up (Ben Boeckel)
7d708e4 cmake: support 2.8.7 (Ben Boeckel)
464c280 cmake: fix up miniupnpc's define (Ben Boeckel)
9689df9 cmake: clean up EXTRA_LIBRARIES (Ben Boeckel)
3b7bdcb cmake: set the project name (Ben Boeckel)
0f0efc4 cmake: prepend to CMAKE_MODULE_PATH (Ben Boeckel)
031e3da cmake: remove scream-make (all-caps functions) (Ben Boeckel)
abbd5c0 unbound: import cmake build system (Ben Boeckel)
e59b5b7 miniupnpc: clean up build system (Ben Boeckel)
475fe20 cmake: minor cleanups (indentation and typos) (Ben Boeckel)
799e8b2 cmake: use option() for STATIC (Ben Boeckel)
c24d22b cmake: clean up if auto-dereferencing (Ben Boeckel)
9ed415a build: inform the build of what generates version.h (Ben Boeckel)
eba180a cmake: support git info in released tarballs (Ben Boeckel)
eeffac6 cmake: fix up BOOST_IGNORE_SYSTEM_PATHS (Ben Boeckel)
a43f1a8 cmake: remove configuration variables (Ben Boeckel)
a87ce09 cmake: factor out error messages (Ben Boeckel)
9aa48b6 miniupnpc: clear out else/endfoo command arguments (Ben Boeckel)
8a86ac8 daemon_tests: update cmake code (Ben Boeckel)
fa3ff75 gtest: support an external gtest (Ben Boeckel)
7bfcffa cmake: put each test executable in its own directory (Ben Boeckel)
f53f047 cmake: handle private vs. public headers (Ben Boeckel)
55ca7d3 cmake: refactor common code with libraries (Ben Boeckel)
c773f46 cmake: refactor common code with executables (Ben Boeckel)
89cff7b cmake: put each library into its own directory (Ben Boeckel)
b94b8cd Added unit test. Fails for Japanese for some reason. (Oran Juice)
4c8a628 Remove iostream header put in during testing (Oran Juice)
9875f5b Variable unique prefix lengths for seed (Oran Juice)
b419075 added hardcoded checkpoint at block 249380 (Riccardo Spagni)
7c1d3b5 fixed miniupnpc dynamic target (Riccardo Spagni)
26728d9 mingw library tweaks (Riccardo Spagni)
2cf94c1 fix for mingw not playing nicely with libunbound configure, fix for correctly finding static libs on various operating systems (Riccardo Spagni)
b05e27f need to link with -ldl on Linux when building statically (Riccardo Spagni)
e55982a set the winsock variable differently on Windows (Riccardo Spagni)
0586a27 fixed configure_command for libunbound under mingw (Riccardo Spagni)
f8fd036 turns out mingw does actually produce .a libs and not .dll.a (Riccardo Spagni)
6be3d1d added bison/yacc static build dependency (Riccardo Spagni)
1afba73 quieten CMake when it can't find packages (Riccardo Spagni)
735a017 removed required flags from miniupnp and unbound (Riccardo Spagni)
6b7ac4b enforce static libraries when building statically (Riccardo Spagni)
1d5991e include openssl in linbunbound static linking, added errors for openssl and expat (Riccardo Spagni)
6ef8ae8 reference the correct unbound static lib (Riccardo Spagni)
23cc93c added libtool's install command (Riccardo Spagni)
409e9ba use the correct CMake variable for static builds (Riccardo Spagni)
3a0b00a build libunbound correctly (Riccardo Spagni)
75a616f fix missing parentheses (Riccardo Spagni)
59172ee build libunbound from external if no local libunbound or for static builds (Riccardo Spagni)
1dee71a added static build dependency instructions to README (Riccardo Spagni)
ff3a766 added static build dependency instructions to README (Riccardo Spagni)
5829b35 include openssl in linbunbound static linking, added errors for openssl and expat (Riccardo Spagni)
c70fa86 reference the correct unbound static lib (Riccardo Spagni)
954a244 added libtool's install command (Riccardo Spagni)
fd52015 use the correct CMake variable for static builds (Riccardo Spagni)
cc0f6a6 build libunbound correctly (Riccardo Spagni)
8c00098 fix missing parentheses (Riccardo Spagni)
cee8747 build libunbound from external if no local libunbound or for static builds (Riccardo Spagni)
add1a60 added static build dependency instructions to README (Riccardo Spagni)
46f26ff another typo fix (David G. Andersen)
ac6bc48 fix typo (David G. Andersen)
d744dd1 More documentation (David G. Andersen)
4d493f6 initial doxygen commenting of the CryptoNight proof-of-work code (David G. Andersen)
c0bdd51 Daemon should now exit on conflicting checkpoints (Thomas Winget)
f0b4138 various changes to runtime checkpoint updating (Thomas Winget)
7568f89 Fixed segfault with checkpoints loading (Thomas Winget)
b261d92 DNS checkpoint updating added, and daemon flag to enforce them (Thomas Winget)
30caebf reload checkpoints file every ~hr and print if any fail (Thomas Winget)
0e14491 updated DNSResolver/things that use it for DNSSEC (Thomas Winget)
6f2c2e1 Adding an identical existing checkpoint should not error (Thomas Winget)
If the "enforce DNS checkpoints" flag is not enabled, it should not exit
if DNS checkpoints conflict with the others, but should still print a
warning to the user.
json checkpoints will be checked every 10 minutes, dns every 60.
json checkpoints always enforced, dns still with flag.
conflicting checkpoints is hard fail, but soft if dns enforce flag not
set and dns checkpoints are wonky.
Bounds checking on blockchain_storage' m_blocks.size() when validating
against checkpoints. Also moved initial json & DNS checkpoints load to
after blockchain init.
The daemon should now check for updated checkpoints from
checkpoints.moneropulse.org as well as from the configured json file
every ~1hr (and on launch).
The daemon now has a flag to enable enforcing these checkpoints (rather
than just printing a warning when they fail).
TODO: an easily configurable list of DNS servers to check for
checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
Note: DNSResolver does not yet *use* DNSSEC, but rather this commit is
preparation for including DNSSEC validation. The function in
src/wallet/wallet2.cpp that uses DNSResolver still needs its parameters
updated accordingly.
For checkpoints being read at runtime to work correctly, the checkpoint
add code needs to not return false if a checkpoint is added that already
exists. In this case, instead return false if the checkpoint is for a
height that already has a checkpoint and the hashes are different.
ldns dependency was only still around for constants defined in ldns/rr.h,
but those constants are RFC specified DNS constants, and to reduce deps
have been replicated in dns_utils.h instead of including ldns/rr.h.
Simplewallet should now do a DNS query on an address if it fails to
convert the given string to a binary representation of an address
(base58->binary). It will prompt the user, telling of what the "url"
passed was, what monero address it translated to, and whether or not
DNSSEC validation was a success, and ask for a confirmation that these
are all acceptable.
Also implemented rudimentary IPv6 support, but commented it out because
it's not widely supported by ISPs for now, and thus is not currently
supported by Monero.
CMake config file written, but was unable to test/get it working
properly because of a bug in CMake with functions related to
find_package. Simple "-lunbound" flag used in its stead for now. May
not build on non-Linux systems, not sure yet.
On 32-bit MinGW-w64, time_t is int32_t. The existing code was serializing
time_t directly and implicitly assuming that time_t is int64_t. This commit
formalizes that assumption by serializing int64_t directly and casting to
time_t where appropriate.
Thanks go to greatwolf for reporting this issue.
monero-project/bitmonero#88
bb2b606 fix incorrect error message (obvious cut and paste bug from upstream) (iamsmooth)
6b77e83 Change wallet to not try to extract tx public key when tx has no outputs (fixes 202612 tx format messages and is otherwise correct) (iamsmooth)
08205f0 output rng fix from boolberry (iamsmooth)
if a new block has the same block id as 202612 but the wrong blobdata,
this will tell the caller that the block id is actually null_hash rather
than the 202612 block id.
Since we need to fix tree_hash, but doing so would invalidate the block
id for block 202612, this fix should check to see if we're trying to get
the block id for 202612 (if its blob hash matches) and return the "old"
block id, for backwards compatibility.
time_t is implementation-, architecture-, and apparently
compiler-dependent. As an example, on my machine if I build a 64-bit
binary, sizeof(time_t) is 8, but for a 32-bit binary it's 4. uint64_t
is therefore much more consistent for serialization, given that RPC
calls are potentially made between different machines.
The previous implementation was almost certainly a typo.
full_block_size is the maximum index in the encoded_block_sizes array,
and size is used as an index in this array. So now 1 <= size <=
full_block_size == 8 instead of 1 <= size <= sizeof(full_block_size) ==
size_of(size_t) == ? (maybe 4 on 32-bit systems!)
Per my reading this change makes sense since a subset of the exclusive
peers could be priority peers. Priority peers that are not exclusive
will not get loaded, and priority peers that *are* exclusive will get
special treatment. Prior to this change it looks like priority peers
were silently ignored when exclusive peers were provided.
wallet RPC now uses wallet2::create_transactions and wallet2::commit_tx instead
of wallet2::transfer. This made it possible to add the RPC call /transfer_split, which
will split transactions automatically if they are too large. The old call to
/transfer will return an error stating to use /transfer_split if multiple
transactions are needed to fulfill the request.
1. Added huge pages support and optimized scratchpad twiddling. (credits to dga).
2. Added aes-ni key expansion support.
3. Minor speedup to scratchpad initialization/finalization.
