Bump version to 0.20.1-dev

92b9597f8b Rename `set_current_height` to `current_height` (Alekos Filini)

Pull request description:

  ### Description

  Usually we don't have any prefix except for methods that can *add* to a list or replace the list entirely (e.g. `add_recipients` vs `set_recipients`)

  I missed this during review of #611

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

ACKs for top commit:
  danielabrozzoni:
    utACK 92b9597f8b - I'm sorry I didn't notice it!

Tree-SHA512: 3391068b2761bcd04d740ef41f9e772039fca7bc0e0736afcbc582ec74b6c91eb155d9e09dd7a07462eec29e32ac86e41ba339d9a550af3f754164cab6bdbf61

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,26 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: 'bug'
+assignees: ''
+
+---
+
+**Describe the bug**  
+<!-- A clear and concise description of what the bug is. -->
+
+**To Reproduce**  
+<!-- Steps or code to reproduce the behavior. -->
+
+**Expected behavior**  
+<!-- A clear and concise description of what you expected to happen. -->
+
+**Build environment**  
+ - BDK tag/commit: <!-- e.g. v0.13.0, 3a07614 --> 
+ - OS+version: <!-- e.g. ubuntu 20.04.01, macOS 12.0.1, windows -->  
+ - Rust/Cargo version: <!-- e.g. 1.56.0 --> 
+ - Rust/Cargo target: <!-- e.g. x86_64-apple-darwin, x86_64-unknown-linux-gnu, etc. -->  
+
+**Additional context**  
+<!-- Add any other context about the problem here. --> 

.github/ISSUE_TEMPLATE/summer_project.md

@@ -0,0 +1,77 @@
+---
+name: Summer of Bitcoin Project
+about: Template to suggest a new https://www.summerofbitcoin.org/ project.
+title: ''
+labels: 'summer-of-bitcoin'
+assignees: ''
+
+---
+
+<!--
+## Overview  
+
+Project ideas are scoped for a university-level student with a basic background in CS and bitcoin 
+fundamentals - achievable over 12-weeks. Below are just a few types of ideas: 
+
+ - Low-hanging fruit: Relatively short projects with clear goals; requires basic technical knowledge 
+   and minimal familiarity with the codebase.
+ - Core development: These projects derive from the ongoing work from the core of your development 
+   team. The list of features and bugs is never-ending, and help is always welcome.
+ - Risky/Exploratory: These projects push the scope boundaries of your development effort. They 
+   might require expertise in an area not covered by your current development team. They might take 
+   advantage of a new technology. There is a reasonable chance that the project might be less 
+   successful, but the potential rewards make it worth the attempt.
+ - Infrastructure/Automation: These projects are the code that your organization uses to get its 
+   development work done; for example, projects that improve the automation of releases, regression 
+   tests and automated builds. This is a category where a Summer of Bitcoin student can be really 
+   helpful, doing work that the development team has been putting off while they focus on core 
+   development.
+ - Quality Assurance/Testing: Projects that work on and test your project's software development 
+   process. Additionally, projects that involve a thorough test and review of individual PRs.
+ - Fun/Peripheral: These projects might not be related to the current core development focus, but 
+   create new innovations and new perspectives for your project.
+-->
+
+**Description**
+<!-- Description: 3-7 sentences describing the project background and tasks to be done. -->  
+
+**Expected Outcomes**  
+<!-- Short bullet list describing what is to be accomplished -->   
+
+**Resources**  
+<!-- 2-3 reading materials for candidate to learn about the repo, project, scope etc -->  
+<!-- Recommended reading such as a developer/contributor guide -->  
+<!-- [Another example a paper citation](https://arxiv.org/pdf/1802.08091.pdf) -->  
+<!-- [Another example an existing issue](https://github.com/opencv/opencv/issues/11013) -->  
+<!-- [An existing related module](https://github.com/opencv/opencv_contrib/tree/master/modules/optflow) -->  
+
+**Skills Required**  
+<!-- 3-4 technical skills that the candidate should know -->  
+<!-- hands on experience with git -->  
+<!-- mastery plus experience coding in C++ -->  
+<!-- basic knowledge in matrix and tensor computations, college course work in cryptography -->  
+<!-- strong mathematical background -->
+<!-- Bonus - has experience with React Native. Best if you have also worked with OSSFuzz -->
+
+**Mentor(s)**  
+<!-- names of mentor(s) for this project go here -->
+
+**Difficulty**  
+<!-- Easy, Medium, Hard -->
+
+**Competency Test (optional)**  
+<!-- 2-3 technical tasks related to the project idea or repository you’d like a candidate to 
+    perform in order to demonstrate competency, good first bugs, warm-up exercises -->
+<!-- ex. Read the instructions here to get Bitcoin core running on your machine -->
+<!-- ex. pick an issue labeled as “newcomer” in the repository, and send a merge request to the 
+    repository. You can also suggest some other improvement that we did not think of yet, or 
+    something that you find interesting or useful -->
+<!-- ex. fixes for coding style are usually easy to do, and are good issues for first time 
+    contributions for those learning how to interact with the project. After you are done with the 
+    coding style issue, try making a different contribution. -->
+<!-- ex. setup a full Debian packaging development environment and learn the basics of Debian 
+    packaging. Then identify and package the missing dependencies to package Specter Desktop -->
+<!-- ex. write a pull parser for CSV files. You'll be judged by the decisions to store the parser 
+    state and how flexible it is to wrap this parser in other scenarios. -->
+<!-- ex. Stretch Goal: Implement some basic metaprogram/app to prove you're very familiar with BDK. 
+    Be prepared to make adjustments as we judge your solution. -->

.github/workflows/code_coverage.yml

@@ -3,25 +3,35 @@ on: [push]
 name: Code Coverage
 
 jobs:
-  tarpaulin-codecov:
-    name: Tarpaulin to codecov.io
+
+  Codecov:
+    name: Code Coverage
     runs-on: ubuntu-latest
+    env:
+      CARGO_INCREMENTAL: '0'
+      RUSTFLAGS: '-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off'
+      RUSTDOCFLAGS: '-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off'
+
     steps:
       - name: Checkout
         uses: actions/checkout@v2
-
+      - name: Install rustup
+        run: curl https://sh.rustup.rs -sSf | sh -s -- -y
       - name: Set default toolchain
         run: rustup default nightly
       - name: Set profile
         run: rustup set profile minimal
+      - name: Update toolchain
+        run: rustup update
+      - name: Test
+        run: cargo test --features all-keys,compiler,esplora,ureq,compact_filters --no-default-features
+                    
+      - id: coverage
+        name: Generate coverage
+        uses: actions-rs/grcov@v0.1.5
 
-      - name: Install tarpaulin
-        run: cargo install cargo-tarpaulin
-      - name: Tarpaulin
-        run: cargo tarpaulin --features all-keys,compiler,esplora,compact_filters --run-types Tests,Doctests --exclude-files "testutils/*" --out Xml
-
-      - name: Publish to codecov.io
-        uses: codecov/codecov-action@v1.0.15
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v2
         with:
-          fail_ci_if_error: true
-          file: ./cobertura.xml
+          file: ${{ steps.coverage.outputs.report }}
+          directory: ./coverage/reports/

.github/workflows/cont_integration.yml

@@ -10,23 +10,30 @@ jobs:
     strategy:
       matrix:
         rust:
-          - stable
-          - 1.45.0 # MSRV
+          - version: 1.60.0 # STABLE
+            clippy: true
+          - version: 1.56.1 # MSRV
         features:
           - default
           - minimal
           - all-keys
-          - minimal,esplora
+          - minimal,use-esplora-ureq
           - key-value-db
           - electrum
           - compact_filters
-          - esplora,key-value-db,electrum
+          - esplora,ureq,key-value-db,electrum
           - compiler
+          - rpc
+          - verify
+          - async-interface
+          - use-esplora-reqwest
+          - sqlite
+          - sqlite-bundled
     steps:
       - name: checkout
         uses: actions/checkout@v2
       - name: Generate cache key
-        run: echo "${{ matrix.rust }} ${{ matrix.features }}" | tee .cache_key
+        run: echo "${{ matrix.rust.version }} ${{ matrix.features }}" | tee .cache_key
       - name: cache
         uses: actions/cache@v2
         with:
@@ -36,17 +43,19 @@ jobs:
             target
           key: ${{ runner.os }}-cargo-${{ hashFiles('.cache_key') }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
       - name: Set default toolchain
-        run: rustup default ${{ matrix.rust }}
+        run: rustup default ${{ matrix.rust.version }}
       - name: Set profile
         run: rustup set profile minimal
       - name: Add clippy
+        if: ${{ matrix.rust.clippy }}
         run: rustup component add clippy
       - name: Update toolchain
         run: rustup update
       - name: Build
         run: cargo build --features ${{ matrix.features }} --no-default-features
       - name: Clippy
-        run: cargo clippy --features ${{ matrix.features }} --no-default-features -- -D warnings
+        if: ${{ matrix.rust.clippy }}
+        run: cargo clippy --all-targets --features ${{ matrix.features }} --no-default-features -- -D warnings
       - name: Test
         run: cargo test --features ${{ matrix.features }} --no-default-features
 
@@ -73,16 +82,28 @@ jobs:
       - name: Test
         run: cargo test --features test-md-docs --no-default-features -- doctest::ReadmeDoctests
 
-  test-electrum:
-    name: Test electrum
-    runs-on: ubuntu-16.04
-    container: bitcoindevkit/electrs
-    env:
-      MAGICAL_RPC_AUTH: USER_PASS
-      MAGICAL_RPC_USER: admin
-      MAGICAL_RPC_PASS: passw
-      MAGICAL_RPC_URL: 127.0.0.1:18443
-      MAGICAL_ELECTRUM_URL: tcp://127.0.0.1:60401
+  test-blockchains:
+    name: Blockchain ${{ matrix.blockchain.features }}
+    runs-on: ubuntu-20.04
+    strategy:
+      fail-fast: false
+      matrix:
+        blockchain:
+          - name: electrum
+            testprefix: blockchain::electrum::test
+            features: test-electrum,verify
+          - name: rpc
+            testprefix: blockchain::rpc::test
+            features: test-rpc
+          - name: rpc-legacy
+            testprefix: blockchain::rpc::test
+            features: test-rpc-legacy
+          - name: esplora
+            testprefix: esplora
+            features: test-esplora,use-esplora-reqwest,verify
+          - name: esplora
+            testprefix: esplora
+            features: test-esplora,use-esplora-ureq,verify
     steps:
       - name: Checkout
         uses: actions/checkout@v2
@@ -94,22 +115,17 @@ jobs:
             ~/.cargo/git
             target
           key: ${{ runner.os }}-cargo-${{ github.job }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
-      - name: Install rustup
-        run: curl https://sh.rustup.rs -sSf | sh -s -- -y
-      - name: Set default toolchain
-        run: $HOME/.cargo/bin/rustup default stable
-      - name: Set profile
-        run: $HOME/.cargo/bin/rustup set profile minimal
-      - name: Update toolchain
-        run: $HOME/.cargo/bin/rustup update
-      - name: Start core
-        run: ./ci/start-core.sh
+      - name: Setup rust toolchain
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
       - name: Test
-        run: $HOME/.cargo/bin/cargo test --features test-electrum --no-default-features
+        run: cargo test --no-default-features --features ${{ matrix.blockchain.features }} ${{ matrix.blockchain.testprefix }}::bdk_blockchain_tests
 
   check-wasm:
     name: Check WASM
-    runs-on: ubuntu-16.04
+    runs-on: ubuntu-20.04
     env:
       CC: clang-10
       CFLAGS: -I/usr/include
@@ -126,11 +142,11 @@ jobs:
           key: ${{ runner.os }}-cargo-${{ github.job }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
         # Install a recent version of clang that supports wasm32
       - run: wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - || exit 1
-      - run: sudo apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" || exit 1
+      - run: sudo apt-add-repository "deb http://apt.llvm.org/focal/ llvm-toolchain-focal-10 main" || exit 1
       - run: sudo apt-get update || exit 1
-      - run: sudo apt-get install -y clang-10 libc6-dev-i386 || exit 1
+      - run: sudo apt-get install -y libclang-common-10-dev clang-10 libc6-dev-i386 || exit 1
       - name: Set default toolchain
-        run: rustup default stable
+        run: rustup default 1.56.1 # STABLE
       - name: Set profile
         run: rustup set profile minimal
       - name: Add target wasm32
@@ -138,7 +154,7 @@ jobs:
       - name: Update toolchain
         run: rustup update
       - name: Check
-        run: cargo check --target wasm32-unknown-unknown --features esplora --no-default-features
+        run: cargo check --target wasm32-unknown-unknown --features use-esplora-reqwest --no-default-features
 
   fmt:
     name: Rust fmt
@@ -147,12 +163,12 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v2
       - name: Set default toolchain
-        run: rustup default stable
+        run: rustup default nightly
       - name: Set profile
         run: rustup set profile minimal
-      - name: Add clippy
+      - name: Add rustfmt
         run: rustup component add rustfmt
       - name: Update toolchain
         run: rustup update
       - name: Check fmt
-        run: cargo fmt --all -- --check
+        run: cargo fmt --all -- --config format_code_in_doc_comments=true --check

.github/workflows/nightly_docs.yml

@@ -17,17 +17,14 @@ jobs:
             ~/.cargo/git
             target
           key: nightly-docs-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
-      - name: Install nightly toolchain
-        uses: actions-rs/toolchain@v1
-        with:
-          profile: minimal
-          toolchain: nightly
-          override: true
+      - name: Set default toolchain
+        run: rustup default nightly-2022-01-25
+      - name: Set profile
+        run: rustup set profile minimal
+      - name: Update toolchain
+        run: rustup update
       - name: Build docs
-        uses: actions-rs/cargo@v1
-        with:
-          command: rustdoc
-          args: --verbose --features=compiler,electrum,esplora,compact_filters,key-value-db,all-keys -- --cfg docsrs -Dwarnings
+        run: cargo rustdoc --verbose --features=compiler,electrum,esplora,ureq,compact_filters,key-value-db,all-keys,sqlite -- --cfg docsrs -Dwarnings
       - name: Upload artifact
         uses: actions/upload-artifact@v2
         with:
@@ -47,18 +44,18 @@ jobs:
           repository: bitcoindevkit/bitcoindevkit.org
           ref: master
       - name: Create directories
-        run: mkdir -p ./static/docs-rs/bdk/nightly
+        run: mkdir -p ./docs/.vuepress/public/docs-rs/bdk/nightly
       - name: Remove old latest
-        run: rm -rf ./static/docs-rs/bdk/nightly/latest
+        run: rm -rf ./docs/.vuepress/public/docs-rs/bdk/nightly/latest
       - name: Download built docs
         uses: actions/download-artifact@v1
         with:
           name: built-docs
-          path: ./static/docs-rs/bdk/nightly/latest
+          path: ./docs/.vuepress/public/docs-rs/bdk/nightly/latest
       - name: Configure git
         run: git config user.email "github-actions@github.com" && git config user.name "github-actions"
       - name: Commit
         continue-on-error: true         # If there's nothing to commit this step fails, but it's fine
-        run: git add ./static && git commit -m "Publish autogenerated nightly docs"
+        run: git add ./docs/.vuepress/public/docs-rs && git commit -m "Publish autogenerated nightly docs"
       - name: Push
         run: git push origin master

CHANGELOG.md

@@ -6,6 +6,186 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+
+## [v0.20.0] - [v0.19.0]
+
+- New MSRV set to `1.56.1`
+- Fee sniping discouraging through nLockTime - if the user specifies a `current_height`, we use that as a nlocktime, otherwise we use the last sync height (or 0 if we never synced)
+- Fix hang when `ElectrumBlockchainConfig::stop_gap` is zero.
+- Set coin type in BIP44, BIP49, and BIP84 templates
+- Get block hash given a block height - A `get_block_hash` method is now defined on the `GetBlockHash` trait and implemented on every blockchain backend. This method expects a block height and returns the corresponding block hash. 
+- Add `remove_partial_sigs` and `try_finalize` to `SignOptions`
+- Deprecate `AddressValidator`
+- Fix Electrum wallet sync potentially causing address index decrement - compare proposed index and current index before applying batch operations during sync.
+
+## [v0.19.0] - [v0.18.0]
+
+- added `OldestFirstCoinSelection` impl to `CoinSelectionAlgorithm`
+- New MSRV set to `1.56`
+- Unpinned tokio to `1`
+- Add traits to reuse `Blockchain`s across multiple wallets (`BlockchainFactory` and `StatelessBlockchain`).
+- Upgrade to rust-bitcoin `0.28`
+- If using the `sqlite-db` feature all cached wallet data is deleted due to a possible UTXO inconsistency, a wallet.sync will recreate it  
+- Update `PkOrF` in the policy module to become an enum
+- Add experimental support for Taproot, including:
+  - Support for `tr()` descriptors with complex tapscript trees
+  - Creation of Taproot PSBTs (BIP-371)
+  - Signing Taproot PSBTs (key spend and script spend)
+  - Support for `tr()` descriptors in the `descriptor!()` macro
+- Add support for Bitcoin Core 23.0 when using the `rpc` blockchain
+
+## [v0.18.0] - [v0.17.0]
+
+- Add `sqlite-bundled` feature for deployments that need a bundled version of sqlite, ie. for mobile platforms.
+- Added `Wallet::get_signers()`, `Wallet::descriptor_checksum()` and `Wallet::get_address_validators()`, exposed the `AsDerived` trait.
+- Deprecate `database::Database::flush()`, the function is only needed for the sled database on mobile, instead for mobile use the sqlite database.
+- Add `keychain: KeychainKind` to `wallet::AddressInfo`.
+- Improve key generation traits
+- Rename `WalletExport` to `FullyNodedExport`, deprecate the former.
+- Bump `miniscript` dependency version to `^6.1`. 
+
+## [v0.17.0] - [v0.16.1]
+
+- Removed default verification from `wallet::sync`. sync-time verification is added in `script_sync` and is activated by `verify` feature flag.
+- `verify` flag removed from `TransactionDetails`.
+- Add `get_internal_address` to allow you to get internal addresses just as you get external addresses.
+- added `ensure_addresses_cached` to `Wallet` to let offline wallets load and cache addresses in their database
+- Add `is_spent` field to `LocalUtxo`; when we notice that a utxo has been spent we set `is_spent` field to true instead of deleting it from the db.
+
+### Sync API change
+
+To decouple the `Wallet` from the `Blockchain` we've made major changes:
+
+- Removed `Blockchain` from Wallet.
+- Removed `Wallet::broadcast` (just use `Blockchain::broadcast`)
+- Deprecated `Wallet::new_offline` (all wallets are offline now)
+- Changed `Wallet::sync` to take a `Blockchain`.
+- Stop making a request for the block height when calling `Wallet:new`.
+- Added `SyncOptions` to capture extra (future) arguments to `Wallet::sync`.
+- Removed `max_addresses` sync parameter which determined how many addresses to cache before syncing since this can just be done with `ensure_addresses_cached`.
+- remove `flush` method from the `Database` trait.
+
+## [v0.16.1] - [v0.16.0]
+
+- Pin tokio dependency version to ~1.14 to prevent errors due to their new MSRV 1.49.0
+
+## [v0.16.0] - [v0.15.0]
+
+- Disable `reqwest` default features.
+- Added `reqwest-default-tls` feature: Use this to restore the TLS defaults of reqwest if you don't want to add a dependency to it in your own manifest.
+- Use dust_value from rust-bitcoin
+- Fixed generating WIF in the correct network format.
+
+## [v0.15.0] - [v0.14.0]
+
+- Overhauled sync logic for electrum and esplora.
+- Unify ureq and reqwest esplora backends to have the same configuration parameters. This means reqwest now has a timeout parameter and ureq has a concurrency parameter.
+- Fixed esplora fee estimation.
+
+## [v0.14.0] - [v0.13.0]
+
+- BIP39 implementation dependency, in `keys::bip39` changed from tiny-bip39 to rust-bip39.
+- Add new method on the `TxBuilder` to embed data in the transaction via `OP_RETURN`. To allow that a fix to check the dust only on spendable output has been introduced.
+- Update the `Database` trait to store the last sync timestamp and block height
+- Rename `ConfirmationTime` to `BlockTime`
+
+## [v0.13.0] - [v0.12.0]
+
+- Exposed `get_tx()` method from `Database` to `Wallet`.
+
+## [v0.12.0] - [v0.11.0]
+
+- Activate `miniscript/use-serde` feature to allow consumers of the library to access it via the re-exported `miniscript` crate.
+- Add support for proxies in `EsploraBlockchain`
+- Added `SqliteDatabase` that implements `Database` backed by a sqlite database using `rusqlite` crate.
+
+## [v0.11.0] - [v0.10.0]
+
+- Added `flush` method to the `Database` trait to explicitly flush to disk latest changes on the db.
+
+## [v0.10.0] - [v0.9.0]
+
+- Added `RpcBlockchain` in the `AnyBlockchain` struct to allow using Rpc backend where `AnyBlockchain` is used (eg `bdk-cli`)
+- Removed hard dependency on `tokio`.
+
+### Wallet
+
+- Removed and replaced `set_single_recipient` with more general `drain_to` and replaced `maintain_single_recipient` with `allow_shrinking`.
+
+### Blockchain
+
+- Removed `stop_gap` from `Blockchain` trait and added it to only `ElectrumBlockchain` and `EsploraBlockchain` structs.
+- Added a `ureq` backend for use when not using feature `async-interface` or target WASM. `ureq` is a blocking HTTP client.
+
+## [v0.9.0] - [v0.8.0]
+
+### Wallet
+
+- Added Bitcoin core RPC added as blockchain backend
+- Added a `verify` feature that can be enable to verify the unconfirmed txs we download against the consensus rules
+
+## [v0.8.0] - [v0.7.0]
+
+### Wallet
+- Added an option that must be explicitly enabled to allow signing using non-`SIGHASH_ALL` sighashes (#350)
+#### Changed
+`get_address` now returns an `AddressInfo` struct that includes the index and derefs to `Address`.
+
+## [v0.7.0] - [v0.6.0]
+
+### Policy
+#### Changed
+Removed `fill_satisfaction` method in favor of enum parameter in `extract_policy` method
+
+#### Added
+Timelocks are considered (optionally) in building the `satisfaction` field
+
+### Wallet
+
+- Changed `Wallet::{sign, finalize_psbt}` now take a `&mut psbt` rather than consuming it.
+- Require and validate `non_witness_utxo` for SegWit signatures by default, can be adjusted with `SignOptions`
+- Replace the opt-in builder option `force_non_witness_utxo` with the opposite `only_witness_utxo`. From now on we will provide the `non_witness_utxo`, unless explicitly asked not to.
+
+## [v0.6.0] - [v0.5.1]
+
+### Misc
+#### Changed
+- New minimum supported rust version is 1.46.0
+- Changed `AnyBlockchainConfig` to use serde tagged representation.
+
+### Descriptor
+#### Added
+- Added ability to analyze a `PSBT` to check which and how many signatures are already available
+
+### Wallet
+#### Changed
+- `get_new_address()` refactored to `get_address(AddressIndex::New)` to support different `get_address()` index selection strategies
+
+#### Added
+- Added `get_address(AddressIndex::LastUnused)` which returns the last derived address if it has not been used or if used in a received transaction returns a new address
+- Added `get_address(AddressIndex::Peek(u32))` which returns a derived address for a specified descriptor index but does not change the current index
+- Added `get_address(AddressIndex::Reset(u32))` which returns a derived address for a specified descriptor index and resets current index to the given value
+- Added `get_psbt_input` to create the corresponding psbt input for a local utxo.
+
+#### Fixed
+- Fixed `coin_select` calculation for UTXOs where `value < fee` that caused over-/underflow errors.
+
+## [v0.5.1] - [v0.5.0]
+
+### Misc
+#### Changed
+- Pin `hyper` to `=0.14.4` to make it compile on Rust 1.45
+
+## [v0.5.0] - [v0.4.0]
+
+### Misc
+#### Changed
+- Updated `electrum-client` to version `0.7`
+
+### Wallet
+#### Changed
+- `FeeRate` constructors `from_sat_per_vb` and `default_min_relay_fee` are now `const` functions
+
 ## [v0.4.0] - [v0.3.0]
 
 ### Keys
@@ -50,7 +230,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   type to mark for a missing client.
 - Upgrade `tokio` to `1.0`.
 
-#### Transaction Creation Overhaul
+### Transaction Creation Overhaul
 
 The `TxBuilder` is now created from the `build_tx` or `build_fee_bump` functions on wallet and the
 final transaction is created by calling `finish` on the builder.
@@ -61,6 +241,13 @@ final transaction is created by calling `finish` on the builder.
 - Added `Wallet::get_utxo`
 - Added `Wallet::get_descriptor_for_keychain`
 
+### `add_foreign_utxo`
+
+- Renamed `UTXO` to `LocalUtxo`
+- Added `WeightedUtxo` to replace floating `(UTXO, usize)`.
+- Added `Utxo` enum to incorporate both local utxos and foreign utxos
+- Added `TxBuilder::add_foreign_utxo` which allows adding a utxo external to the wallet.
+
 ### CLI
 #### Changed
 - Remove `cli.rs` module, `cli-utils` feature and `repl.rs` example; moved to new [`bdk-cli`](https://github.com/bitcoindevkit/bdk-cli) repository
@@ -270,8 +457,26 @@ final transaction is created by calling `finish` on the builder.
 - Use `MemoryDatabase` in the compiler example
 - Make the REPL return JSON
 
-[unreleased]: https://github.com/bitcoindevkit/bdk/compare/v0.4.0...HEAD
 [0.1.0-beta.1]: https://github.com/bitcoindevkit/bdk/compare/96c87ea5...0.1.0-beta.1
 [v0.2.0]: https://github.com/bitcoindevkit/bdk/compare/0.1.0-beta.1...v0.2.0
 [v0.3.0]: https://github.com/bitcoindevkit/bdk/compare/v0.2.0...v0.3.0
 [v0.4.0]: https://github.com/bitcoindevkit/bdk/compare/v0.3.0...v0.4.0
+[v0.5.0]: https://github.com/bitcoindevkit/bdk/compare/v0.4.0...v0.5.0
+[v0.5.1]: https://github.com/bitcoindevkit/bdk/compare/v0.5.0...v0.5.1
+[v0.6.0]: https://github.com/bitcoindevkit/bdk/compare/v0.5.1...v0.6.0
+[v0.7.0]: https://github.com/bitcoindevkit/bdk/compare/v0.6.0...v0.7.0
+[v0.8.0]: https://github.com/bitcoindevkit/bdk/compare/v0.7.0...v0.8.0
+[v0.9.0]: https://github.com/bitcoindevkit/bdk/compare/v0.8.0...v0.9.0
+[v0.10.0]: https://github.com/bitcoindevkit/bdk/compare/v0.9.0...v0.10.0
+[v0.11.0]: https://github.com/bitcoindevkit/bdk/compare/v0.10.0...v0.11.0
+[v0.12.0]: https://github.com/bitcoindevkit/bdk/compare/v0.11.0...v0.12.0
+[v0.13.0]: https://github.com/bitcoindevkit/bdk/compare/v0.12.0...v0.13.0
+[v0.14.0]: https://github.com/bitcoindevkit/bdk/compare/v0.13.0...v0.14.0
+[v0.15.0]: https://github.com/bitcoindevkit/bdk/compare/v0.14.0...v0.15.0
+[v0.16.0]: https://github.com/bitcoindevkit/bdk/compare/v0.15.0...v0.16.0
+[v0.16.1]: https://github.com/bitcoindevkit/bdk/compare/v0.16.0...v0.16.1
+[v0.17.0]: https://github.com/bitcoindevkit/bdk/compare/v0.16.1...v0.17.0
+[v0.18.0]: https://github.com/bitcoindevkit/bdk/compare/v0.17.0...v0.18.0
+[v0.19.0]: https://github.com/bitcoindevkit/bdk/compare/v0.18.0...v0.19.0
+[v0.20.0]: https://github.com/bitcoindevkit/bdk/compare/v0.19.0...v0.20.0
+[unreleased]: https://github.com/bitcoindevkit/bdk/compare/v0.20.0...HEAD

CONTRIBUTING.md

@@ -46,7 +46,7 @@ Every new feature should be covered by functional tests where possible.
 When refactoring, structure your PR to make it easy to review and don't
 hesitate to split it into multiple small, focused PRs.
 
-The Minimal Supported Rust Version is 1.45 (enforced by our CI).
+The Minimal Supported Rust Version is 1.46 (enforced by our CI).
 
 Commits should cover both the issue fixed and the solution's rationale.
 These [guidelines](https://chris.beams.io/posts/git-commit/) should be kept in mind.
@@ -57,6 +57,21 @@ comment suggesting that you're working on it. If someone is already assigned,
 don't hesitate to ask if the assigned party or previous commenters are still
 working on it if it has been awhile.
 
+Deprecation policy
+------------------
+
+Where possible, breaking existing APIs should be avoided. Instead, add new APIs and
+use [`#[deprecated]`](https://github.com/rust-lang/rfcs/blob/master/text/1270-deprecation.md)
+to discourage use of the old one.
+
+Deprecated APIs are typically maintained for one release cycle. In other words, an
+API that has been deprecated with the 0.10 release can be expected to be removed in the
+0.11 release. This allows for smoother upgrades without incurring too much technical
+debt inside this library.
+
+If you deprecated an API as part of a contribution, we encourage you to "own" that API
+and send a follow-up to remove it as part of the next release cycle.
+
 Peer review
 -----------
 

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "bdk"
-version = "0.4.1-dev"
+version = "0.20.1-dev"
 edition = "2018"
 authors = ["Alekos Filini <alekos.filini@gmail.com>", "Riccardo Casatta <riccardo@casatta.it>"]
 homepage = "https://bitcoindevkit.org"
@@ -9,28 +9,36 @@ documentation = "https://docs.rs/bdk"
 description = "A modern, lightweight, descriptor-based wallet library"
 keywords = ["bitcoin", "wallet", "descriptor", "psbt"]
 readme = "README.md"
-license = "MIT"
+license = "MIT OR Apache-2.0"
 
 [dependencies]
-bdk-macros = "^0.3"
+bdk-macros = "^0.6"
 log = "^0.4"
-miniscript = "5.1"
-bitcoin = { version = "^0.26", features = ["use-serde"] }
+miniscript = { version = "7.0", features = ["use-serde"] }
+bitcoin = { version = "0.28.1", features = ["use-serde", "base64", "rand"] }
 serde = { version = "^1.0", features = ["derive"] }
 serde_json = { version = "^1.0" }
 rand = "^0.7"
 
 # Optional dependencies
 sled = { version = "0.34", optional = true }
-electrum-client = { version = "0.6", optional = true }
-reqwest = { version = "0.11", optional = true, features = ["json"] }
+electrum-client = { version = "0.10", optional = true }
+rusqlite = { version = "0.27.0", optional = true }
+ahash = { version = "0.7.6", optional = true }
+reqwest = { version = "0.11", optional = true, default-features = false, features = ["json"] }
+ureq = { version = "~2.2.0", features = ["json"], optional = true }
 futures = { version = "0.3", optional = true }
 async-trait = { version = "0.1", optional = true }
-rocksdb = { version = "0.14", optional = true }
+rocksdb = { version = "0.14", default-features = false, features = ["snappy"], optional = true }
 cc = { version = ">=1.0.64", optional = true }
 socks = { version = "0.3", optional = true }
 lazy_static = { version = "1.4", optional = true }
-tiny-bip39 = { version = "^0.8", optional = true }
+
+bip39 = { version = "1.0.1", optional = true }
+bitcoinconsensus = { version = "0.19.0-3", optional = true }
+
+# Needed by bdk_blockchain_tests macro and the `rpc` feature
+bitcoincore-rpc = { version = "0.15", optional = true }
 
 # Platform-specific dependencies
 [target.'cfg(not(target_arch = "wasm32"))'.dependencies]
@@ -44,28 +52,53 @@ rand = { version = "^0.7", features = ["wasm-bindgen"] }
 [features]
 minimal = []
 compiler = ["miniscript/compiler"]
+verify = ["bitcoinconsensus"]
 default = ["key-value-db", "electrum"]
-electrum = ["electrum-client"]
-esplora = ["reqwest", "futures"]
+sqlite = ["rusqlite", "ahash"]
+sqlite-bundled = ["sqlite", "rusqlite/bundled"]
 compact_filters = ["rocksdb", "socks", "lazy_static", "cc"]
 key-value-db = ["sled"]
-async-interface = ["async-trait"]
 all-keys = ["keys-bip39"]
-keys-bip39 = ["tiny-bip39"]
+keys-bip39 = ["bip39"]
+rpc = ["bitcoincore-rpc"]
+
+# We currently provide mulitple implementations of `Blockchain`, all are
+# blocking except for the `EsploraBlockchain` which can be either async or
+# blocking, depending on the HTTP client in use.
+#
+# - Users wanting asynchronous HTTP calls should enable `async-interface` to get
+#   access to the asynchronous method implementations. Then, if Esplora is wanted,
+#   enable `esplora` AND `reqwest` (`--features=use-esplora-reqwest`).
+# - Users wanting blocking HTTP calls can use any of the other blockchain
+#   implementations (`compact_filters`, `electrum`, or `esplora`). Users wanting to
+#   use Esplora should enable `esplora` AND `ureq`  (`--features=use-esplora-ureq`).
+#
+# WARNING: Please take care with the features below, various combinations will
+# fail to build. We cannot currently build `bdk` with `--all-features`.
+async-interface = ["async-trait"]
+electrum = ["electrum-client"]
+# MUST ALSO USE `--no-default-features`.
+use-esplora-reqwest = ["esplora", "reqwest", "reqwest/socks", "futures"]
+use-esplora-ureq = ["esplora", "ureq", "ureq/socks"]
+# Typical configurations will not need to use `esplora` feature directly.
+esplora = []
+
+# Use below feature with `use-esplora-reqwest` to enable reqwest default TLS support
+reqwest-default-tls = ["reqwest/default-tls"]
 
 # Debug/Test features
-debug-proc-macros = ["bdk-macros/debug", "bdk-testutils-macros/debug"]
-test-electrum = ["electrum"]
+test-blockchains = ["bitcoincore-rpc", "electrum-client"]
+test-electrum = ["electrum", "electrsd/electrs_0_8_10", "electrsd/bitcoind_22_0", "test-blockchains"]
+test-rpc = ["rpc", "electrsd/electrs_0_8_10", "electrsd/bitcoind_22_0", "test-blockchains"]
+test-rpc-legacy = ["rpc", "electrsd/electrs_0_8_10", "electrsd/bitcoind_0_20_0", "test-blockchains"]
+test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "electrsd/bitcoind_22_0", "test-blockchains"]
 test-md-docs = ["electrum"]
 
 [dev-dependencies]
-bdk-testutils = "^0.3"
-bdk-testutils-macros = "^0.3"
-serial_test = "0.4"
 lazy_static = "1.4"
 env_logger = "0.7"
-base64 = "^0.11"
 clap = "2.33"
+electrsd = "0.19.1"
 
 [[example]]
 name = "address_validator"
@@ -78,12 +111,14 @@ name = "miniscriptc"
 path = "examples/compiler.rs"
 required-features = ["compiler"]
 
-[workspace]
-members = ["macros", "testutils", "testutils-macros"]
+[[example]]
+name = "rpcwallet"
+path = "examples/rpcwallet.rs"
+required-features = ["keys-bip39", "key-value-db", "rpc"]
 
-# Generate docs with nightly to add the "features required" badge
-# https://stackoverflow.com/questions/61417452/how-to-get-a-feature-requirement-tag-in-the-documentation-generated-by-cargo-do
+[workspace]
+members = ["macros"]
 [package.metadata.docs.rs]
-features = ["compiler", "electrum", "esplora", "compact_filters", "key-value-db", "all-keys"]
+features = ["compiler", "electrum", "esplora", "ureq", "compact_filters", "rpc", "key-value-db", "sqlite", "all-keys", "verify"]
 # defines the configuration attribute `docsrs`
 rustdoc-args = ["--cfg", "docsrs"]

DEVELOPMENT_CYCLE.md

@@ -20,7 +20,7 @@ As soon as the release is tagged and published, the `release` branch will be mer
 
 ## Making the Release
 
-What follows are notes and procedures that maintaners can refer to when making releases. All the commits and tags must be signed and, ideally, also [timestamped](https://github.com/opentimestamps/opentimestamps-client/blob/master/doc/git-integration.md).
+What follows are notes and procedures that maintainers can refer to when making releases. All the commits and tags must be signed and, ideally, also [timestamped](https://github.com/opentimestamps/opentimestamps-client/blob/master/doc/git-integration.md).
 
 Pre-`v1.0.0` our "major" releases only affect the "minor" semver value. Accordingly, our "minor" releases will only affect the "patch" value.
 
@@ -32,14 +32,15 @@ Pre-`v1.0.0` our "major" releases only affect the "minor" semver value. Accordin
     - If it's a minor issue you can just fix it in the release branch, since it will be merged back to `master` eventually
     - For bigger issues you can fix them on `master` and then *cherry-pick* the commit to the release branch
 6. Update the changelog with the new release version.
-7. Update `src/lib.rs` with the new version (line ~59)
+7. Update `src/lib.rs` with the new version (line ~43)
 8. On release day, make a commit on the release branch to bump the version to `x.y.z`. The message should be "Bump version to x.y.z".
 9. Add a tag to this commit. The tag name should be `vx.y.z` (for example `v0.5.0`), and the message "Release x.y.z". Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
 10. Push the new commits to the upstream release branch, wait for the CI to finish one last time.
 11. Publish **all** the updated crates to crates.io.
 12. Make a new commit to bump the version value to `x.y.(z+1)-dev`. The message should be "Bump version to x.y.(z+1)-dev".
 13. Merge the release branch back into `master`.
-14. Create the release on GitHub: go to "tags", click on the dots on the right and select "Create Release". Then set the title to `vx.y.z` and write down some brief release notes.
-15. Make sure the new release shows up on crates.io and that the docs are built correctly on docs.rs.
-16. Announce the release on Twitter, Discord and Telegram.
-17. Celebrate :tada:
+14. If the `master` branch contains any unreleased changes to the `bdk-macros` crate, change the `bdk` Cargo.toml `[dependencies]` to point to the local path (ie. `bdk-macros = { path = "./macros"}`)
+15. Create the release on GitHub: go to "tags", click on the dots on the right and select "Create Release". Then set the title to `vx.y.z` and write down some brief release notes.
+16. Make sure the new release shows up on crates.io and that the docs are built correctly on docs.rs.
+17. Announce the release on Twitter, Discord and Telegram.
+18. Celebrate :tada:

LICENSE

@@ -1,21 +1,14 @@
-MIT License
+This software is licensed under [Apache 2.0](LICENSE-APACHE) or
+[MIT](LICENSE-MIT), at your option.
 
-Copyright (c) 2020 Magical Bitcoin
+Some files retain their own copyright notice, however, for full authorship
+information, see version control history.
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+Except as otherwise noted in individual files, all files in this repository are
+licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+http://www.apache.org/licenses/LICENSE-2.0> or the MIT license <LICENSE-MIT or
+http://opensource.org/licenses/MIT>, at your option.
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+You may not use, copy, modify, merge, publish, distribute, sublicense, and/or
+sell copies of this software or any files in this repository except in
+accordance with one or both of these licenses.

LICENSE-APACHE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

LICENSE-MIT

@@ -0,0 +1,16 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -1,7 +1,7 @@
 <div align="center">
   <h1>BDK</h1>
 
-  <img src="./static/bdk.svg" width="220" />
+  <img src="./static/bdk.png" width="220" />
 
   <p>
     <strong>A modern, lightweight, descriptor-based wallet library written in Rust!</strong>
@@ -9,11 +9,11 @@
 
   <p>
     <a href="https://crates.io/crates/bdk"><img alt="Crate Info" src="https://img.shields.io/crates/v/bdk.svg"/></a>
-    <a href="https://github.com/bitcoindevkit/bdk/blob/master/LICENSE"><img alt="MIT Licensed" src="https://img.shields.io/badge/license-MIT-blue.svg"/></a>
+    <a href="https://github.com/bitcoindevkit/bdk/blob/master/LICENSE"><img alt="MIT or Apache-2.0 Licensed" src="https://img.shields.io/badge/license-MIT%2FApache--2.0-blue.svg"/></a>
     <a href="https://github.com/bitcoindevkit/bdk/actions?query=workflow%3ACI"><img alt="CI Status" src="https://github.com/bitcoindevkit/bdk/workflows/CI/badge.svg"></a>
     <a href="https://codecov.io/gh/bitcoindevkit/bdk"><img src="https://codecov.io/gh/bitcoindevkit/bdk/branch/master/graph/badge.svg"/></a>
     <a href="https://docs.rs/bdk"><img alt="API Docs" src="https://img.shields.io/badge/docs.rs-bdk-green"/></a>
-    <a href="https://blog.rust-lang.org/2020/07/16/Rust-1.45.0.html"><img alt="Rustc Version 1.45+" src="https://img.shields.io/badge/rustc-1.45%2B-lightgrey.svg"/></a>
+    <a href="https://blog.rust-lang.org/2021/11/01/Rust-1.56.1.html"><img alt="Rustc Version 1.56.1+" src="https://img.shields.io/badge/rustc-1.56.1%2B-lightgrey.svg"/></a>
     <a href="https://discord.gg/d7NkDKm"><img alt="Chat on Discord" src="https://img.shields.io/discord/753336465005608961?logo=discord"></a>
   </p>
 
@@ -41,21 +41,21 @@ The `bdk` library aims to be the core building block for Bitcoin wallets of any
 ```rust,no_run
 use bdk::Wallet;
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::{noop_progress, ElectrumBlockchain};
-
+use bdk::blockchain::ElectrumBlockchain;
+use bdk::SyncOptions;
 use bdk::electrum_client::Client;
+use bdk::bitcoin::Network;
 
 fn main() -> Result<(), bdk::Error> {
-    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let blockchain = ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
-        bitcoin::Network::Testnet,
+        Network::Testnet,
         MemoryDatabase::default(),
-        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(noop_progress(), None)?;
+    wallet.sync(&blockchain, SyncOptions::default())?;
 
     println!("Descriptor balance: {} SAT", wallet.get_balance()?);
 
@@ -67,18 +67,19 @@ fn main() -> Result<(), bdk::Error> {
 
 ```rust
 use bdk::{Wallet, database::MemoryDatabase};
+use bdk::wallet::AddressIndex::New;
 
 fn main() -> Result<(), bdk::Error> {
-    let wallet = Wallet::new_offline(
+    let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
     )?;
 
-    println!("Address #0: {}", wallet.get_new_address()?);
-    println!("Address #1: {}", wallet.get_new_address()?);
-    println!("Address #2: {}", wallet.get_new_address()?);
+    println!("Address #0: {}", wallet.get_address(New)?);
+    println!("Address #1: {}", wallet.get_address(New)?);
+    println!("Address #2: {}", wallet.get_address(New)?);
 
     Ok(())
 }
@@ -87,27 +88,27 @@ fn main() -> Result<(), bdk::Error> {
 ### Create a transaction
 
 ```rust,no_run
-use bdk::{FeeRate, Wallet};
+use bdk::{FeeRate, Wallet, SyncOptions};
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::{noop_progress, ElectrumBlockchain};
+use bdk::blockchain::ElectrumBlockchain;
 
 use bdk::electrum_client::Client;
+use bdk::wallet::AddressIndex::New;
 
 use bitcoin::consensus::serialize;
 
 fn main() -> Result<(), bdk::Error> {
-    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let blockchain = ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
-        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(noop_progress(), None)?;
+    wallet.sync(&blockchain, SyncOptions::default())?;
 
-    let send_to = wallet.get_new_address()?;
+    let send_to = wallet.get_address(New)?;
     let (psbt, details) = {
         let mut builder = wallet.build_tx();
         builder
@@ -128,12 +129,12 @@ fn main() -> Result<(), bdk::Error> {
 ### Sign a transaction
 
 ```rust,no_run
-use bdk::{Wallet, database::MemoryDatabase};
+use bdk::{Wallet, SignOptions, database::MemoryDatabase};
 
 use bitcoin::consensus::deserialize;
 
 fn main() -> Result<(), bdk::Error> {
-    let wallet = Wallet::new_offline(
+    let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
         bitcoin::Network::Testnet,
@@ -141,10 +142,46 @@ fn main() -> Result<(), bdk::Error> {
     )?;
 
     let psbt = "...";
-    let psbt = deserialize(&base64::decode(psbt).unwrap())?;
+    let mut psbt = deserialize(&base64::decode(psbt).unwrap())?;
 
-    let (signed_psbt, finalized) = wallet.sign(psbt, None)?;
+    let finalized = wallet.sign(&mut psbt, SignOptions::default())?;
 
     Ok(())
 }
 ```
+
+## Testing
+
+### Unit testing
+
+```
+cargo test
+```
+
+### Integration testing
+
+Integration testing require testing features, for example:
+
+```
+cargo test --features test-electrum
+```
+
+The other options are `test-esplora`, `test-rpc` or `test-rpc-legacy` which runs against an older version of Bitcoin Core.
+Note that `electrs` and `bitcoind` binaries are automatically downloaded (on mac and linux), to specify you already have installed binaries you must use `--no-default-features` and provide `BITCOIND_EXE` and `ELECTRS_EXE` as environment variables.
+
+## License
+
+Licensed under either of
+
+ * Apache License, Version 2.0
+   ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
+ * MIT license
+   ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
+
+at your option.
+
+## Contribution
+
+Unless you explicitly state otherwise, any contribution intentionally submitted
+for inclusion in the work by you, as defined in the Apache-2.0 license, shall be
+dual licensed as above, without any additional terms or conditions.

ci/start-core.sh

@@ -1,17 +1,14 @@
 #!/usr/bin/env sh
 
 echo "Starting bitcoin node."
-/root/bitcoind -regtest -server -daemon -fallbackfee=0.0002 -rpcuser=admin -rpcpassword=passw -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0
+mkdir $GITHUB_WORKSPACE/.bitcoin
+/root/bitcoind -regtest -server -daemon -datadir=$GITHUB_WORKSPACE/.bitcoin -fallbackfee=0.0002 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0 -blockfilterindex=1 -peerblockfilters=1
 
 echo "Waiting for bitcoin node."
-until /root/bitcoin-cli -regtest -rpcuser=admin -rpcpassword=passw getblockchaininfo; do
+until /root/bitcoin-cli -regtest -datadir=$GITHUB_WORKSPACE/.bitcoin getblockchaininfo; do
     sleep 1
 done
-
+/root/bitcoin-cli -regtest -datadir=$GITHUB_WORKSPACE/.bitcoin createwallet $BDK_RPC_WALLET
 echo "Generating 150 bitcoin blocks."
-ADDR=$(/root/bitcoin-cli -regtest -rpcuser=admin -rpcpassword=passw getnewaddress)
-/root/bitcoin-cli -regtest -rpcuser=admin -rpcpassword=passw generatetoaddress 150 $ADDR
-
-echo "Starting electrs node."
-nohup /root/electrs --network regtest --jsonrpc-import &
-sleep 5
+ADDR=$(/root/bitcoin-cli -regtest -datadir=$GITHUB_WORKSPACE/.bitcoin -rpcwallet=$BDK_RPC_WALLET getnewaddress)
+/root/bitcoin-cli -regtest -datadir=$GITHUB_WORKSPACE/.bitcoin generatetoaddress 150 $ADDR

codecov.yaml

@@ -0,0 +1,13 @@
+coverage:
+  status:
+    project:
+      default:
+        target: auto
+        threshold: 1%
+        base: auto
+        informational: false
+    patch:
+      default:
+        target: auto
+        threshold: 100%
+        base: auto

examples/address_validator.rs

@@ -1,47 +1,37 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 use std::sync::Arc;
 
 use bdk::bitcoin;
 use bdk::database::MemoryDatabase;
-use bdk::descriptor::HDKeyPaths;
+use bdk::descriptor::HdKeyPaths;
+#[allow(deprecated)]
 use bdk::wallet::address_validator::{AddressValidator, AddressValidatorError};
 use bdk::KeychainKind;
 use bdk::Wallet;
 
+use bdk::wallet::AddressIndex::New;
 use bitcoin::hashes::hex::FromHex;
 use bitcoin::util::bip32::Fingerprint;
 use bitcoin::{Network, Script};
 
 #[derive(Debug)]
 struct DummyValidator;
+#[allow(deprecated)]
 impl AddressValidator for DummyValidator {
     fn validate(
         &self,
         keychain: KeychainKind,
-        hd_keypaths: &HDKeyPaths,
+        hd_keypaths: &HdKeyPaths,
         script: &Script,
     ) -> Result<(), AddressValidatorError> {
         let (_, path) = hd_keypaths
@@ -60,14 +50,14 @@ impl AddressValidator for DummyValidator {
 
 fn main() -> Result<(), bdk::Error> {
     let descriptor = "sh(and_v(v:pk(tpubDDpWvmUrPZrhSPmUzCMBHffvC3HyMAPnWDSAQNBTnj1iZeJa7BZQEttFiP4DS4GCcXQHezdXhn86Hj6LHX5EDstXPWrMaSneRWM8yUf6NFd/*),after(630000)))";
-    let mut wallet =
-        Wallet::new_offline(descriptor, None, Network::Regtest, MemoryDatabase::new())?;
+    let mut wallet = Wallet::new(descriptor, None, Network::Regtest, MemoryDatabase::new())?;
 
+    #[allow(deprecated)]
     wallet.add_address_validator(Arc::new(DummyValidator));
 
-    wallet.get_new_address()?;
-    wallet.get_new_address()?;
-    wallet.get_new_address()?;
+    wallet.get_address(New)?;
+    wallet.get_address(New)?;
+    wallet.get_address(New)?;
 
     Ok(())
 }

examples/compact_filters_balance.rs

@@ -1,5 +1,15 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
 use bdk::blockchain::compact_filters::*;
-use bdk::blockchain::noop_progress;
 use bdk::database::MemoryDatabase;
 use bdk::*;
 use bitcoin::*;
@@ -24,9 +34,8 @@ fn main() -> Result<(), CompactFiltersError> {
     let descriptor = "wpkh(tpubD6NzVbkrYhZ4X2yy78HWrr1M9NT8dKeWfzNiQqDdMqqa9UmmGztGGz6TaLFGsLfdft5iu32gxq1T4eMNxExNNWzVCpf9Y6JZi5TnqoC9wJq/*)";
 
     let database = MemoryDatabase::default();
-    let wallet =
-        Arc::new(Wallet::new(descriptor, None, Network::Testnet, database, blockchain).unwrap());
-    wallet.sync(noop_progress(), None).unwrap();
+    let wallet = Arc::new(Wallet::new(descriptor, None, Network::Testnet, database).unwrap());
+    wallet.sync(&blockchain, SyncOptions::default()).unwrap();
     info!("balance: {}", wallet.get_balance()?);
     Ok(())
 }

examples/compiler.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 extern crate bdk;
 extern crate bitcoin;
@@ -41,6 +28,7 @@ use miniscript::policy::Concrete;
 use miniscript::Descriptor;
 
 use bdk::database::memory::MemoryDatabase;
+use bdk::wallet::AddressIndex::New;
 use bdk::{KeychainKind, Wallet};
 
 fn main() -> Result<(), Box<dyn Error>> {
@@ -82,7 +70,7 @@ fn main() -> Result<(), Box<dyn Error>> {
     let policy_str = matches.value_of("POLICY").unwrap();
     info!("Compiling policy: {}", policy_str);
 
-    let policy = Concrete::<String>::from_str(&policy_str)?;
+    let policy = Concrete::<String>::from_str(policy_str)?;
 
     let descriptor = match matches.value_of("TYPE").unwrap() {
         "sh" => Descriptor::new_sh(policy.compile()?)?,
@@ -97,13 +85,13 @@ fn main() -> Result<(), Box<dyn Error>> {
 
     let network = matches
         .value_of("network")
-        .and_then(|n| Some(Network::from_str(n)))
+        .map(Network::from_str)
         .transpose()
         .unwrap()
         .unwrap_or(Network::Testnet);
-    let wallet = Wallet::new_offline(&format!("{}", descriptor), None, network, database)?;
+    let wallet = Wallet::new(&format!("{}", descriptor), None, network, database)?;
 
-    info!("... First address: {}", wallet.get_new_address()?);
+    info!("... First address: {}", wallet.get_address(New)?);
 
     if matches.is_present("parsed_policy") {
         let spending_policy = wallet.policies(KeychainKind::External)?;

examples/rpcwallet.rs

@@ -0,0 +1,229 @@
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use bdk::bitcoin::secp256k1::Secp256k1;
+use bdk::bitcoin::Amount;
+use bdk::bitcoin::Network;
+use bdk::bitcoincore_rpc::RpcApi;
+
+use bdk::blockchain::rpc::{Auth, RpcBlockchain, RpcConfig};
+use bdk::blockchain::ConfigurableBlockchain;
+
+use bdk::keys::bip39::{Language, Mnemonic, WordCount};
+use bdk::keys::{DerivableKey, GeneratableKey, GeneratedKey};
+
+use bdk::miniscript::miniscript::Segwitv0;
+
+use bdk::sled;
+use bdk::template::Bip84;
+use bdk::wallet::{signer::SignOptions, wallet_name_from_descriptor, AddressIndex, SyncOptions};
+use bdk::KeychainKind;
+use bdk::Wallet;
+
+use bdk::blockchain::Blockchain;
+
+use electrsd;
+
+use std::error::Error;
+use std::path::PathBuf;
+use std::str::FromStr;
+
+/// This example demonstrates a typical way to create a wallet and work with bdk.
+///
+/// This example bdk wallet is connected to a bitcoin core rpc regtest node,
+/// and will attempt to receive, create and broadcast transactions.
+///
+/// To start a bitcoind regtest node programmatically, this example uses
+/// `electrsd` library, which is also a bdk dev-dependency.
+///
+/// But you can start your own bitcoind backend, and the rest of the example should work fine.
+
+fn main() -> Result<(), Box<dyn Error>> {
+    // -- Setting up background bitcoind process
+
+    println!(">> Setting up bitcoind");
+
+    // Start the bitcoind process
+    let bitcoind_conf = electrsd::bitcoind::Conf::default();
+
+    // electrsd will automatically download the bitcoin core binaries
+    let bitcoind_exe =
+        electrsd::bitcoind::downloaded_exe_path().expect("We should always have downloaded path");
+
+    // Launch bitcoind and gather authentication access
+    let bitcoind = electrsd::bitcoind::BitcoinD::with_conf(bitcoind_exe, &bitcoind_conf).unwrap();
+    let bitcoind_auth = Auth::Cookie {
+        file: bitcoind.params.cookie_file.clone(),
+    };
+
+    // Get a new core address
+    let core_address = bitcoind.client.get_new_address(None, None)?;
+
+    // Generate 101 blocks and use the above address as coinbase
+    bitcoind.client.generate_to_address(101, &core_address)?;
+
+    println!(">> bitcoind setup complete");
+    println!(
+        "Available coins in Core wallet : {}",
+        bitcoind.client.get_balance(None, None)?
+    );
+
+    // -- Setting up the Wallet
+
+    println!("\n>> Setting up BDK wallet");
+
+    // Get a random private key
+    let xprv = generate_random_ext_privkey()?;
+
+    // Use the derived descriptors from the privatekey to
+    // create unique wallet name.
+    // This is a special utility function exposed via `bdk::wallet_name_from_descriptor()`
+    let wallet_name = wallet_name_from_descriptor(
+        Bip84(xprv.clone(), KeychainKind::External),
+        Some(Bip84(xprv.clone(), KeychainKind::Internal)),
+        Network::Regtest,
+        &Secp256k1::new(),
+    )?;
+
+    // Create a database (using default sled type) to store wallet data
+    let mut datadir = PathBuf::from_str("/tmp/")?;
+    datadir.push(".bdk-example");
+    let database = sled::open(datadir)?;
+    let database = database.open_tree(wallet_name.clone())?;
+
+    // Create a RPC configuration of the running bitcoind backend we created in last step
+    // Note: If you are using custom regtest node, use the appropriate url and auth
+    let rpc_config = RpcConfig {
+        url: bitcoind.params.rpc_socket.to_string(),
+        auth: bitcoind_auth,
+        network: Network::Regtest,
+        wallet_name,
+        skip_blocks: None,
+    };
+
+    // Use the above configuration to create a RPC blockchain backend
+    let blockchain = RpcBlockchain::from_config(&rpc_config)?;
+
+    // Combine Database + Descriptor to create the final wallet
+    let wallet = Wallet::new(
+        Bip84(xprv.clone(), KeychainKind::External),
+        Some(Bip84(xprv.clone(), KeychainKind::Internal)),
+        Network::Regtest,
+        database,
+    )?;
+
+    // The `wallet` and the `blockchain` are independent structs.
+    // The wallet will be used to do all wallet level actions
+    // The blockchain can be used to do all blockchain level actions.
+    // For certain actions (like sync) the wallet will ask for a blockchain.
+
+    // Sync the wallet
+    // The first sync is important as this will instantiate the
+    // wallet files.
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> BDK wallet setup complete.");
+    println!(
+        "Available initial coins in BDK wallet : {} sats",
+        wallet.get_balance()?
+    );
+
+    // -- Wallet transaction demonstration
+
+    println!("\n>> Sending coins: Core --> BDK, 10 BTC");
+    // Get a new address to receive coins
+    let bdk_new_addr = wallet.get_address(AddressIndex::New)?.address;
+
+    // Send 10 BTC from core wallet to bdk wallet
+    bitcoind.client.send_to_address(
+        &bdk_new_addr,
+        Amount::from_btc(10.0)?,
+        None,
+        None,
+        None,
+        None,
+        None,
+        None,
+    )?;
+
+    // Confirm transaction by generating 1 block
+    bitcoind.client.generate_to_address(1, &core_address)?;
+
+    // Sync the BDK wallet
+    // This time the sync will fetch the new transaction and update it in
+    // wallet database
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> Received coins in BDK wallet");
+    println!(
+        "Available balance in BDK wallet: {} sats",
+        wallet.get_balance()?
+    );
+
+    println!("\n>> Sending coins: BDK --> Core, 5 BTC");
+    // Attempt to send back 5.0 BTC to core address by creating a transaction
+    //
+    // Transactions are created using a `TxBuilder`.
+    // This helps us to systematically build a transaction with all
+    // required customization.
+    // A full list of APIs offered by `TxBuilder` can be found at
+    // https://docs.rs/bdk/latest/bdk/wallet/tx_builder/struct.TxBuilder.html
+    let mut tx_builder = wallet.build_tx();
+
+    // For a regular transaction, just set the recipient and amount
+    tx_builder.set_recipients(vec![(core_address.script_pubkey(), 500000000)]);
+
+    // Finalize the transaction and extract the PSBT
+    let (mut psbt, _) = tx_builder.finish()?;
+
+    // Set signing option
+    let signopt = SignOptions {
+        assume_height: None,
+        ..Default::default()
+    };
+
+    // Sign the psbt
+    wallet.sign(&mut psbt, signopt)?;
+
+    // Extract the signed transaction
+    let tx = psbt.extract_tx();
+
+    // Broadcast the transaction
+    blockchain.broadcast(&tx)?;
+
+    // Confirm transaction by generating some blocks
+    bitcoind.client.generate_to_address(1, &core_address)?;
+
+    // Sync the BDK wallet
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> Coins sent to Core wallet");
+    println!(
+        "Remaining BDK wallet balance: {} sats",
+        wallet.get_balance()?
+    );
+    println!("\nCongrats!! you made your first test transaction with bdk and bitcoin core.");
+
+    Ok(())
+}
+
+// Helper function demonstrating privatekey extraction using bip39 mnemonic
+// The mnemonic can be shown to user to safekeeping and the same wallet
+// private descriptors can be recreated from it.
+fn generate_random_ext_privkey() -> Result<impl DerivableKey<Segwitv0> + Clone, Box<dyn Error>> {
+    // a Bip39 passphrase can be set optionally
+    let password = Some("random password".to_string());
+
+    // Generate a random mnemonic, and use that to create a "DerivableKey"
+    let mnemonic: GeneratedKey<_, _> = Mnemonic::generate((WordCount::Words12, Language::English))
+        .map_err(|e| e.expect("Unknown Error"))?;
+
+    // `Ok(mnemonic)` would also work if there's no passphrase and it would
+    // yield the same result as this construct with `password` = `None`.
+    Ok((mnemonic, password))
+}

macros/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "bdk-macros"
-version = "0.3.0"
+version = "0.6.0"
 authors = ["Alekos Filini <alekos.filini@gmail.com>"]
 edition = "2018"
 homepage = "https://bitcoindevkit.org"
@@ -8,7 +8,7 @@ repository = "https://github.com/bitcoindevkit/bdk"
 documentation = "https://docs.rs/bdk-macros"
 description = "Supporting macros for `bdk`"
 keywords = ["bdk"]
-license = "MIT"
+license = "MIT OR Apache-2.0"
 
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 

macros/src/lib.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 #[macro_use]
 extern crate quote;

src/blockchain/any.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Runtime-checked blockchain types
 //!
@@ -29,65 +16,32 @@
 //!
 //! ## Example
 //!
-//! In this example both `wallet_electrum` and `wallet_esplora` have the same type of
-//! `Wallet<AnyBlockchain, MemoryDatabase>`. This means that they could both, for instance, be
-//! assigned to a struct member.
-//!
-//! ```no_run
-//! # use bitcoin::Network;
-//! # use bdk::blockchain::*;
-//! # use bdk::database::MemoryDatabase;
-//! # use bdk::Wallet;
-//! # #[cfg(feature = "electrum")]
-//! # {
-//! let electrum_blockchain = ElectrumBlockchain::from(electrum_client::Client::new("...")?);
-//! let wallet_electrum: Wallet<AnyBlockchain, _> = Wallet::new(
-//!     "...",
-//!     None,
-//!     Network::Testnet,
-//!     MemoryDatabase::default(),
-//!     electrum_blockchain.into(),
-//! )?;
-//! # }
-//!
-//! # #[cfg(feature = "esplora")]
-//! # {
-//! let esplora_blockchain = EsploraBlockchain::new("...", None);
-//! let wallet_esplora: Wallet<AnyBlockchain, _> = Wallet::new(
-//!     "...",
-//!     None,
-//!     Network::Testnet,
-//!     MemoryDatabase::default(),
-//!     esplora_blockchain.into(),
-//! )?;
-//! # }
-//!
-//! # Ok::<(), bdk::Error>(())
-//! ```
-//!
-//! When paired with the use of [`ConfigurableBlockchain`], it allows creating wallets with any
+//! When paired with the use of [`ConfigurableBlockchain`], it allows creating any
 //! blockchain type supported using a single line of code:
 //!
 //! ```no_run
 //! # use bitcoin::Network;
 //! # use bdk::blockchain::*;
-//! # use bdk::database::MemoryDatabase;
-//! # use bdk::Wallet;
+//! # #[cfg(all(feature = "esplora", feature = "ureq"))]
+//! # {
 //! let config = serde_json::from_str("...")?;
 //! let blockchain = AnyBlockchain::from_config(&config)?;
-//! let wallet = Wallet::new(
-//!     "...",
-//!     None,
-//!     Network::Testnet,
-//!     MemoryDatabase::default(),
-//!     blockchain,
-//! )?;
+//! let height = blockchain.get_height();
+//! # }
 //! # Ok::<(), bdk::Error>(())
 //! ```
 
 use super::*;
 
 macro_rules! impl_from {
+    ( boxed $from:ty, $to:ty, $variant:ident, $( $cfg:tt )* ) => {
+        $( $cfg )*
+        impl From<$from> for $to {
+            fn from(inner: $from) -> Self {
+                <$to>::$variant(Box::new(inner))
+            }
+        }
+    };
     ( $from:ty, $to:ty, $variant:ident, $( $cfg:tt )* ) => {
         $( $cfg )*
         impl From<$from> for $to {
@@ -107,6 +61,8 @@ macro_rules! impl_inner_method {
             AnyBlockchain::Esplora(inner) => inner.$name( $($args, )* ),
             #[cfg(feature = "compact_filters")]
             AnyBlockchain::CompactFilters(inner) => inner.$name( $($args, )* ),
+            #[cfg(feature = "rpc")]
+            AnyBlockchain::Rpc(inner) => inner.$name( $($args, )* ),
         }
     }
 }
@@ -120,15 +76,19 @@ pub enum AnyBlockchain {
     #[cfg(feature = "electrum")]
     #[cfg_attr(docsrs, doc(cfg(feature = "electrum")))]
     /// Electrum client
-    Electrum(electrum::ElectrumBlockchain),
+    Electrum(Box<electrum::ElectrumBlockchain>),
     #[cfg(feature = "esplora")]
     #[cfg_attr(docsrs, doc(cfg(feature = "esplora")))]
     /// Esplora client
-    Esplora(esplora::EsploraBlockchain),
+    Esplora(Box<esplora::EsploraBlockchain>),
     #[cfg(feature = "compact_filters")]
     #[cfg_attr(docsrs, doc(cfg(feature = "compact_filters")))]
     /// Compact filters client
-    CompactFilters(compact_filters::CompactFiltersBlockchain),
+    CompactFilters(Box<compact_filters::CompactFiltersBlockchain>),
+    #[cfg(feature = "rpc")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "rpc")))]
+    /// RPC client
+    Rpc(Box<rpc::RpcBlockchain>),
 }
 
 #[maybe_async]
@@ -137,60 +97,105 @@ impl Blockchain for AnyBlockchain {
         maybe_await!(impl_inner_method!(self, get_capabilities))
     }
 
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(impl_inner_method!(
-            self,
-            setup,
-            stop_gap,
-            database,
-            progress_update
-        ))
-    }
-    fn sync<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(impl_inner_method!(
-            self,
-            sync,
-            stop_gap,
-            database,
-            progress_update
-        ))
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        maybe_await!(impl_inner_method!(self, get_tx, txid))
-    }
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         maybe_await!(impl_inner_method!(self, broadcast, tx))
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        maybe_await!(impl_inner_method!(self, get_height))
-    }
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         maybe_await!(impl_inner_method!(self, estimate_fee, target))
     }
 }
 
-impl_from!(electrum::ElectrumBlockchain, AnyBlockchain, Electrum, #[cfg(feature = "electrum")]);
-impl_from!(esplora::EsploraBlockchain, AnyBlockchain, Esplora, #[cfg(feature = "esplora")]);
-impl_from!(compact_filters::CompactFiltersBlockchain, AnyBlockchain, CompactFilters, #[cfg(feature = "compact_filters")]);
+#[maybe_async]
+impl GetHeight for AnyBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        maybe_await!(impl_inner_method!(self, get_height))
+    }
+}
+
+#[maybe_async]
+impl GetTx for AnyBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        maybe_await!(impl_inner_method!(self, get_tx, txid))
+    }
+}
+
+#[maybe_async]
+impl GetBlockHash for AnyBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        maybe_await!(impl_inner_method!(self, get_block_hash, height))
+    }
+}
+
+#[maybe_async]
+impl WalletSync for AnyBlockchain {
+    fn wallet_sync<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(
+            self,
+            wallet_sync,
+            database,
+            progress_update
+        ))
+    }
+
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(
+            self,
+            wallet_setup,
+            database,
+            progress_update
+        ))
+    }
+}
+
+impl_from!(boxed electrum::ElectrumBlockchain, AnyBlockchain, Electrum, #[cfg(feature = "electrum")]);
+impl_from!(boxed esplora::EsploraBlockchain, AnyBlockchain, Esplora, #[cfg(feature = "esplora")]);
+impl_from!(boxed compact_filters::CompactFiltersBlockchain, AnyBlockchain, CompactFilters, #[cfg(feature = "compact_filters")]);
+impl_from!(boxed rpc::RpcBlockchain, AnyBlockchain, Rpc, #[cfg(feature = "rpc")]);
 
 /// Type that can contain any of the blockchain configurations defined by the library
 ///
 /// This allows storing a single configuration that can be loaded into an [`AnyBlockchain`]
 /// instance. Wallets that plan to offer users the ability to switch blockchain backend at runtime
 /// will find this particularly useful.
-#[derive(Debug, serde::Serialize, serde::Deserialize)]
+///
+/// This type can be serialized from a JSON object like:
+///
+/// ```
+/// # #[cfg(feature = "electrum")]
+/// # {
+/// use bdk::blockchain::{electrum::ElectrumBlockchainConfig, AnyBlockchainConfig};
+/// let config: AnyBlockchainConfig = serde_json::from_str(
+///     r#"{
+///    "type" : "electrum",
+///    "url" : "ssl://electrum.blockstream.info:50002",
+///    "retry": 2,
+///    "stop_gap": 20
+/// }"#,
+/// )
+/// .unwrap();
+/// assert_eq!(
+///     config,
+///     AnyBlockchainConfig::Electrum(ElectrumBlockchainConfig {
+///         url: "ssl://electrum.blockstream.info:50002".into(),
+///         retry: 2,
+///         socks5: None,
+///         timeout: None,
+///         stop_gap: 20,
+///     })
+/// );
+/// # }
+/// ```
+#[derive(Debug, serde::Serialize, serde::Deserialize, Clone, PartialEq)]
+#[serde(tag = "type", rename_all = "snake_case")]
 pub enum AnyBlockchainConfig {
     #[cfg(feature = "electrum")]
     #[cfg_attr(docsrs, doc(cfg(feature = "electrum")))]
@@ -204,6 +209,10 @@ pub enum AnyBlockchainConfig {
     #[cfg_attr(docsrs, doc(cfg(feature = "compact_filters")))]
     /// Compact filters client
     CompactFilters(compact_filters::CompactFiltersBlockchainConfig),
+    #[cfg(feature = "rpc")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "rpc")))]
+    /// RPC client configuration
+    Rpc(rpc::RpcConfig),
 }
 
 impl ConfigurableBlockchain for AnyBlockchain {
@@ -213,16 +222,20 @@ impl ConfigurableBlockchain for AnyBlockchain {
         Ok(match config {
             #[cfg(feature = "electrum")]
             AnyBlockchainConfig::Electrum(inner) => {
-                AnyBlockchain::Electrum(electrum::ElectrumBlockchain::from_config(inner)?)
+                AnyBlockchain::Electrum(Box::new(electrum::ElectrumBlockchain::from_config(inner)?))
             }
             #[cfg(feature = "esplora")]
             AnyBlockchainConfig::Esplora(inner) => {
-                AnyBlockchain::Esplora(esplora::EsploraBlockchain::from_config(inner)?)
+                AnyBlockchain::Esplora(Box::new(esplora::EsploraBlockchain::from_config(inner)?))
             }
             #[cfg(feature = "compact_filters")]
-            AnyBlockchainConfig::CompactFilters(inner) => AnyBlockchain::CompactFilters(
+            AnyBlockchainConfig::CompactFilters(inner) => AnyBlockchain::CompactFilters(Box::new(
                 compact_filters::CompactFiltersBlockchain::from_config(inner)?,
-            ),
+            )),
+            #[cfg(feature = "rpc")]
+            AnyBlockchainConfig::Rpc(inner) => {
+                AnyBlockchain::Rpc(Box::new(rpc::RpcBlockchain::from_config(inner)?))
+            }
         })
     }
 }
@@ -230,3 +243,4 @@ impl ConfigurableBlockchain for AnyBlockchain {
 impl_from!(electrum::ElectrumBlockchainConfig, AnyBlockchainConfig, Electrum, #[cfg(feature = "electrum")]);
 impl_from!(esplora::EsploraBlockchainConfig, AnyBlockchainConfig, Esplora, #[cfg(feature = "esplora")]);
 impl_from!(compact_filters::CompactFiltersBlockchainConfig, AnyBlockchainConfig, CompactFilters, #[cfg(feature = "compact_filters")]);
+impl_from!(rpc::RpcConfig, AnyBlockchainConfig, Rpc, #[cfg(feature = "rpc")]);

src/blockchain/compact_filters/mod.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Compact Filters
 //!
@@ -80,11 +67,11 @@ mod peer;
 mod store;
 mod sync;
 
-use super::{Blockchain, Capability, ConfigurableBlockchain, Progress};
+use crate::blockchain::*;
 use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
 use crate::error::Error;
-use crate::types::{KeychainKind, TransactionDetails, UTXO};
-use crate::FeeRate;
+use crate::types::{KeychainKind, LocalUtxo, TransactionDetails};
+use crate::{BlockTime, FeeRate};
 
 use peer::*;
 use store::*;
@@ -159,7 +146,7 @@ impl CompactFiltersBlockchain {
         database: &mut D,
         tx: &Transaction,
         height: Option<u32>,
-        timestamp: u64,
+        timestamp: Option<u64>,
         internal_max_deriv: &mut Option<u32>,
         external_max_deriv: &mut Option<u32>,
     ) -> Result<(), Error> {
@@ -176,11 +163,19 @@ impl CompactFiltersBlockchain {
             if let Some(previous_output) = database.get_previous_output(&input.previous_output)? {
                 inputs_sum += previous_output.value;
 
-                if database.is_mine(&previous_output.script_pubkey)? {
+                // this output is ours, we have a path to derive it
+                if let Some((keychain, _)) =
+                    database.get_path_from_script_pubkey(&previous_output.script_pubkey)?
+                {
                     outgoing += previous_output.value;
 
-                    debug!("{} input #{} is mine, removing from utxo", tx.txid(), i);
-                    updates.del_utxo(&input.previous_output)?;
+                    debug!("{} input #{} is mine, setting utxo as spent", tx.txid(), i);
+                    updates.set_utxo(&LocalUtxo {
+                        outpoint: input.previous_output,
+                        txout: previous_output.clone(),
+                        keychain,
+                        is_spent: true,
+                    })?;
                 }
             }
         }
@@ -194,10 +189,11 @@ impl CompactFiltersBlockchain {
                 database.get_path_from_script_pubkey(&output.script_pubkey)?
             {
                 debug!("{} output #{} is mine, adding utxo", tx.txid(), i);
-                updates.set_utxo(&UTXO {
+                updates.set_utxo(&LocalUtxo {
                     outpoint: OutPoint::new(tx.txid(), i as u32),
                     txout: output.clone(),
                     keychain,
+                    is_spent: false,
                 })?;
                 incoming += output.value;
 
@@ -219,9 +215,8 @@ impl CompactFiltersBlockchain {
                 transaction: Some(tx.clone()),
                 received: incoming,
                 sent: outgoing,
-                height,
-                timestamp,
-                fees: inputs_sum.saturating_sub(outputs_sum),
+                confirmation_time: BlockTime::new(height, timestamp),
+                fee: Some(inputs_sum.saturating_sub(outputs_sum)),
             };
 
             info!("Saving tx {}", tx.txid);
@@ -239,17 +234,54 @@ impl Blockchain for CompactFiltersBlockchain {
         vec![Capability::FullHistory].into_iter().collect()
     }
 
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        self.peers[0].broadcast_tx(tx.clone())?;
+
+        Ok(())
+    }
+
+    fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
+        // TODO
+        Ok(FeeRate::default())
+    }
+}
+
+impl GetHeight for CompactFiltersBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.headers.get_height()? as u32)
+    }
+}
+
+impl GetTx for CompactFiltersBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.peers[0]
+            .get_mempool()
+            .get_tx(&Inventory::Transaction(*txid)))
+    }
+}
+
+impl GetBlockHash for CompactFiltersBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        self.headers
+            .get_block_hash(height as usize)?
+            .ok_or(Error::CompactFilters(
+                CompactFiltersError::BlockHashNotFound,
+            ))
+    }
+}
+
+impl WalletSync for CompactFiltersBlockchain {
+    #[allow(clippy::mutex_atomic)] // Mutex is easier to understand than a CAS loop.
+    fn wallet_setup<D: BatchDatabase>(
         &self,
-        _stop_gap: Option<usize>, // TODO: move to electrum and esplora only
         database: &mut D,
-        progress_update: P,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error> {
         let first_peer = &self.peers[0];
 
         let skip_blocks = self.skip_blocks.unwrap_or(0);
 
-        let cf_sync = Arc::new(CFSync::new(Arc::clone(&self.headers), skip_blocks, 0x00)?);
+        let cf_sync = Arc::new(CfSync::new(Arc::clone(&self.headers), skip_blocks, 0x00)?);
 
         let initial_height = self.headers.get_height()?;
         let total_bundles = (first_peer.get_version().start_height as usize)
@@ -267,7 +299,7 @@ impl Blockchain for CompactFiltersBlockchain {
         let total_cost = headers_cost + filters_cost + PROCESS_BLOCKS_COST;
 
         if let Some(snapshot) = sync::sync_headers(
-            Arc::clone(&first_peer),
+            Arc::clone(first_peer),
             Arc::clone(&self.headers),
             |new_height| {
                 let local_headers_cost =
@@ -288,7 +320,7 @@ impl Blockchain for CompactFiltersBlockchain {
         let buried_height = synced_height.saturating_sub(sync::BURIED_CONFIRMATIONS);
         info!("Synced headers to height: {}", synced_height);
 
-        cf_sync.prepare_sync(Arc::clone(&first_peer))?;
+        cf_sync.prepare_sync(Arc::clone(first_peer))?;
 
         let all_scripts = Arc::new(
             database
@@ -307,7 +339,7 @@ impl Blockchain for CompactFiltersBlockchain {
         let mut threads = Vec::with_capacity(self.peers.len());
         for peer in &self.peers {
             let cf_sync = Arc::clone(&cf_sync);
-            let peer = Arc::clone(&peer);
+            let peer = Arc::clone(peer);
             let headers = Arc::clone(&self.headers);
             let all_scripts = Arc::clone(&all_scripts);
             let last_synced_block = Arc::clone(&last_synced_block);
@@ -376,8 +408,8 @@ impl Blockchain for CompactFiltersBlockchain {
         );
         let mut updates = database.begin_batch();
         for details in database.iter_txs(false)? {
-            match details.height {
-                Some(height) if (height as usize) < last_synced_block => continue,
+            match details.confirmation_time {
+                Some(c) if (c.height as usize) < last_synced_block => continue,
                 _ => updates.del_tx(&details.txid, false)?,
             };
         }
@@ -399,7 +431,7 @@ impl Blockchain for CompactFiltersBlockchain {
                     database,
                     tx,
                     Some(height as u32),
-                    0,
+                    None,
                     &mut internal_max_deriv,
                     &mut external_max_deriv,
                 )?;
@@ -410,7 +442,7 @@ impl Blockchain for CompactFiltersBlockchain {
                 database,
                 tx,
                 None,
-                0,
+                None,
                 &mut internal_max_deriv,
                 &mut external_max_deriv,
             )?;
@@ -444,31 +476,10 @@ impl Blockchain for CompactFiltersBlockchain {
 
         Ok(())
     }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.peers[0]
-            .get_mempool()
-            .get_tx(&Inventory::Transaction(*txid)))
-    }
-
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        self.peers[0].broadcast_tx(tx.clone())?;
-
-        Ok(())
-    }
-
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(self.headers.get_height()? as u32)
-    }
-
-    fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
-        // TODO
-        Ok(FeeRate::default())
-    }
 }
 
 /// Data to connect to a Bitcoin P2P peer
-#[derive(Debug, serde::Deserialize, serde::Serialize)]
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
 pub struct BitcoinPeerConfig {
     /// Peer address such as 127.0.0.1:18333
     pub address: String,
@@ -479,13 +490,13 @@ pub struct BitcoinPeerConfig {
 }
 
 /// Configuration for a [`CompactFiltersBlockchain`]
-#[derive(Debug, serde::Deserialize, serde::Serialize)]
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
 pub struct CompactFiltersBlockchainConfig {
     /// List of peers to try to connect to for asking headers and filters
     pub peers: Vec<BitcoinPeerConfig>,
     /// Network used
     pub network: Network,
-    /// Storage dir to save partially downloaded headers and full blocks
+    /// Storage dir to save partially downloaded headers and full blocks. Should be a separate directory per descriptor. Consider using [crate::wallet::wallet_name_from_descriptor] for this.
     pub storage_dir: String,
     /// Optionally skip initial `skip_blocks` blocks (default: 0)
     pub skip_blocks: Option<usize>,
@@ -535,6 +546,8 @@ pub enum CompactFiltersError {
     InvalidFilter,
     /// The peer is missing a block in the valid chain
     MissingBlock,
+    /// Block hash at specified height not found
+    BlockHashNotFound,
     /// The data stored in the block filters storage are corrupted
     DataCorruption,
 
@@ -549,11 +562,11 @@ pub enum CompactFiltersError {
     NoPeers,
 
     /// Internal database error
-    DB(rocksdb::Error),
+    Db(rocksdb::Error),
     /// Internal I/O error
-    IO(std::io::Error),
+    Io(std::io::Error),
     /// Invalid BIP158 filter
-    BIP158(bitcoin::util::bip158::Error),
+    Bip158(bitcoin::util::bip158::Error),
     /// Internal system time error
     Time(std::time::SystemTimeError),
 
@@ -569,9 +582,9 @@ impl fmt::Display for CompactFiltersError {
 
 impl std::error::Error for CompactFiltersError {}
 
-impl_error!(rocksdb::Error, DB, CompactFiltersError);
-impl_error!(std::io::Error, IO, CompactFiltersError);
-impl_error!(bitcoin::util::bip158::Error, BIP158, CompactFiltersError);
+impl_error!(rocksdb::Error, Db, CompactFiltersError);
+impl_error!(std::io::Error, Io, CompactFiltersError);
+impl_error!(bitcoin::util::bip158::Error, Bip158, CompactFiltersError);
 impl_error!(std::time::SystemTimeError, Time, CompactFiltersError);
 
 impl From<crate::error::Error> for CompactFiltersError {

src/blockchain/compact_filters/peer.rs

@@ -1,28 +1,16 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 use std::collections::HashMap;
+use std::io::BufReader;
 use std::net::{TcpStream, ToSocketAddrs};
 use std::sync::{Arc, Condvar, Mutex, RwLock};
 use std::thread;
@@ -32,14 +20,13 @@ use socks::{Socks5Stream, ToTargetAddr};
 
 use rand::{thread_rng, Rng};
 
-use bitcoin::consensus::Encodable;
+use bitcoin::consensus::{Decodable, Encodable};
 use bitcoin::hash_types::BlockHash;
 use bitcoin::network::constants::ServiceFlags;
 use bitcoin::network::message::{NetworkMessage, RawNetworkMessage};
 use bitcoin::network::message_blockdata::*;
 use bitcoin::network::message_filter::*;
 use bitcoin::network::message_network::VersionMessage;
-use bitcoin::network::stream_reader::StreamReader;
 use bitcoin::network::Address;
 use bitcoin::{Block, Network, Transaction, Txid, Wtxid};
 
@@ -107,8 +94,7 @@ impl Mempool {
             TxIdentifier::Wtxid(wtxid) => self.0.read().unwrap().wtxids.get(&wtxid).cloned(),
         };
 
-        txid.map(|txid| self.0.read().unwrap().txs.get(&txid).cloned())
-            .flatten()
+        txid.and_then(|txid| self.0.read().unwrap().txs.get(&txid).cloned())
     }
 
     /// Return whether or not the mempool contains a transaction with a given txid
@@ -124,6 +110,7 @@ impl Mempool {
 
 /// A Bitcoin peer
 #[derive(Debug)]
+#[allow(dead_code)]
 pub struct Peer {
     writer: Arc<Mutex<TcpStream>>,
     responses: Arc<RwLock<ResponsesMap>>,
@@ -240,12 +227,12 @@ impl Peer {
 
         Ok(Peer {
             writer,
-            reader_thread,
             responses,
+            reader_thread,
             connected,
             mempool,
-            network,
             version,
+            network,
         })
     }
 
@@ -275,7 +262,7 @@ impl Peer {
         let message_resp = {
             let mut lock = responses.write().unwrap();
             let message_resp = lock.entry(wait_for).or_default();
-            Arc::clone(&message_resp)
+            Arc::clone(message_resp)
         };
 
         let (lock, cvar) = &*message_resp;
@@ -340,9 +327,10 @@ impl Peer {
             };
         }
 
-        let mut reader = StreamReader::new(connection, None);
+        let mut reader = BufReader::new(connection);
         loop {
-            let raw_message: RawNetworkMessage = check_disconnect!(reader.read_next());
+            let raw_message: RawNetworkMessage =
+                check_disconnect!(Decodable::consensus_decode(&mut reader));
 
             let in_message = if raw_message.magic != network.magic() {
                 continue;
@@ -392,7 +380,7 @@ impl Peer {
             let message_resp = {
                 let mut lock = reader_thread_responses.write().unwrap();
                 let message_resp = lock.entry(in_message.cmd()).or_default();
-                Arc::clone(&message_resp)
+                Arc::clone(message_resp)
             };
 
             let (lock, cvar) = &*message_resp;

src/blockchain/compact_filters/store.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 use std::convert::TryInto;
 use std::fmt;
@@ -46,6 +33,8 @@ use bitcoin::BlockHash;
 use bitcoin::BlockHeader;
 use bitcoin::Network;
 
+use lazy_static::lazy_static;
+
 use super::CompactFiltersError;
 
 lazy_static! {
@@ -119,7 +108,7 @@ where
     }
 
     fn deserialize(data: &[u8]) -> Result<Self, CompactFiltersError> {
-        Ok(deserialize(data).map_err(|_| CompactFiltersError::DataCorruption)?)
+        deserialize(data).map_err(|_| CompactFiltersError::DataCorruption)
     }
 }
 
@@ -131,7 +120,7 @@ impl Encodable for BundleStatus {
             BundleStatus::Init => {
                 written += 0x00u8.consensus_encode(&mut e)?;
             }
-            BundleStatus::CFHeaders { cf_headers } => {
+            BundleStatus::CfHeaders { cf_headers } => {
                 written += 0x01u8.consensus_encode(&mut e)?;
                 written += VarInt(cf_headers.len() as u64).consensus_encode(&mut e)?;
                 for header in cf_headers {
@@ -182,7 +171,7 @@ impl Decodable for BundleStatus {
                     cf_headers.push(FilterHeader::consensus_decode(&mut d)?);
                 }
 
-                Ok(BundleStatus::CFHeaders { cf_headers })
+                Ok(BundleStatus::CfHeaders { cf_headers })
             }
             0x02 => {
                 let num = VarInt::consensus_decode(&mut d)?;
@@ -409,7 +398,7 @@ impl ChainStore<Full> {
             );
         }
 
-        // Delete full blocks overriden by snapshot
+        // Delete full blocks overridden by snapshot
         let from_key = StoreEntry::Block(Some(snaphost.min_height)).get_key();
         let to_key = StoreEntry::Block(Some(usize::MAX)).get_key();
         batch.delete_range(&from_key, &to_key);
@@ -436,15 +425,14 @@ impl ChainStore<Full> {
 
         let key = StoreEntry::BlockHeaderIndex(Some(*block_hash)).get_key();
         let data = read_store.get_pinned_cf(cf_handle, key)?;
-        Ok(data
-            .map(|data| {
-                Ok::<_, CompactFiltersError>(usize::from_be_bytes(
-                    data.as_ref()
-                        .try_into()
-                        .map_err(|_| CompactFiltersError::DataCorruption)?,
-                ))
-            })
-            .transpose()?)
+        data.map(|data| {
+            Ok::<_, CompactFiltersError>(usize::from_be_bytes(
+                data.as_ref()
+                    .try_into()
+                    .map_err(|_| CompactFiltersError::DataCorruption)?,
+            ))
+        })
+        .transpose()
     }
 
     pub fn get_block_hash(&self, height: usize) -> Result<Option<BlockHash>, CompactFiltersError> {
@@ -453,13 +441,12 @@ impl ChainStore<Full> {
 
         let key = StoreEntry::BlockHeader(Some(height)).get_key();
         let data = read_store.get_pinned_cf(cf_handle, key)?;
-        Ok(data
-            .map(|data| {
-                let (header, _): (BlockHeader, Uint256) =
-                    deserialize(&data).map_err(|_| CompactFiltersError::DataCorruption)?;
-                Ok::<_, CompactFiltersError>(header.block_hash())
-            })
-            .transpose()?)
+        data.map(|data| {
+            let (header, _): (BlockHeader, Uint256) =
+                deserialize(&data).map_err(|_| CompactFiltersError::DataCorruption)?;
+            Ok::<_, CompactFiltersError>(header.block_hash())
+        })
+        .transpose()
     }
 
     pub fn save_full_block(&self, block: &Block, height: usize) -> Result<(), CompactFiltersError> {
@@ -475,10 +462,10 @@ impl ChainStore<Full> {
         let key = StoreEntry::Block(Some(height)).get_key();
         let opt_block = read_store.get_pinned(key)?;
 
-        Ok(opt_block
+        opt_block
             .map(|data| deserialize(&data))
             .transpose()
-            .map_err(|_| CompactFiltersError::DataCorruption)?)
+            .map_err(|_| CompactFiltersError::DataCorruption)
     }
 
     pub fn delete_blocks_until(&self, height: usize) -> Result<(), CompactFiltersError> {
@@ -565,14 +552,14 @@ impl<T: StoreType> ChainStore<T> {
         let prefix = StoreEntry::BlockHeader(None).get_key();
         let iterator = read_store.prefix_iterator_cf(cf_handle, prefix);
 
-        Ok(iterator
+        iterator
             .last()
             .map(|(_, v)| -> Result<_, CompactFiltersError> {
                 let (header, _): (BlockHeader, Uint256) = SerializeDb::deserialize(&v)?;
 
                 Ok(header.block_hash())
             })
-            .transpose()?)
+            .transpose()
     }
 
     pub fn apply(
@@ -636,26 +623,26 @@ impl<T: StoreType> fmt::Debug for ChainStore<T> {
 
 pub enum BundleStatus {
     Init,
-    CFHeaders { cf_headers: Vec<FilterHeader> },
+    CfHeaders { cf_headers: Vec<FilterHeader> },
     CFilters { cf_filters: Vec<Vec<u8>> },
     Processed { cf_filters: Vec<Vec<u8>> },
     Tip { cf_filters: Vec<Vec<u8>> },
     Pruned,
 }
 
-pub struct CFStore {
+pub struct CfStore {
     store: Arc<RwLock<DB>>,
     filter_type: u8,
 }
 
 type BundleEntry = (BundleStatus, FilterHeader);
 
-impl CFStore {
+impl CfStore {
     pub fn new(
         headers_store: &ChainStore<Full>,
         filter_type: u8,
     ) -> Result<Self, CompactFiltersError> {
-        let cf_store = CFStore {
+        let cf_store = CfStore {
             store: Arc::clone(&headers_store.store),
             filter_type,
         };
@@ -716,11 +703,11 @@ impl CFStore {
 
         // FIXME: we have to filter manually because rocksdb sometimes returns stuff that doesn't
         // have the right prefix
-        Ok(iterator
+        iterator
             .filter(|(k, _)| k.starts_with(&prefix))
             .skip(1)
             .map(|(_, data)| Ok::<_, CompactFiltersError>(BundleEntry::deserialize(&data)?.1))
-            .collect::<Result<_, _>>()?)
+            .collect::<Result<_, _>>()
     }
 
     pub fn replace_checkpoints(
@@ -773,7 +760,7 @@ impl CFStore {
         let cf_headers: Vec<FilterHeader> = filter_hashes
             .into_iter()
             .scan(checkpoint, |prev_header, filter_hash| {
-                let filter_header = filter_hash.filter_header(&prev_header);
+                let filter_header = filter_hash.filter_header(prev_header);
                 *prev_header = filter_header;
 
                 Some(filter_header)
@@ -795,7 +782,7 @@ impl CFStore {
         }
 
         let key = StoreEntry::CFilterTable((self.filter_type, Some(bundle))).get_key();
-        let value = (BundleStatus::CFHeaders { cf_headers }, checkpoint);
+        let value = (BundleStatus::CfHeaders { cf_headers }, checkpoint);
 
         read_store.put(key, value.serialize())?;
 
@@ -814,7 +801,7 @@ impl CFStore {
             .zip(headers.into_iter())
             .scan(checkpoint, |prev_header, ((_, filter_content), header)| {
                 let filter = BlockFilter::new(&filter_content);
-                if header != filter.filter_header(&prev_header) {
+                if header != filter.filter_header(prev_header) {
                     return Some(Err(CompactFiltersError::InvalidFilter));
                 }
                 *prev_header = header;

src/blockchain/compact_filters/sync.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 use std::collections::{BTreeMap, HashMap, VecDeque};
 use std::sync::{Arc, Mutex};
@@ -38,22 +25,22 @@ use crate::error::Error;
 
 pub(crate) const BURIED_CONFIRMATIONS: usize = 100;
 
-pub struct CFSync {
+pub struct CfSync {
     headers_store: Arc<ChainStore<Full>>,
-    cf_store: Arc<CFStore>,
+    cf_store: Arc<CfStore>,
     skip_blocks: usize,
     bundles: Mutex<VecDeque<(BundleStatus, FilterHeader, usize)>>,
 }
 
-impl CFSync {
+impl CfSync {
     pub fn new(
         headers_store: Arc<ChainStore<Full>>,
         skip_blocks: usize,
         filter_type: u8,
     ) -> Result<Self, CompactFiltersError> {
-        let cf_store = Arc::new(CFStore::new(&headers_store, filter_type)?);
+        let cf_store = Arc::new(CfStore::new(&headers_store, filter_type)?);
 
-        Ok(CFSync {
+        Ok(CfSync {
             headers_store,
             cf_store,
             skip_blocks,
@@ -164,7 +151,7 @@ impl CFSync {
                     checkpoint,
                     headers_resp.filter_hashes,
                 )? {
-                    BundleStatus::CFHeaders { cf_headers } => cf_headers,
+                    BundleStatus::CfHeaders { cf_headers } => cf_headers,
                     _ => return Err(CompactFiltersError::InvalidResponse),
                 };
 
@@ -184,7 +171,7 @@ impl CFSync {
                     .cf_store
                     .advance_to_cf_filters(index, checkpoint, cf_headers, filters)?;
             }
-            if let BundleStatus::CFHeaders { cf_headers } = status {
+            if let BundleStatus::CfHeaders { cf_headers } = status {
                 log::trace!("status: CFHeaders");
 
                 peer.get_cf_filters(
@@ -218,7 +205,7 @@ impl CFSync {
                     let block_hash = self.headers_store.get_block_hash(height)?.unwrap();
 
                     // TODO: also download random blocks?
-                    if process(&block_hash, &BlockFilter::new(&filter))? {
+                    if process(&block_hash, &BlockFilter::new(filter))? {
                         log::debug!("Downloading block {}", block_hash);
 
                         let block = peer

src/blockchain/electrum.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Electrum
 //!
@@ -37,37 +24,36 @@
 //! # Ok::<(), bdk::Error>(())
 //! ```
 
-use std::collections::HashSet;
+use std::collections::{HashMap, HashSet};
 
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
-use bitcoin::{BlockHeader, Script, Transaction, Txid};
+use bitcoin::{Transaction, Txid};
 
 use electrum_client::{Client, ConfigBuilder, ElectrumApi, Socks5Config};
 
-use self::utils::{ELSGetHistoryRes, ElectrumLikeSync};
+use super::script_sync::Request;
 use super::*;
-use crate::database::BatchDatabase;
+use crate::database::{BatchDatabase, Database};
 use crate::error::Error;
-use crate::FeeRate;
+use crate::{BlockTime, FeeRate};
 
 /// Wrapper over an Electrum Client that implements the required blockchain traits
 ///
 /// ## Example
 /// See the [`blockchain::electrum`](crate::blockchain::electrum) module for a usage example.
-pub struct ElectrumBlockchain(Client);
-
-#[cfg(test)]
-#[cfg(feature = "test-electrum")]
-#[bdk_blockchain_tests(crate)]
-fn local_electrs() -> ElectrumBlockchain {
-    ElectrumBlockchain::from(Client::new(&testutils::get_electrum_url()).unwrap())
+pub struct ElectrumBlockchain {
+    client: Client,
+    stop_gap: usize,
 }
 
 impl std::convert::From<Client> for ElectrumBlockchain {
     fn from(client: Client) -> Self {
-        ElectrumBlockchain(client)
+        ElectrumBlockchain {
+            client,
+            stop_gap: 20,
+        }
     }
 }
 
@@ -82,82 +68,226 @@ impl Blockchain for ElectrumBlockchain {
         .collect()
     }
 
-    fn setup<D: BatchDatabase, P: Progress>(
-        &self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        self.0
-            .electrum_like_setup(stop_gap, database, progress_update)
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.0.transaction_get(txid).map(Option::Some)?)
-    }
-
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        Ok(self.0.transaction_broadcast(tx).map(|_| ())?)
-    }
-
-    fn get_height(&self) -> Result<u32, Error> {
-        // TODO: unsubscribe when added to the client, or is there a better call to use here?
-
-        Ok(self
-            .0
-            .block_headers_subscribe()
-            .map(|data| data.height as u32)?)
+        Ok(self.client.transaction_broadcast(tx).map(|_| ())?)
     }
 
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         Ok(FeeRate::from_btc_per_kvb(
-            self.0.estimate_fee(target)? as f32
+            self.client.estimate_fee(target)? as f32
         ))
     }
 }
 
-impl ElectrumLikeSync for Client {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script> + Clone>(
+impl StatelessBlockchain for ElectrumBlockchain {}
+
+impl GetHeight for ElectrumBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        // TODO: unsubscribe when added to the client, or is there a better call to use here?
+
+        Ok(self
+            .client
+            .block_headers_subscribe()
+            .map(|data| data.height as u32)?)
+    }
+}
+
+impl GetTx for ElectrumBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.client.transaction_get(txid).map(Option::Some)?)
+    }
+}
+
+impl GetBlockHash for ElectrumBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        let block_header = self.client.block_header(height as usize)?;
+        Ok(block_header.block_hash())
+    }
+}
+
+impl WalletSync for ElectrumBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
         &self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ELSGetHistoryRes>>, Error> {
-        self.batch_script_get_history(scripts)
-            .map(|v| {
-                v.into_iter()
-                    .map(|v| {
-                        v.into_iter()
-                            .map(
-                                |electrum_client::GetHistoryRes {
-                                     height, tx_hash, ..
-                                 }| ELSGetHistoryRes {
-                                    height,
-                                    tx_hash,
-                                },
-                            )
-                            .collect()
-                    })
-                    .collect()
-            })
-            .map_err(Error::Electrum)
+        database: &mut D,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut block_times = HashMap::<u32, u32>::new();
+        let mut txid_to_height = HashMap::<Txid, u32>::new();
+        let mut tx_cache = TxCache::new(database, &self.client);
+
+        // Set chunk_size to the smallest value capable of finding a gap greater than stop_gap.
+        let chunk_size = self.stop_gap + 1;
+
+        // The electrum server has been inconsistent somehow in its responses during sync. For
+        // example, we do a batch request of transactions and the response contains less
+        // tranascations than in the request. This should never happen but we don't want to panic.
+        let electrum_goof = || Error::Generic("electrum server misbehaving".to_string());
+
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let scripts = script_req.request().take(chunk_size);
+                    let txids_per_script: Vec<Vec<_>> = self
+                        .client
+                        .batch_script_get_history(scripts)
+                        .map_err(Error::Electrum)?
+                        .into_iter()
+                        .map(|txs| {
+                            txs.into_iter()
+                                .map(|tx| {
+                                    let tx_height = match tx.height {
+                                        none if none <= 0 => None,
+                                        height => {
+                                            txid_to_height.insert(tx.tx_hash, height as u32);
+                                            Some(height as u32)
+                                        }
+                                    };
+                                    (tx.tx_hash, tx_height)
+                                })
+                                .collect()
+                        })
+                        .collect();
+
+                    script_req.satisfy(txids_per_script)?
+                }
+
+                Request::Conftime(conftime_req) => {
+                    // collect up to chunk_size heights to fetch from electrum
+                    let needs_block_height = conftime_req
+                        .request()
+                        .filter_map(|txid| txid_to_height.get(txid).cloned())
+                        .filter(|height| block_times.get(height).is_none())
+                        .take(chunk_size)
+                        .collect::<HashSet<u32>>();
+
+                    let new_block_headers = self
+                        .client
+                        .batch_block_header(needs_block_height.iter().cloned())?;
+
+                    for (height, header) in needs_block_height.into_iter().zip(new_block_headers) {
+                        block_times.insert(height, header.time);
+                    }
+
+                    let conftimes = conftime_req
+                        .request()
+                        .take(chunk_size)
+                        .map(|txid| {
+                            let confirmation_time = txid_to_height
+                                .get(txid)
+                                .map(|height| {
+                                    let timestamp =
+                                        *block_times.get(height).ok_or_else(electrum_goof)?;
+                                    Result::<_, Error>::Ok(BlockTime {
+                                        height: *height,
+                                        timestamp: timestamp.into(),
+                                    })
+                                })
+                                .transpose()?;
+                            Ok(confirmation_time)
+                        })
+                        .collect::<Result<_, Error>>()?;
+
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let needs_full = tx_req.request().take(chunk_size);
+                    tx_cache.save_txs(needs_full.clone())?;
+                    let full_transactions = needs_full
+                        .map(|txid| tx_cache.get(*txid).ok_or_else(electrum_goof))
+                        .collect::<Result<Vec<_>, _>>()?;
+                    let input_txs = full_transactions.iter().flat_map(|tx| {
+                        tx.input
+                            .iter()
+                            .filter(|input| !input.previous_output.is_null())
+                            .map(|input| &input.previous_output.txid)
+                    });
+                    tx_cache.save_txs(input_txs)?;
+
+                    let full_details = full_transactions
+                        .into_iter()
+                        .map(|tx| {
+                            let mut input_index = 0usize;
+                            let prev_outputs = tx
+                                .input
+                                .iter()
+                                .map(|input| {
+                                    if input.previous_output.is_null() {
+                                        return Ok(None);
+                                    }
+                                    let prev_tx = tx_cache
+                                        .get(input.previous_output.txid)
+                                        .ok_or_else(electrum_goof)?;
+                                    let txout = prev_tx
+                                        .output
+                                        .get(input.previous_output.vout as usize)
+                                        .ok_or_else(electrum_goof)?;
+                                    input_index += 1;
+                                    Ok(Some(txout.clone()))
+                                })
+                                .collect::<Result<Vec<_>, Error>>()?;
+                            Ok((prev_outputs, tx))
+                        })
+                        .collect::<Result<Vec<_>, Error>>()?;
+
+                    tx_req.satisfy(full_details)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+        Ok(())
+    }
+}
+
+struct TxCache<'a, 'b, D> {
+    db: &'a D,
+    client: &'b Client,
+    cache: HashMap<Txid, Transaction>,
+}
+
+impl<'a, 'b, D: Database> TxCache<'a, 'b, D> {
+    fn new(db: &'a D, client: &'b Client) -> Self {
+        TxCache {
+            db,
+            client,
+            cache: HashMap::default(),
+        }
+    }
+    fn save_txs<'c>(&mut self, txids: impl Iterator<Item = &'c Txid>) -> Result<(), Error> {
+        let mut need_fetch = vec![];
+        for txid in txids {
+            if self.cache.get(txid).is_some() {
+                continue;
+            } else if let Some(transaction) = self.db.get_raw_tx(txid)? {
+                self.cache.insert(*txid, transaction);
+            } else {
+                need_fetch.push(txid);
+            }
+        }
+
+        if !need_fetch.is_empty() {
+            let txs = self
+                .client
+                .batch_transaction_get(need_fetch.clone())
+                .map_err(Error::Electrum)?;
+            for (tx, _txid) in txs.into_iter().zip(need_fetch) {
+                debug_assert_eq!(*_txid, tx.txid());
+                self.cache.insert(tx.txid(), tx);
+            }
+        }
+
+        Ok(())
     }
 
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid> + Clone>(
-        &self,
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error> {
-        self.batch_transaction_get(txids).map_err(Error::Electrum)
-    }
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32> + Clone>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error> {
-        self.batch_block_header(heights).map_err(Error::Electrum)
+    fn get(&self, txid: Txid) -> Option<Transaction> {
+        self.cache.get(&txid).map(Clone::clone)
     }
 }
 
 /// Configuration for an [`ElectrumBlockchain`]
-#[derive(Debug, serde::Deserialize, serde::Serialize)]
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
 pub struct ElectrumBlockchainConfig {
     /// URL of the Electrum server (such as ElectrumX, Esplora, BWT) may start with `ssl://` or `tcp://` and include a port
     ///
@@ -169,6 +299,8 @@ pub struct ElectrumBlockchainConfig {
     pub retry: u8,
     /// Request timeout (seconds)
     pub timeout: Option<u8>,
+    /// Stop searching addresses for transactions after finding an unused gap of this length
+    pub stop_gap: usize,
 }
 
 impl ConfigurableBlockchain for ElectrumBlockchain {
@@ -182,9 +314,102 @@ impl ConfigurableBlockchain for ElectrumBlockchain {
             .socks5(socks5)?
             .build();
 
-        Ok(ElectrumBlockchain(Client::from_config(
-            config.url.as_str(),
-            electrum_config,
-        )?))
+        Ok(ElectrumBlockchain {
+            client: Client::from_config(config.url.as_str(), electrum_config)?,
+            stop_gap: config.stop_gap,
+        })
+    }
+}
+
+#[cfg(test)]
+#[cfg(feature = "test-electrum")]
+mod test {
+    use std::sync::Arc;
+
+    use super::*;
+    use crate::database::MemoryDatabase;
+    use crate::testutils::blockchain_tests::TestClient;
+    use crate::testutils::configurable_blockchain_tests::ConfigurableBlockchainTester;
+    use crate::wallet::{AddressIndex, Wallet};
+
+    crate::bdk_blockchain_tests! {
+        fn test_instance(test_client: &TestClient) -> ElectrumBlockchain {
+            ElectrumBlockchain::from(Client::new(&test_client.electrsd.electrum_url).unwrap())
+        }
+    }
+
+    fn get_factory() -> (TestClient, Arc<ElectrumBlockchain>) {
+        let test_client = TestClient::default();
+
+        let factory = Arc::new(ElectrumBlockchain::from(
+            Client::new(&test_client.electrsd.electrum_url).unwrap(),
+        ));
+
+        (test_client, factory)
+    }
+
+    #[test]
+    fn test_electrum_blockchain_factory() {
+        let (_test_client, factory) = get_factory();
+
+        let a = factory.build("aaaaaa", None).unwrap();
+        let b = factory.build("bbbbbb", None).unwrap();
+
+        assert_eq!(
+            a.client.block_headers_subscribe().unwrap().height,
+            b.client.block_headers_subscribe().unwrap().height
+        );
+    }
+
+    #[test]
+    fn test_electrum_blockchain_factory_sync_wallet() {
+        let (mut test_client, factory) = get_factory();
+
+        let db = MemoryDatabase::new();
+        let wallet = Wallet::new(
+            "wpkh(L5EZftvrYaSudiozVRzTqLcHLNDoVn7H5HSfM9BAN6tMJX8oTWz6)",
+            None,
+            bitcoin::Network::Regtest,
+            db,
+        )
+        .unwrap();
+
+        let address = wallet.get_address(AddressIndex::New).unwrap();
+
+        let tx = testutils! {
+            @tx ( (@addr address.address) => 50_000 )
+        };
+        test_client.receive(tx);
+
+        factory
+            .sync_wallet(&wallet, None, Default::default())
+            .unwrap();
+
+        assert_eq!(wallet.get_balance().unwrap(), 50_000);
+    }
+
+    #[test]
+    fn test_electrum_with_variable_configs() {
+        struct ElectrumTester;
+
+        impl ConfigurableBlockchainTester<ElectrumBlockchain> for ElectrumTester {
+            const BLOCKCHAIN_NAME: &'static str = "Electrum";
+
+            fn config_with_stop_gap(
+                &self,
+                test_client: &mut TestClient,
+                stop_gap: usize,
+            ) -> Option<ElectrumBlockchainConfig> {
+                Some(ElectrumBlockchainConfig {
+                    url: test_client.electrsd.electrum_url.clone(),
+                    socks5: None,
+                    retry: 0,
+                    timeout: None,
+                    stop_gap: stop_gap,
+                })
+            }
+        }
+
+        ElectrumTester.run();
     }
 }

src/blockchain/esplora.rs

@@ -1,429 +0,0 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
-//
-// Copyright (c) 2020 Magical Bitcoin
-//
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
-
-//! Esplora
-//!
-//! This module defines a [`Blockchain`] struct that can query an Esplora backend
-//! populate the wallet's [database](crate::database::Database) by
-//!
-//! ## Example
-//!
-//! ```no_run
-//! # use bdk::blockchain::esplora::EsploraBlockchain;
-//! let blockchain = EsploraBlockchain::new("https://blockstream.info/testnet/api", None);
-//! # Ok::<(), bdk::Error>(())
-//! ```
-
-use std::collections::{HashMap, HashSet};
-use std::fmt;
-
-use futures::stream::{self, FuturesOrdered, StreamExt, TryStreamExt};
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
-
-use serde::Deserialize;
-
-use reqwest::{Client, StatusCode};
-
-use bitcoin::consensus::{self, deserialize, serialize};
-use bitcoin::hashes::hex::{FromHex, ToHex};
-use bitcoin::hashes::{sha256, Hash};
-use bitcoin::{BlockHash, BlockHeader, Script, Transaction, Txid};
-
-use self::utils::{ELSGetHistoryRes, ElectrumLikeSync};
-use super::*;
-use crate::database::BatchDatabase;
-use crate::error::Error;
-use crate::wallet::utils::ChunksIterator;
-use crate::FeeRate;
-
-const DEFAULT_CONCURRENT_REQUESTS: u8 = 4;
-
-#[derive(Debug)]
-struct UrlClient {
-    url: String,
-    // We use the async client instead of the blocking one because it automatically uses `fetch`
-    // when the target platform is wasm32.
-    client: Client,
-    concurrency: u8,
-}
-
-/// Structure that implements the logic to sync with Esplora
-///
-/// ## Example
-/// See the [`blockchain::esplora`](crate::blockchain::esplora) module for a usage example.
-#[derive(Debug)]
-pub struct EsploraBlockchain(UrlClient);
-
-impl std::convert::From<UrlClient> for EsploraBlockchain {
-    fn from(url_client: UrlClient) -> Self {
-        EsploraBlockchain(url_client)
-    }
-}
-
-impl EsploraBlockchain {
-    /// Create a new instance of the client from a base URL
-    pub fn new(base_url: &str, concurrency: Option<u8>) -> Self {
-        EsploraBlockchain(UrlClient {
-            url: base_url.to_string(),
-            client: Client::new(),
-            concurrency: concurrency.unwrap_or(DEFAULT_CONCURRENT_REQUESTS),
-        })
-    }
-}
-
-#[maybe_async]
-impl Blockchain for EsploraBlockchain {
-    fn get_capabilities(&self) -> HashSet<Capability> {
-        vec![
-            Capability::FullHistory,
-            Capability::GetAnyTx,
-            Capability::AccurateFees,
-        ]
-        .into_iter()
-        .collect()
-    }
-
-    fn setup<D: BatchDatabase, P: Progress>(
-        &self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(self
-            .0
-            .electrum_like_setup(stop_gap, database, progress_update))
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(await_or_block!(self.0._get_tx(txid))?)
-    }
-
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        Ok(await_or_block!(self.0._broadcast(tx))?)
-    }
-
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(await_or_block!(self.0._get_height())?)
-    }
-
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
-        let estimates = await_or_block!(self.0._get_fee_estimates())?;
-
-        let fee_val = estimates
-            .into_iter()
-            .map(|(k, v)| Ok::<_, std::num::ParseIntError>((k.parse::<usize>()?, v)))
-            .collect::<Result<Vec<_>, _>>()
-            .map_err(|e| Error::Generic(e.to_string()))?
-            .into_iter()
-            .take_while(|(k, _)| k <= &target)
-            .map(|(_, v)| v)
-            .last()
-            .unwrap_or(1.0);
-
-        Ok(FeeRate::from_sat_per_vb(fee_val as f32))
-    }
-}
-
-impl UrlClient {
-    fn script_to_scripthash(script: &Script) -> String {
-        sha256::Hash::hash(script.as_bytes()).into_inner().to_hex()
-    }
-
-    async fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
-        let resp = self
-            .client
-            .get(&format!("{}/tx/{}/raw", self.url, txid))
-            .send()
-            .await?;
-
-        if let StatusCode::NOT_FOUND = resp.status() {
-            return Ok(None);
-        }
-
-        Ok(Some(deserialize(&resp.error_for_status()?.bytes().await?)?))
-    }
-
-    async fn _get_tx_no_opt(&self, txid: &Txid) -> Result<Transaction, EsploraError> {
-        match self._get_tx(txid).await {
-            Ok(Some(tx)) => Ok(tx),
-            Ok(None) => Err(EsploraError::TransactionNotFound(*txid)),
-            Err(e) => Err(e),
-        }
-    }
-
-    async fn _get_header(&self, block_height: u32) -> Result<BlockHeader, EsploraError> {
-        let resp = self
-            .client
-            .get(&format!("{}/block-height/{}", self.url, block_height))
-            .send()
-            .await?;
-
-        if let StatusCode::NOT_FOUND = resp.status() {
-            return Err(EsploraError::HeaderHeightNotFound(block_height));
-        }
-        let bytes = resp.bytes().await?;
-        let hash = std::str::from_utf8(&bytes)
-            .map_err(|_| EsploraError::HeaderHeightNotFound(block_height))?;
-
-        let resp = self
-            .client
-            .get(&format!("{}/block/{}/header", self.url, hash))
-            .send()
-            .await?;
-
-        let header = deserialize(&Vec::from_hex(&resp.text().await?)?)?;
-
-        Ok(header)
-    }
-
-    async fn _broadcast(&self, transaction: &Transaction) -> Result<(), EsploraError> {
-        self.client
-            .post(&format!("{}/tx", self.url))
-            .body(serialize(transaction).to_hex())
-            .send()
-            .await?
-            .error_for_status()?;
-
-        Ok(())
-    }
-
-    async fn _get_height(&self) -> Result<u32, EsploraError> {
-        let req = self
-            .client
-            .get(&format!("{}/blocks/tip/height", self.url))
-            .send()
-            .await?;
-
-        Ok(req.error_for_status()?.text().await?.parse()?)
-    }
-
-    async fn _script_get_history(
-        &self,
-        script: &Script,
-    ) -> Result<Vec<ELSGetHistoryRes>, EsploraError> {
-        let mut result = Vec::new();
-        let scripthash = Self::script_to_scripthash(script);
-
-        // Add the unconfirmed transactions first
-        result.extend(
-            self.client
-                .get(&format!(
-                    "{}/scripthash/{}/txs/mempool",
-                    self.url, scripthash
-                ))
-                .send()
-                .await?
-                .error_for_status()?
-                .json::<Vec<EsploraGetHistory>>()
-                .await?
-                .into_iter()
-                .map(|x| ELSGetHistoryRes {
-                    tx_hash: x.txid,
-                    height: x.status.block_height.unwrap_or(0) as i32,
-                }),
-        );
-
-        debug!(
-            "Found {} mempool txs for {} - {:?}",
-            result.len(),
-            scripthash,
-            script
-        );
-
-        // Then go through all the pages of confirmed transactions
-        let mut last_txid = String::new();
-        loop {
-            let response = self
-                .client
-                .get(&format!(
-                    "{}/scripthash/{}/txs/chain/{}",
-                    self.url, scripthash, last_txid
-                ))
-                .send()
-                .await?
-                .error_for_status()?
-                .json::<Vec<EsploraGetHistory>>()
-                .await?;
-            let len = response.len();
-            if let Some(elem) = response.last() {
-                last_txid = elem.txid.to_hex();
-            }
-
-            debug!("... adding {} confirmed transactions", len);
-
-            result.extend(response.into_iter().map(|x| ELSGetHistoryRes {
-                tx_hash: x.txid,
-                height: x.status.block_height.unwrap_or(0) as i32,
-            }));
-
-            if len < 25 {
-                break;
-            }
-        }
-
-        Ok(result)
-    }
-
-    async fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
-        Ok(self
-            .client
-            .get(&format!("{}/fee-estimates", self.url,))
-            .send()
-            .await?
-            .error_for_status()?
-            .json::<HashMap<String, f64>>()
-            .await?)
-    }
-}
-
-#[maybe_async]
-impl ElectrumLikeSync for UrlClient {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script>>(
-        &self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ELSGetHistoryRes>>, Error> {
-        let future = async {
-            let mut results = vec![];
-            for chunk in ChunksIterator::new(scripts.into_iter(), self.concurrency as usize) {
-                let mut futs = FuturesOrdered::new();
-                for script in chunk {
-                    futs.push(self._script_get_history(&script));
-                }
-                let partial_results: Vec<Vec<ELSGetHistoryRes>> = futs.try_collect().await?;
-                results.extend(partial_results);
-            }
-            Ok(stream::iter(results).collect().await)
-        };
-
-        await_or_block!(future)
-    }
-
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid>>(
-        &self,
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error> {
-        let future = async {
-            let mut results = vec![];
-            for chunk in ChunksIterator::new(txids.into_iter(), self.concurrency as usize) {
-                let mut futs = FuturesOrdered::new();
-                for txid in chunk {
-                    futs.push(self._get_tx_no_opt(&txid));
-                }
-                let partial_results: Vec<Transaction> = futs.try_collect().await?;
-                results.extend(partial_results);
-            }
-            Ok(stream::iter(results).collect().await)
-        };
-
-        await_or_block!(future)
-    }
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32>>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error> {
-        let future = async {
-            let mut results = vec![];
-            for chunk in ChunksIterator::new(heights.into_iter(), self.concurrency as usize) {
-                let mut futs = FuturesOrdered::new();
-                for height in chunk {
-                    futs.push(self._get_header(height));
-                }
-                let partial_results: Vec<BlockHeader> = futs.try_collect().await?;
-                results.extend(partial_results);
-            }
-            Ok(stream::iter(results).collect().await)
-        };
-
-        await_or_block!(future)
-    }
-}
-
-#[derive(Deserialize)]
-struct EsploraGetHistoryStatus {
-    block_height: Option<usize>,
-}
-
-#[derive(Deserialize)]
-struct EsploraGetHistory {
-    txid: Txid,
-    status: EsploraGetHistoryStatus,
-}
-
-/// Configuration for an [`EsploraBlockchain`]
-#[derive(Debug, serde::Deserialize, serde::Serialize)]
-pub struct EsploraBlockchainConfig {
-    /// Base URL of the esplora service
-    ///
-    /// eg. `https://blockstream.info/api/`
-    pub base_url: String,
-    /// Number of parallel requests sent to the esplora service (default: 4)
-    pub concurrency: Option<u8>,
-}
-
-impl ConfigurableBlockchain for EsploraBlockchain {
-    type Config = EsploraBlockchainConfig;
-
-    fn from_config(config: &Self::Config) -> Result<Self, Error> {
-        Ok(EsploraBlockchain::new(
-            config.base_url.as_str(),
-            config.concurrency,
-        ))
-    }
-}
-
-/// Errors that can happen during a sync with [`EsploraBlockchain`]
-#[derive(Debug)]
-pub enum EsploraError {
-    /// Error with the HTTP call
-    Reqwest(reqwest::Error),
-    /// Invalid number returned
-    Parsing(std::num::ParseIntError),
-    /// Invalid Bitcoin data returned
-    BitcoinEncoding(bitcoin::consensus::encode::Error),
-    /// Invalid Hex data returned
-    Hex(bitcoin::hashes::hex::Error),
-
-    /// Transaction not found
-    TransactionNotFound(Txid),
-    /// Header height not found
-    HeaderHeightNotFound(u32),
-    /// Header hash not found
-    HeaderHashNotFound(BlockHash),
-}
-
-impl fmt::Display for EsploraError {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        write!(f, "{:?}", self)
-    }
-}
-
-impl std::error::Error for EsploraError {}
-
-impl_error!(reqwest::Error, Reqwest, EsploraError);
-impl_error!(std::num::ParseIntError, Parsing, EsploraError);
-impl_error!(consensus::encode::Error, BitcoinEncoding, EsploraError);
-impl_error!(bitcoin::hashes::hex::Error, Hex, EsploraError);

src/blockchain/esplora/api.rs

@@ -0,0 +1,117 @@
+//! structs from the esplora API
+//!
+//! see: <https://github.com/Blockstream/esplora/blob/master/API.md>
+use crate::BlockTime;
+use bitcoin::{OutPoint, Script, Transaction, TxIn, TxOut, Txid, Witness};
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct PrevOut {
+    pub value: u64,
+    pub scriptpubkey: Script,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct Vin {
+    pub txid: Txid,
+    pub vout: u32,
+    // None if coinbase
+    pub prevout: Option<PrevOut>,
+    pub scriptsig: Script,
+    #[serde(deserialize_with = "deserialize_witness", default)]
+    pub witness: Vec<Vec<u8>>,
+    pub sequence: u32,
+    pub is_coinbase: bool,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct Vout {
+    pub value: u64,
+    pub scriptpubkey: Script,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct TxStatus {
+    pub confirmed: bool,
+    pub block_height: Option<u32>,
+    pub block_time: Option<u64>,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct Tx {
+    pub txid: Txid,
+    pub version: i32,
+    pub locktime: u32,
+    pub vin: Vec<Vin>,
+    pub vout: Vec<Vout>,
+    pub status: TxStatus,
+    pub fee: u64,
+}
+
+impl Tx {
+    pub fn to_tx(&self) -> Transaction {
+        Transaction {
+            version: self.version,
+            lock_time: self.locktime,
+            input: self
+                .vin
+                .iter()
+                .cloned()
+                .map(|vin| TxIn {
+                    previous_output: OutPoint {
+                        txid: vin.txid,
+                        vout: vin.vout,
+                    },
+                    script_sig: vin.scriptsig,
+                    sequence: vin.sequence,
+                    witness: Witness::from_vec(vin.witness),
+                })
+                .collect(),
+            output: self
+                .vout
+                .iter()
+                .cloned()
+                .map(|vout| TxOut {
+                    value: vout.value,
+                    script_pubkey: vout.scriptpubkey,
+                })
+                .collect(),
+        }
+    }
+
+    pub fn confirmation_time(&self) -> Option<BlockTime> {
+        match self.status {
+            TxStatus {
+                confirmed: true,
+                block_height: Some(height),
+                block_time: Some(timestamp),
+            } => Some(BlockTime { timestamp, height }),
+            _ => None,
+        }
+    }
+
+    pub fn previous_outputs(&self) -> Vec<Option<TxOut>> {
+        self.vin
+            .iter()
+            .cloned()
+            .map(|vin| {
+                vin.prevout.map(|po| TxOut {
+                    script_pubkey: po.scriptpubkey,
+                    value: po.value,
+                })
+            })
+            .collect()
+    }
+}
+
+fn deserialize_witness<'de, D>(d: D) -> Result<Vec<Vec<u8>>, D::Error>
+where
+    D: serde::de::Deserializer<'de>,
+{
+    use crate::serde::Deserialize;
+    use bitcoin::hashes::hex::FromHex;
+    let list = Vec::<String>::deserialize(d)?;
+    list.into_iter()
+        .map(|hex_str| Vec::<u8>::from_hex(&hex_str))
+        .collect::<Result<Vec<Vec<u8>>, _>>()
+        .map_err(serde::de::Error::custom)
+}

src/blockchain/esplora/mod.rs

@@ -0,0 +1,246 @@
+//! Esplora
+//!
+//! This module defines a [`EsploraBlockchain`] struct that can query an Esplora
+//! backend populate the wallet's [database](crate::database::Database) by:
+//!
+//! ## Example
+//!
+//! ```no_run
+//! # use bdk::blockchain::esplora::EsploraBlockchain;
+//! let blockchain = EsploraBlockchain::new("https://blockstream.info/testnet/api", 20);
+//! # Ok::<(), bdk::Error>(())
+//! ```
+//!
+//! Esplora blockchain can use either `ureq` or `reqwest` for the HTTP client
+//! depending on your needs (blocking or async respectively).
+//!
+//! Please note, to configure the Esplora HTTP client correctly use one of:
+//! Blocking:  --features='esplora,ureq'
+//! Async:     --features='async-interface,esplora,reqwest' --no-default-features
+use std::collections::HashMap;
+use std::fmt;
+use std::io;
+
+use bitcoin::consensus;
+use bitcoin::{BlockHash, Txid};
+
+use crate::error::Error;
+use crate::FeeRate;
+
+#[cfg(feature = "reqwest")]
+mod reqwest;
+
+#[cfg(feature = "reqwest")]
+pub use self::reqwest::*;
+
+#[cfg(feature = "ureq")]
+mod ureq;
+
+#[cfg(feature = "ureq")]
+pub use self::ureq::*;
+
+mod api;
+
+fn into_fee_rate(target: usize, estimates: HashMap<String, f64>) -> Result<FeeRate, Error> {
+    let fee_val = {
+        let mut pairs = estimates
+            .into_iter()
+            .filter_map(|(k, v)| Some((k.parse::<usize>().ok()?, v)))
+            .collect::<Vec<_>>();
+        pairs.sort_unstable_by_key(|(k, _)| std::cmp::Reverse(*k));
+        pairs
+            .into_iter()
+            .find(|(k, _)| k <= &target)
+            .map(|(_, v)| v)
+            .unwrap_or(1.0)
+    };
+    Ok(FeeRate::from_sat_per_vb(fee_val as f32))
+}
+
+/// Errors that can happen during a sync with [`EsploraBlockchain`]
+#[derive(Debug)]
+pub enum EsploraError {
+    /// Error during ureq HTTP request
+    #[cfg(feature = "ureq")]
+    Ureq(::ureq::Error),
+    /// Transport error during the ureq HTTP call
+    #[cfg(feature = "ureq")]
+    UreqTransport(::ureq::Transport),
+    /// Error during reqwest HTTP request
+    #[cfg(feature = "reqwest")]
+    Reqwest(::reqwest::Error),
+    /// HTTP response error
+    HttpResponse(u16),
+    /// IO error during ureq response read
+    Io(io::Error),
+    /// No header found in ureq response
+    NoHeader,
+    /// Invalid number returned
+    Parsing(std::num::ParseIntError),
+    /// Invalid Bitcoin data returned
+    BitcoinEncoding(bitcoin::consensus::encode::Error),
+    /// Invalid Hex data returned
+    Hex(bitcoin::hashes::hex::Error),
+
+    /// Transaction not found
+    TransactionNotFound(Txid),
+    /// Header height not found
+    HeaderHeightNotFound(u32),
+    /// Header hash not found
+    HeaderHashNotFound(BlockHash),
+}
+
+impl fmt::Display for EsploraError {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(f, "{:?}", self)
+    }
+}
+
+/// Configuration for an [`EsploraBlockchain`]
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
+pub struct EsploraBlockchainConfig {
+    /// Base URL of the esplora service
+    ///
+    /// eg. `https://blockstream.info/api/`
+    pub base_url: String,
+    /// Optional URL of the proxy to use to make requests to the Esplora server
+    ///
+    /// The string should be formatted as: `<protocol>://<user>:<password>@host:<port>`.
+    ///
+    /// Note that the format of this value and the supported protocols change slightly between the
+    /// sync version of esplora (using `ureq`) and the async version (using `reqwest`). For more
+    /// details check with the documentation of the two crates. Both of them are compiled with
+    /// the `socks` feature enabled.
+    ///
+    /// The proxy is ignored when targeting `wasm32`.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub proxy: Option<String>,
+    /// Number of parallel requests sent to the esplora service (default: 4)
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub concurrency: Option<u8>,
+    /// Stop searching addresses for transactions after finding an unused gap of this length.
+    pub stop_gap: usize,
+    /// Socket timeout.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub timeout: Option<u64>,
+}
+
+impl EsploraBlockchainConfig {
+    /// create a config with default values given the base url and stop gap
+    pub fn new(base_url: String, stop_gap: usize) -> Self {
+        Self {
+            base_url,
+            proxy: None,
+            timeout: None,
+            stop_gap,
+            concurrency: None,
+        }
+    }
+}
+
+impl std::error::Error for EsploraError {}
+
+#[cfg(feature = "ureq")]
+impl_error!(::ureq::Transport, UreqTransport, EsploraError);
+#[cfg(feature = "reqwest")]
+impl_error!(::reqwest::Error, Reqwest, EsploraError);
+impl_error!(io::Error, Io, EsploraError);
+impl_error!(std::num::ParseIntError, Parsing, EsploraError);
+impl_error!(consensus::encode::Error, BitcoinEncoding, EsploraError);
+impl_error!(bitcoin::hashes::hex::Error, Hex, EsploraError);
+
+#[cfg(test)]
+#[cfg(feature = "test-esplora")]
+crate::bdk_blockchain_tests! {
+    fn test_instance(test_client: &TestClient) -> EsploraBlockchain {
+        EsploraBlockchain::new(&format!("http://{}",test_client.electrsd.esplora_url.as_ref().unwrap()), 20)
+    }
+}
+
+const DEFAULT_CONCURRENT_REQUESTS: u8 = 4;
+
+#[cfg(test)]
+mod test {
+    use super::*;
+
+    #[test]
+    fn feerate_parsing() {
+        let esplora_fees = serde_json::from_str::<HashMap<String, f64>>(
+            r#"{
+  "25": 1.015,
+  "5": 2.3280000000000003,
+  "12": 2.0109999999999997,
+  "15": 1.018,
+  "17": 1.018,
+  "11": 2.0109999999999997,
+  "3": 3.01,
+  "2": 4.9830000000000005,
+  "6": 2.2359999999999998,
+  "21": 1.018,
+  "13": 1.081,
+  "7": 2.2359999999999998,
+  "8": 2.2359999999999998,
+  "16": 1.018,
+  "20": 1.018,
+  "22": 1.017,
+  "23": 1.017,
+  "504": 1,
+  "9": 2.2359999999999998,
+  "14": 1.018,
+  "10": 2.0109999999999997,
+  "24": 1.017,
+  "1008": 1,
+  "1": 4.9830000000000005,
+  "4": 2.3280000000000003,
+  "19": 1.018,
+  "144": 1,
+  "18": 1.018
+}
+"#,
+        )
+        .unwrap();
+        assert_eq!(
+            into_fee_rate(6, esplora_fees.clone()).unwrap(),
+            FeeRate::from_sat_per_vb(2.236)
+        );
+        assert_eq!(
+            into_fee_rate(26, esplora_fees).unwrap(),
+            FeeRate::from_sat_per_vb(1.015),
+            "should inherit from value for 25"
+        );
+    }
+
+    #[test]
+    #[cfg(feature = "test-esplora")]
+    fn test_esplora_with_variable_configs() {
+        use crate::testutils::{
+            blockchain_tests::TestClient,
+            configurable_blockchain_tests::ConfigurableBlockchainTester,
+        };
+
+        struct EsploraTester;
+
+        impl ConfigurableBlockchainTester<EsploraBlockchain> for EsploraTester {
+            const BLOCKCHAIN_NAME: &'static str = "Esplora";
+
+            fn config_with_stop_gap(
+                &self,
+                test_client: &mut TestClient,
+                stop_gap: usize,
+            ) -> Option<EsploraBlockchainConfig> {
+                Some(EsploraBlockchainConfig {
+                    base_url: format!(
+                        "http://{}",
+                        test_client.electrsd.esplora_url.as_ref().unwrap()
+                    ),
+                    proxy: None,
+                    concurrency: None,
+                    stop_gap: stop_gap,
+                    timeout: None,
+                })
+            }
+        }
+
+        EsploraTester.run();
+    }
+}

src/blockchain/esplora/reqwest.rs

@@ -0,0 +1,350 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Esplora by way of `reqwest` HTTP client.
+
+use std::collections::{HashMap, HashSet};
+
+use bitcoin::consensus::{deserialize, serialize};
+use bitcoin::hashes::hex::{FromHex, ToHex};
+use bitcoin::hashes::{sha256, Hash};
+use bitcoin::{BlockHeader, Script, Transaction, Txid};
+
+#[allow(unused_imports)]
+use log::{debug, error, info, trace};
+
+use ::reqwest::{Client, StatusCode};
+use futures::stream::{FuturesOrdered, TryStreamExt};
+
+use super::api::Tx;
+use crate::blockchain::esplora::EsploraError;
+use crate::blockchain::*;
+use crate::database::BatchDatabase;
+use crate::error::Error;
+use crate::FeeRate;
+
+#[derive(Debug)]
+struct UrlClient {
+    url: String,
+    // We use the async client instead of the blocking one because it automatically uses `fetch`
+    // when the target platform is wasm32.
+    client: Client,
+    concurrency: u8,
+}
+
+/// Structure that implements the logic to sync with Esplora
+///
+/// ## Example
+/// See the [`blockchain::esplora`](crate::blockchain::esplora) module for a usage example.
+#[derive(Debug)]
+pub struct EsploraBlockchain {
+    url_client: UrlClient,
+    stop_gap: usize,
+}
+
+impl std::convert::From<UrlClient> for EsploraBlockchain {
+    fn from(url_client: UrlClient) -> Self {
+        EsploraBlockchain {
+            url_client,
+            stop_gap: 20,
+        }
+    }
+}
+
+impl EsploraBlockchain {
+    /// Create a new instance of the client from a base URL and `stop_gap`.
+    pub fn new(base_url: &str, stop_gap: usize) -> Self {
+        EsploraBlockchain {
+            url_client: UrlClient {
+                url: base_url.to_string(),
+                client: Client::new(),
+                concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
+            },
+            stop_gap,
+        }
+    }
+
+    /// Set the concurrency to use when doing batch queries against the Esplora instance.
+    pub fn with_concurrency(mut self, concurrency: u8) -> Self {
+        self.url_client.concurrency = concurrency;
+        self
+    }
+}
+
+#[maybe_async]
+impl Blockchain for EsploraBlockchain {
+    fn get_capabilities(&self) -> HashSet<Capability> {
+        vec![
+            Capability::FullHistory,
+            Capability::GetAnyTx,
+            Capability::AccurateFees,
+        ]
+        .into_iter()
+        .collect()
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        Ok(await_or_block!(self.url_client._broadcast(tx))?)
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        let estimates = await_or_block!(self.url_client._get_fee_estimates())?;
+        super::into_fee_rate(target, estimates)
+    }
+}
+
+impl StatelessBlockchain for EsploraBlockchain {}
+
+#[maybe_async]
+impl GetHeight for EsploraBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(await_or_block!(self.url_client._get_height())?)
+    }
+}
+
+#[maybe_async]
+impl GetTx for EsploraBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(await_or_block!(self.url_client._get_tx(txid))?)
+    }
+}
+
+#[maybe_async]
+impl GetBlockHash for EsploraBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        let block_header = await_or_block!(self.url_client._get_header(height as u32))?;
+        Ok(block_header.block_hash())
+    }
+}
+
+#[maybe_async]
+impl WalletSync for EsploraBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        use crate::blockchain::script_sync::Request;
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut tx_index: HashMap<Txid, Tx> = HashMap::new();
+
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let futures: FuturesOrdered<_> = script_req
+                        .request()
+                        .take(self.url_client.concurrency as usize)
+                        .map(|script| async move {
+                            let mut related_txs: Vec<Tx> =
+                                self.url_client._scripthash_txs(script, None).await?;
+
+                            let n_confirmed =
+                                related_txs.iter().filter(|tx| tx.status.confirmed).count();
+                            // esplora pages on 25 confirmed transactions. If there's 25 or more we
+                            // keep requesting to see if there's more.
+                            if n_confirmed >= 25 {
+                                loop {
+                                    let new_related_txs: Vec<Tx> = self
+                                        .url_client
+                                        ._scripthash_txs(
+                                            script,
+                                            Some(related_txs.last().unwrap().txid),
+                                        )
+                                        .await?;
+                                    let n = new_related_txs.len();
+                                    related_txs.extend(new_related_txs);
+                                    // we've reached the end
+                                    if n < 25 {
+                                        break;
+                                    }
+                                }
+                            }
+                            Result::<_, Error>::Ok(related_txs)
+                        })
+                        .collect();
+                    let txs_per_script: Vec<Vec<Tx>> = await_or_block!(futures.try_collect())?;
+                    let mut satisfaction = vec![];
+
+                    for txs in txs_per_script {
+                        satisfaction.push(
+                            txs.iter()
+                                .map(|tx| (tx.txid, tx.status.block_height))
+                                .collect(),
+                        );
+                        for tx in txs {
+                            tx_index.insert(tx.txid, tx);
+                        }
+                    }
+
+                    script_req.satisfy(satisfaction)?
+                }
+                Request::Conftime(conftime_req) => {
+                    let conftimes = conftime_req
+                        .request()
+                        .map(|txid| {
+                            tx_index
+                                .get(txid)
+                                .expect("must be in index")
+                                .confirmation_time()
+                        })
+                        .collect();
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let full_txs = tx_req
+                        .request()
+                        .map(|txid| {
+                            let tx = tx_index.get(txid).expect("must be in index");
+                            Ok((tx.previous_outputs(), tx.to_tx()))
+                        })
+                        .collect::<Result<_, Error>>()?;
+                    tx_req.satisfy(full_txs)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+
+        Ok(())
+    }
+}
+
+impl UrlClient {
+    async fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
+        let resp = self
+            .client
+            .get(&format!("{}/tx/{}/raw", self.url, txid))
+            .send()
+            .await?;
+
+        if let StatusCode::NOT_FOUND = resp.status() {
+            return Ok(None);
+        }
+
+        Ok(Some(deserialize(&resp.error_for_status()?.bytes().await?)?))
+    }
+
+    async fn _get_tx_no_opt(&self, txid: &Txid) -> Result<Transaction, EsploraError> {
+        match self._get_tx(txid).await {
+            Ok(Some(tx)) => Ok(tx),
+            Ok(None) => Err(EsploraError::TransactionNotFound(*txid)),
+            Err(e) => Err(e),
+        }
+    }
+
+    async fn _get_header(&self, block_height: u32) -> Result<BlockHeader, EsploraError> {
+        let resp = self
+            .client
+            .get(&format!("{}/block-height/{}", self.url, block_height))
+            .send()
+            .await?;
+
+        if let StatusCode::NOT_FOUND = resp.status() {
+            return Err(EsploraError::HeaderHeightNotFound(block_height));
+        }
+        let bytes = resp.bytes().await?;
+        let hash = std::str::from_utf8(&bytes)
+            .map_err(|_| EsploraError::HeaderHeightNotFound(block_height))?;
+
+        let resp = self
+            .client
+            .get(&format!("{}/block/{}/header", self.url, hash))
+            .send()
+            .await?;
+
+        let header = deserialize(&Vec::from_hex(&resp.text().await?)?)?;
+
+        Ok(header)
+    }
+
+    async fn _broadcast(&self, transaction: &Transaction) -> Result<(), EsploraError> {
+        self.client
+            .post(&format!("{}/tx", self.url))
+            .body(serialize(transaction).to_hex())
+            .send()
+            .await?
+            .error_for_status()?;
+
+        Ok(())
+    }
+
+    async fn _get_height(&self) -> Result<u32, EsploraError> {
+        let req = self
+            .client
+            .get(&format!("{}/blocks/tip/height", self.url))
+            .send()
+            .await?;
+
+        Ok(req.error_for_status()?.text().await?.parse()?)
+    }
+
+    async fn _scripthash_txs(
+        &self,
+        script: &Script,
+        last_seen: Option<Txid>,
+    ) -> Result<Vec<Tx>, EsploraError> {
+        let script_hash = sha256::Hash::hash(script.as_bytes()).into_inner().to_hex();
+        let url = match last_seen {
+            Some(last_seen) => format!(
+                "{}/scripthash/{}/txs/chain/{}",
+                self.url, script_hash, last_seen
+            ),
+            None => format!("{}/scripthash/{}/txs", self.url, script_hash),
+        };
+        Ok(self
+            .client
+            .get(url)
+            .send()
+            .await?
+            .error_for_status()?
+            .json::<Vec<Tx>>()
+            .await?)
+    }
+
+    async fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
+        Ok(self
+            .client
+            .get(&format!("{}/fee-estimates", self.url,))
+            .send()
+            .await?
+            .error_for_status()?
+            .json::<HashMap<String, f64>>()
+            .await?)
+    }
+}
+
+impl ConfigurableBlockchain for EsploraBlockchain {
+    type Config = super::EsploraBlockchainConfig;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        let map_e = |e: reqwest::Error| Error::Esplora(Box::new(e.into()));
+
+        let mut blockchain = EsploraBlockchain::new(config.base_url.as_str(), config.stop_gap);
+        if let Some(concurrency) = config.concurrency {
+            blockchain.url_client.concurrency = concurrency;
+        }
+        let mut builder = Client::builder();
+        #[cfg(not(target_arch = "wasm32"))]
+        if let Some(proxy) = &config.proxy {
+            builder = builder.proxy(reqwest::Proxy::all(proxy).map_err(map_e)?);
+        }
+
+        #[cfg(not(target_arch = "wasm32"))]
+        if let Some(timeout) = config.timeout {
+            builder = builder.timeout(core::time::Duration::from_secs(timeout));
+        }
+
+        blockchain.url_client.client = builder.build().map_err(map_e)?;
+
+        Ok(blockchain)
+    }
+}

src/blockchain/esplora/ureq.rs

@@ -0,0 +1,388 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Esplora by way of `ureq` HTTP client.
+
+use std::collections::{HashMap, HashSet};
+use std::io;
+use std::io::Read;
+use std::time::Duration;
+
+#[allow(unused_imports)]
+use log::{debug, error, info, trace};
+
+use ureq::{Agent, Proxy, Response};
+
+use bitcoin::consensus::{deserialize, serialize};
+use bitcoin::hashes::hex::{FromHex, ToHex};
+use bitcoin::hashes::{sha256, Hash};
+use bitcoin::{BlockHeader, Script, Transaction, Txid};
+
+use super::api::Tx;
+use crate::blockchain::esplora::EsploraError;
+use crate::blockchain::*;
+use crate::database::BatchDatabase;
+use crate::error::Error;
+use crate::FeeRate;
+
+#[derive(Debug, Clone)]
+struct UrlClient {
+    url: String,
+    agent: Agent,
+}
+
+/// Structure that implements the logic to sync with Esplora
+///
+/// ## Example
+/// See the [`blockchain::esplora`](crate::blockchain::esplora) module for a usage example.
+#[derive(Debug)]
+pub struct EsploraBlockchain {
+    url_client: UrlClient,
+    stop_gap: usize,
+    concurrency: u8,
+}
+
+impl EsploraBlockchain {
+    /// Create a new instance of the client from a base URL and the `stop_gap`.
+    pub fn new(base_url: &str, stop_gap: usize) -> Self {
+        EsploraBlockchain {
+            url_client: UrlClient {
+                url: base_url.to_string(),
+                agent: Agent::new(),
+            },
+            concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
+            stop_gap,
+        }
+    }
+
+    /// Set the inner `ureq` agent.
+    pub fn with_agent(mut self, agent: Agent) -> Self {
+        self.url_client.agent = agent;
+        self
+    }
+
+    /// Set the number of parallel requests the client can make.
+    pub fn with_concurrency(mut self, concurrency: u8) -> Self {
+        self.concurrency = concurrency;
+        self
+    }
+}
+
+impl Blockchain for EsploraBlockchain {
+    fn get_capabilities(&self) -> HashSet<Capability> {
+        vec![
+            Capability::FullHistory,
+            Capability::GetAnyTx,
+            Capability::AccurateFees,
+        ]
+        .into_iter()
+        .collect()
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        self.url_client._broadcast(tx)?;
+        Ok(())
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        let estimates = self.url_client._get_fee_estimates()?;
+        super::into_fee_rate(target, estimates)
+    }
+}
+
+impl StatelessBlockchain for EsploraBlockchain {}
+
+impl GetHeight for EsploraBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.url_client._get_height()?)
+    }
+}
+
+impl GetTx for EsploraBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.url_client._get_tx(txid)?)
+    }
+}
+
+impl GetBlockHash for EsploraBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        let block_header = self.url_client._get_header(height as u32)?;
+        Ok(block_header.block_hash())
+    }
+}
+
+impl WalletSync for EsploraBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        use crate::blockchain::script_sync::Request;
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut tx_index: HashMap<Txid, Tx> = HashMap::new();
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let scripts = script_req
+                        .request()
+                        .take(self.concurrency as usize)
+                        .cloned();
+
+                    let mut handles = vec![];
+                    for script in scripts {
+                        let client = self.url_client.clone();
+                        // make each request in its own thread.
+                        handles.push(std::thread::spawn(move || {
+                            let mut related_txs: Vec<Tx> = client._scripthash_txs(&script, None)?;
+
+                            let n_confirmed =
+                                related_txs.iter().filter(|tx| tx.status.confirmed).count();
+                            // esplora pages on 25 confirmed transactions. If there's 25 or more we
+                            // keep requesting to see if there's more.
+                            if n_confirmed >= 25 {
+                                loop {
+                                    let new_related_txs: Vec<Tx> = client._scripthash_txs(
+                                        &script,
+                                        Some(related_txs.last().unwrap().txid),
+                                    )?;
+                                    let n = new_related_txs.len();
+                                    related_txs.extend(new_related_txs);
+                                    // we've reached the end
+                                    if n < 25 {
+                                        break;
+                                    }
+                                }
+                            }
+                            Result::<_, Error>::Ok(related_txs)
+                        }));
+                    }
+
+                    let txs_per_script: Vec<Vec<Tx>> = handles
+                        .into_iter()
+                        .map(|handle| handle.join().unwrap())
+                        .collect::<Result<_, _>>()?;
+                    let mut satisfaction = vec![];
+
+                    for txs in txs_per_script {
+                        satisfaction.push(
+                            txs.iter()
+                                .map(|tx| (tx.txid, tx.status.block_height))
+                                .collect(),
+                        );
+                        for tx in txs {
+                            tx_index.insert(tx.txid, tx);
+                        }
+                    }
+
+                    script_req.satisfy(satisfaction)?
+                }
+                Request::Conftime(conftime_req) => {
+                    let conftimes = conftime_req
+                        .request()
+                        .map(|txid| {
+                            tx_index
+                                .get(txid)
+                                .expect("must be in index")
+                                .confirmation_time()
+                        })
+                        .collect();
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let full_txs = tx_req
+                        .request()
+                        .map(|txid| {
+                            let tx = tx_index.get(txid).expect("must be in index");
+                            Ok((tx.previous_outputs(), tx.to_tx()))
+                        })
+                        .collect::<Result<_, Error>>()?;
+                    tx_req.satisfy(full_txs)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+
+        Ok(())
+    }
+}
+
+impl UrlClient {
+    fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
+        let resp = self
+            .agent
+            .get(&format!("{}/tx/{}/raw", self.url, txid))
+            .call();
+
+        match resp {
+            Ok(resp) => Ok(Some(deserialize(&into_bytes(resp)?)?)),
+            Err(ureq::Error::Status(code, _)) => {
+                if is_status_not_found(code) {
+                    return Ok(None);
+                }
+                Err(EsploraError::HttpResponse(code))
+            }
+            Err(e) => Err(EsploraError::Ureq(e)),
+        }
+    }
+
+    fn _get_tx_no_opt(&self, txid: &Txid) -> Result<Transaction, EsploraError> {
+        match self._get_tx(txid) {
+            Ok(Some(tx)) => Ok(tx),
+            Ok(None) => Err(EsploraError::TransactionNotFound(*txid)),
+            Err(e) => Err(e),
+        }
+    }
+
+    fn _get_header(&self, block_height: u32) -> Result<BlockHeader, EsploraError> {
+        let resp = self
+            .agent
+            .get(&format!("{}/block-height/{}", self.url, block_height))
+            .call();
+
+        let bytes = match resp {
+            Ok(resp) => Ok(into_bytes(resp)?),
+            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
+            Err(e) => Err(EsploraError::Ureq(e)),
+        }?;
+
+        let hash = std::str::from_utf8(&bytes)
+            .map_err(|_| EsploraError::HeaderHeightNotFound(block_height))?;
+
+        let resp = self
+            .agent
+            .get(&format!("{}/block/{}/header", self.url, hash))
+            .call();
+
+        match resp {
+            Ok(resp) => Ok(deserialize(&Vec::from_hex(&resp.into_string()?)?)?),
+            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
+            Err(e) => Err(EsploraError::Ureq(e)),
+        }
+    }
+
+    fn _broadcast(&self, transaction: &Transaction) -> Result<(), EsploraError> {
+        let resp = self
+            .agent
+            .post(&format!("{}/tx", self.url))
+            .send_string(&serialize(transaction).to_hex());
+
+        match resp {
+            Ok(_) => Ok(()), // We do not return the txid?
+            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
+            Err(e) => Err(EsploraError::Ureq(e)),
+        }
+    }
+
+    fn _get_height(&self) -> Result<u32, EsploraError> {
+        let resp = self
+            .agent
+            .get(&format!("{}/blocks/tip/height", self.url))
+            .call();
+
+        match resp {
+            Ok(resp) => Ok(resp.into_string()?.parse()?),
+            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
+            Err(e) => Err(EsploraError::Ureq(e)),
+        }
+    }
+
+    fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
+        let resp = self
+            .agent
+            .get(&format!("{}/fee-estimates", self.url,))
+            .call();
+
+        let map = match resp {
+            Ok(resp) => {
+                let map: HashMap<String, f64> = resp.into_json()?;
+                Ok(map)
+            }
+            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
+            Err(e) => Err(EsploraError::Ureq(e)),
+        }?;
+
+        Ok(map)
+    }
+
+    fn _scripthash_txs(
+        &self,
+        script: &Script,
+        last_seen: Option<Txid>,
+    ) -> Result<Vec<Tx>, EsploraError> {
+        let script_hash = sha256::Hash::hash(script.as_bytes()).into_inner().to_hex();
+        let url = match last_seen {
+            Some(last_seen) => format!(
+                "{}/scripthash/{}/txs/chain/{}",
+                self.url, script_hash, last_seen
+            ),
+            None => format!("{}/scripthash/{}/txs", self.url, script_hash),
+        };
+        Ok(self.agent.get(&url).call()?.into_json()?)
+    }
+}
+
+fn is_status_not_found(status: u16) -> bool {
+    status == 404
+}
+
+fn into_bytes(resp: Response) -> Result<Vec<u8>, io::Error> {
+    const BYTES_LIMIT: usize = 10 * 1_024 * 1_024;
+
+    let mut buf: Vec<u8> = vec![];
+    resp.into_reader()
+        .take((BYTES_LIMIT + 1) as u64)
+        .read_to_end(&mut buf)?;
+    if buf.len() > BYTES_LIMIT {
+        return Err(io::Error::new(
+            io::ErrorKind::Other,
+            "response too big for into_bytes",
+        ));
+    }
+
+    Ok(buf)
+}
+
+impl ConfigurableBlockchain for EsploraBlockchain {
+    type Config = super::EsploraBlockchainConfig;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        let mut agent_builder = ureq::AgentBuilder::new();
+
+        if let Some(timeout) = config.timeout {
+            agent_builder = agent_builder.timeout(Duration::from_secs(timeout));
+        }
+
+        if let Some(proxy) = &config.proxy {
+            agent_builder = agent_builder
+                .proxy(Proxy::new(proxy).map_err(|e| Error::Esplora(Box::new(e.into())))?);
+        }
+
+        let mut blockchain = EsploraBlockchain::new(config.base_url.as_str(), config.stop_gap)
+            .with_agent(agent_builder.build());
+
+        if let Some(concurrency) = config.concurrency {
+            blockchain = blockchain.with_concurrency(concurrency);
+        }
+
+        Ok(blockchain)
+    }
+}
+
+impl From<ureq::Error> for EsploraError {
+    fn from(e: ureq::Error) -> Self {
+        match e {
+            ureq::Error::Status(code, _) => EsploraError::HttpResponse(code),
+            e => EsploraError::Ureq(e),
+        }
+    }
+}

src/blockchain/mod.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Blockchain backends
 //!
@@ -34,18 +21,28 @@ use std::ops::Deref;
 use std::sync::mpsc::{channel, Receiver, Sender};
 use std::sync::Arc;
 
-use bitcoin::{Transaction, Txid};
+use bitcoin::{BlockHash, Transaction, Txid};
 
 use crate::database::BatchDatabase;
 use crate::error::Error;
-use crate::FeeRate;
+use crate::wallet::{wallet_name_from_descriptor, Wallet};
+use crate::{FeeRate, KeychainKind};
 
-#[cfg(any(feature = "electrum", feature = "esplora"))]
-pub(crate) mod utils;
-
-#[cfg(any(feature = "electrum", feature = "esplora", feature = "compact_filters"))]
+#[cfg(any(
+    feature = "electrum",
+    feature = "esplora",
+    feature = "compact_filters",
+    feature = "rpc"
+))]
 pub mod any;
-#[cfg(any(feature = "electrum", feature = "esplora", feature = "compact_filters"))]
+mod script_sync;
+
+#[cfg(any(
+    feature = "electrum",
+    feature = "esplora",
+    feature = "compact_filters",
+    feature = "rpc"
+))]
 pub use any::{AnyBlockchain, AnyBlockchainConfig};
 
 #[cfg(feature = "electrum")]
@@ -56,6 +53,14 @@ pub use self::electrum::ElectrumBlockchain;
 #[cfg(feature = "electrum")]
 pub use self::electrum::ElectrumBlockchainConfig;
 
+#[cfg(feature = "rpc")]
+#[cfg_attr(docsrs, doc(cfg(feature = "rpc")))]
+pub mod rpc;
+#[cfg(feature = "rpc")]
+pub use self::rpc::RpcBlockchain;
+#[cfg(feature = "rpc")]
+pub use self::rpc::RpcConfig;
+
 #[cfg(feature = "esplora")]
 #[cfg_attr(docsrs, doc(cfg(feature = "esplora")))]
 pub mod esplora;
@@ -65,6 +70,7 @@ pub use self::esplora::EsploraBlockchain;
 #[cfg(feature = "compact_filters")]
 #[cfg_attr(docsrs, doc(cfg(feature = "compact_filters")))]
 pub mod compact_filters;
+
 #[cfg(feature = "compact_filters")]
 pub use self::compact_filters::CompactFiltersBlockchain;
 
@@ -81,29 +87,57 @@ pub enum Capability {
 
 /// Trait that defines the actions that must be supported by a blockchain backend
 #[maybe_async]
-pub trait Blockchain {
+pub trait Blockchain: WalletSync + GetHeight + GetTx + GetBlockHash {
     /// Return the set of [`Capability`] supported by this backend
     fn get_capabilities(&self) -> HashSet<Capability>;
+    /// Broadcast a transaction
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error>;
+    /// Estimate the fee rate required to confirm a transaction in a given `target` of blocks
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error>;
+}
 
+/// Trait for getting the current height of the blockchain.
+#[maybe_async]
+pub trait GetHeight {
+    /// Return the current height
+    fn get_height(&self) -> Result<u32, Error>;
+}
+
+#[maybe_async]
+/// Trait for getting a transaction by txid
+pub trait GetTx {
+    /// Fetch a transaction given its txid
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
+}
+
+#[maybe_async]
+/// Trait for getting block hash by block height
+pub trait GetBlockHash {
+    /// fetch block hash given its height
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error>;
+}
+
+/// Trait for blockchains that can sync by updating the database directly.
+#[maybe_async]
+pub trait WalletSync {
     /// Setup the backend and populate the internal database for the first time
     ///
-    /// This method is the equivalent of [`Blockchain::sync`], but it's guaranteed to only be
+    /// This method is the equivalent of [`Self::wallet_sync`], but it's guaranteed to only be
     /// called once, at the first [`Wallet::sync`](crate::wallet::Wallet::sync).
     ///
     /// The rationale behind the distinction between `sync` and `setup` is that some custom backends
     /// might need to perform specific actions only the first time they are synced.
     ///
     /// For types that do not have that distinction, only this method can be implemented, since
-    /// [`Blockchain::sync`] defaults to calling this internally if not overridden.
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error>;
+    /// [`WalletSync::wallet_sync`] defaults to calling this internally if not overridden.
     /// Populate the internal database with transactions and UTXOs
-    ///
-    /// If not overridden, it defaults to calling [`Blockchain::setup`] internally.
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error>;
+
+    /// If not overridden, it defaults to calling [`Self::wallet_setup`] internally.
     ///
     /// This method should implement the logic required to iterate over the list of the wallet's
     /// script_pubkeys using [`Database::iter_script_pubkeys`] and look for relevant transactions
@@ -120,24 +154,13 @@ pub trait Blockchain {
     /// [`BatchOperations::set_tx`]: crate::database::BatchOperations::set_tx
     /// [`BatchOperations::set_utxo`]: crate::database::BatchOperations::set_utxo
     /// [`BatchOperations::del_utxo`]: crate::database::BatchOperations::del_utxo
-    fn sync<D: BatchDatabase, P: 'static + Progress>(
+    fn wallet_sync<D: BatchDatabase>(
         &self,
-        stop_gap: Option<usize>,
         database: &mut D,
-        progress_update: P,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error> {
-        maybe_await!(self.setup(stop_gap, database, progress_update))
+        maybe_await!(self.wallet_setup(database, progress_update))
     }
-
-    /// Fetch a transaction from the blockchain given its txid
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
-    /// Broadcast a transaction
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error>;
-
-    /// Return the current height
-    fn get_height(&self) -> Result<u32, Error>;
-    /// Estimate the fee rate required to confirm a transaction in a given `target` of blocks
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error>;
 }
 
 /// Trait for [`Blockchain`] types that can be created given a configuration
@@ -149,12 +172,112 @@ pub trait ConfigurableBlockchain: Blockchain + Sized {
     fn from_config(config: &Self::Config) -> Result<Self, Error>;
 }
 
+/// Trait for blockchains that don't contain any state
+///
+/// Statless blockchains can be used to sync multiple wallets with different descriptors.
+///
+/// [`BlockchainFactory`] is automatically implemented for `Arc<T>` where `T` is a stateless
+/// blockchain.
+pub trait StatelessBlockchain: Blockchain {}
+
+/// Trait for a factory of blockchains that share the underlying connection or configuration
+#[cfg_attr(
+    not(feature = "async-interface"),
+    doc = r##"
+## Example
+
+This example shows how to sync multiple walles and return the sum of their balances
+
+```no_run
+# use bdk::Error;
+# use bdk::blockchain::*;
+# use bdk::database::*;
+# use bdk::wallet::*;
+# use bdk::*;
+fn sum_of_balances<B: BlockchainFactory>(blockchain_factory: B, wallets: &[Wallet<MemoryDatabase>]) -> Result<u64, Error> {
+    Ok(wallets
+        .iter()
+        .map(|w| -> Result<_, Error> {
+            blockchain_factory.sync_wallet(&w, None, SyncOptions::default())?;
+            w.get_balance()
+        })
+        .collect::<Result<Vec<_>, _>>()?
+        .into_iter()
+        .sum())
+}
+```
+"##
+)]
+pub trait BlockchainFactory {
+    /// The type returned when building a blockchain from this factory
+    type Inner: Blockchain;
+
+    /// Build a new blockchain for the given descriptor wallet_name
+    ///
+    /// If `override_skip_blocks` is `None`, the returned blockchain will inherit the number of blocks
+    /// from the factory. Since it's not possible to override the value to `None`, set it to
+    /// `Some(0)` to rescan from the genesis.
+    fn build(
+        &self,
+        wallet_name: &str,
+        override_skip_blocks: Option<u32>,
+    ) -> Result<Self::Inner, Error>;
+
+    /// Build a new blockchain for a given wallet
+    ///
+    /// Internally uses [`wallet_name_from_descriptor`] to derive the name, and then calls
+    /// [`BlockchainFactory::build`] to create the blockchain instance.
+    fn build_for_wallet<D: BatchDatabase>(
+        &self,
+        wallet: &Wallet<D>,
+        override_skip_blocks: Option<u32>,
+    ) -> Result<Self::Inner, Error> {
+        let wallet_name = wallet_name_from_descriptor(
+            wallet.public_descriptor(KeychainKind::External)?.unwrap(),
+            wallet.public_descriptor(KeychainKind::Internal)?,
+            wallet.network(),
+            wallet.secp_ctx(),
+        )?;
+        self.build(&wallet_name, override_skip_blocks)
+    }
+
+    /// Use [`BlockchainFactory::build_for_wallet`] to get a blockchain, then sync the wallet
+    ///
+    /// This can be used when a new blockchain would only be used to sync a wallet and then
+    /// immediately dropped. Keep in mind that specific blockchain factories may perform slow
+    /// operations to build a blockchain for a given wallet, so if a wallet needs to be synced
+    /// often it's recommended to use [`BlockchainFactory::build_for_wallet`] to reuse the same
+    /// blockchain multiple times.
+    #[cfg(not(any(target_arch = "wasm32", feature = "async-interface")))]
+    #[cfg_attr(
+        docsrs,
+        doc(cfg(not(any(target_arch = "wasm32", feature = "async-interface"))))
+    )]
+    fn sync_wallet<D: BatchDatabase>(
+        &self,
+        wallet: &Wallet<D>,
+        override_skip_blocks: Option<u32>,
+        sync_options: crate::wallet::SyncOptions,
+    ) -> Result<(), Error> {
+        let blockchain = self.build_for_wallet(wallet, override_skip_blocks)?;
+        wallet.sync(&blockchain, sync_options)
+    }
+}
+
+impl<T: StatelessBlockchain> BlockchainFactory for Arc<T> {
+    type Inner = Self;
+
+    fn build(&self, _wallet_name: &str, _override_skip_blocks: Option<u32>) -> Result<Self, Error> {
+        Ok(Arc::clone(self))
+    }
+}
+
 /// Data sent with a progress update over a [`channel`]
 pub type ProgressData = (f32, Option<String>);
 
-/// Trait for types that can receive and process progress updates during [`Blockchain::sync`] and
-/// [`Blockchain::setup`]
-pub trait Progress: Send {
+/// Trait for types that can receive and process progress updates during [`WalletSync::wallet_sync`] and
+/// [`WalletSync::wallet_setup`]
+pub trait Progress: Send + 'static + core::fmt::Debug {
     /// Send a new progress update
     ///
     /// The `progress` value should be in the range 0.0 - 100.0, and the `message` value is an
@@ -179,7 +302,7 @@ impl Progress for Sender<ProgressData> {
 }
 
 /// Type that implements [`Progress`] and drops every update received
-#[derive(Clone)]
+#[derive(Clone, Copy, Default, Debug)]
 pub struct NoopProgress;
 
 /// Create a new instance of [`NoopProgress`]
@@ -194,10 +317,10 @@ impl Progress for NoopProgress {
 }
 
 /// Type that implements [`Progress`] and logs at level `INFO` every update received
-#[derive(Clone)]
+#[derive(Clone, Copy, Default, Debug)]
 pub struct LogProgress;
 
-/// Create a nwe instance of [`LogProgress`]
+/// Create a new instance of [`LogProgress`]
 pub fn log_progress() -> LogProgress {
     LogProgress
 }
@@ -220,35 +343,51 @@ impl<T: Blockchain> Blockchain for Arc<T> {
         maybe_await!(self.deref().get_capabilities())
     }
 
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(self.deref().setup(stop_gap, database, progress_update))
-    }
-
-    fn sync<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(self.deref().sync(stop_gap, database, progress_update))
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        maybe_await!(self.deref().get_tx(txid))
-    }
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         maybe_await!(self.deref().broadcast(tx))
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        maybe_await!(self.deref().get_height())
-    }
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         maybe_await!(self.deref().estimate_fee(target))
     }
 }
+
+#[maybe_async]
+impl<T: GetTx> GetTx for Arc<T> {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        maybe_await!(self.deref().get_tx(txid))
+    }
+}
+
+#[maybe_async]
+impl<T: GetHeight> GetHeight for Arc<T> {
+    fn get_height(&self) -> Result<u32, Error> {
+        maybe_await!(self.deref().get_height())
+    }
+}
+
+#[maybe_async]
+impl<T: GetBlockHash> GetBlockHash for Arc<T> {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        maybe_await!(self.deref().get_block_hash(height))
+    }
+}
+
+#[maybe_async]
+impl<T: WalletSync> WalletSync for Arc<T> {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(self.deref().wallet_setup(database, progress_update))
+    }
+
+    fn wallet_sync<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(self.deref().wallet_sync(database, progress_update))
+    }
+}

src/blockchain/rpc.rs

@@ -0,0 +1,635 @@
+// Bitcoin Dev Kit
+// Written in 2021 by Riccardo Casatta <riccardo@casatta.it>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Rpc Blockchain
+//!
+//! Backend that gets blockchain data from Bitcoin Core RPC
+//!
+//! This is an **EXPERIMENTAL** feature, API and other major changes are expected.
+//!
+//! ## Example
+//!
+//! ```no_run
+//! # use bdk::blockchain::{RpcConfig, RpcBlockchain, ConfigurableBlockchain, rpc::Auth};
+//! let config = RpcConfig {
+//!     url: "127.0.0.1:18332".to_string(),
+//!     auth: Auth::Cookie {
+//!         file: "/home/user/.bitcoin/.cookie".into(),
+//!     },
+//!     network: bdk::bitcoin::Network::Testnet,
+//!     wallet_name: "wallet_name".to_string(),
+//!     skip_blocks: None,
+//! };
+//! let blockchain = RpcBlockchain::from_config(&config);
+//! ```
+
+use crate::bitcoin::consensus::deserialize;
+use crate::bitcoin::hashes::hex::ToHex;
+use crate::bitcoin::{Address, Network, OutPoint, Transaction, TxOut, Txid};
+use crate::blockchain::*;
+use crate::database::{BatchDatabase, DatabaseUtils};
+use crate::descriptor::get_checksum;
+use crate::{BlockTime, Error, FeeRate, KeychainKind, LocalUtxo, TransactionDetails};
+use bitcoincore_rpc::json::{
+    GetAddressInfoResultLabel, ImportMultiOptions, ImportMultiRequest,
+    ImportMultiRequestScriptPubkey, ImportMultiRescanSince,
+};
+use bitcoincore_rpc::jsonrpc::serde_json::{json, Value};
+use bitcoincore_rpc::Auth as RpcAuth;
+use bitcoincore_rpc::{Client, RpcApi};
+use log::debug;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::path::PathBuf;
+use std::str::FromStr;
+
+/// The main struct for RPC backend implementing the [crate::blockchain::Blockchain] trait
+#[derive(Debug)]
+pub struct RpcBlockchain {
+    /// Rpc client to the node, includes the wallet name
+    client: Client,
+    /// Whether the wallet is a "descriptor" or "legacy" wallet in Core
+    is_descriptors: bool,
+    /// Blockchain capabilities, cached here at startup
+    capabilities: HashSet<Capability>,
+    /// Skip this many blocks of the blockchain at the first rescan, if None the rescan is done from the genesis block
+    skip_blocks: Option<u32>,
+
+    /// This is a fixed Address used as a hack key to store information on the node
+    _storage_address: Address,
+}
+
+/// RpcBlockchain configuration options
+#[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+pub struct RpcConfig {
+    /// The bitcoin node url
+    pub url: String,
+    /// The bitcoin node authentication mechanism
+    pub auth: Auth,
+    /// The network we are using (it will be checked the bitcoin node network matches this)
+    pub network: Network,
+    /// The wallet name in the bitcoin node, consider using [crate::wallet::wallet_name_from_descriptor] for this
+    pub wallet_name: String,
+    /// Skip this many blocks of the blockchain at the first rescan, if None the rescan is done from the genesis block
+    pub skip_blocks: Option<u32>,
+}
+
+/// This struct is equivalent to [bitcoincore_rpc::Auth] but it implements [serde::Serialize]
+/// To be removed once upstream equivalent is implementing Serialize (json serialization format
+/// should be the same), see [rust-bitcoincore-rpc/pull/181](https://github.com/rust-bitcoin/rust-bitcoincore-rpc/pull/181)
+#[derive(Clone, Debug, Hash, Eq, PartialEq, Ord, PartialOrd, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+#[serde(untagged)]
+pub enum Auth {
+    /// None authentication
+    None,
+    /// Authentication with username and password, usually [Auth::Cookie] should be preferred
+    UserPass {
+        /// Username
+        username: String,
+        /// Password
+        password: String,
+    },
+    /// Authentication with a cookie file
+    Cookie {
+        /// Cookie file
+        file: PathBuf,
+    },
+}
+
+impl From<Auth> for RpcAuth {
+    fn from(auth: Auth) -> Self {
+        match auth {
+            Auth::None => RpcAuth::None,
+            Auth::UserPass { username, password } => RpcAuth::UserPass(username, password),
+            Auth::Cookie { file } => RpcAuth::CookieFile(file),
+        }
+    }
+}
+
+impl RpcBlockchain {
+    fn get_node_synced_height(&self) -> Result<u32, Error> {
+        let info = self.client.get_address_info(&self._storage_address)?;
+        if let Some(GetAddressInfoResultLabel::Simple(label)) = info.labels.first() {
+            Ok(label
+                .parse::<u32>()
+                .unwrap_or_else(|_| self.skip_blocks.unwrap_or(0)))
+        } else {
+            Ok(self.skip_blocks.unwrap_or(0))
+        }
+    }
+
+    /// Set the synced height in the core node by using a label of a fixed address so that
+    /// another client with the same descriptor doesn't rescan the blockchain
+    fn set_node_synced_height(&self, height: u32) -> Result<(), Error> {
+        Ok(self
+            .client
+            .set_label(&self._storage_address, &height.to_string())?)
+    }
+}
+
+impl Blockchain for RpcBlockchain {
+    fn get_capabilities(&self) -> HashSet<Capability> {
+        self.capabilities.clone()
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        Ok(self.client.send_raw_transaction(tx).map(|_| ())?)
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        let sat_per_kb = self
+            .client
+            .estimate_smart_fee(target as u16, None)?
+            .fee_rate
+            .ok_or(Error::FeeRateUnavailable)?
+            .as_sat() as f64;
+
+        Ok(FeeRate::from_sat_per_vb((sat_per_kb / 1000f64) as f32))
+    }
+}
+
+impl GetTx for RpcBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(Some(self.client.get_raw_transaction(txid, None)?))
+    }
+}
+
+impl GetHeight for RpcBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.client.get_blockchain_info().map(|i| i.blocks as u32)?)
+    }
+}
+
+impl GetBlockHash for RpcBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        Ok(self.client.get_block_hash(height)?)
+    }
+}
+
+impl WalletSync for RpcBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        let mut scripts_pubkeys = database.iter_script_pubkeys(Some(KeychainKind::External))?;
+        scripts_pubkeys.extend(database.iter_script_pubkeys(Some(KeychainKind::Internal))?);
+        debug!(
+            "importing {} script_pubkeys (some maybe already imported)",
+            scripts_pubkeys.len()
+        );
+
+        if self.is_descriptors {
+            // Core still doesn't support complex descriptors like BDK, but when the wallet type is
+            // "descriptors" we should import individual addresses using `importdescriptors` rather
+            // than `importmulti`, using the `raw()` descriptor which allows us to specify an
+            // arbitrary script
+            let requests = Value::Array(
+                scripts_pubkeys
+                    .iter()
+                    .map(|s| {
+                        let desc = format!("raw({})", s.to_hex());
+                        json!({
+                            "timestamp": "now",
+                            "desc": format!("{}#{}", desc, get_checksum(&desc).unwrap()),
+                        })
+                    })
+                    .collect(),
+            );
+
+            let res: Vec<Value> = self.client.call("importdescriptors", &[requests])?;
+            res.into_iter()
+                .map(|v| match v["success"].as_bool() {
+                    Some(true) => Ok(()),
+                    Some(false) => Err(Error::Generic(
+                        v["error"]["message"]
+                            .as_str()
+                            .unwrap_or("Unknown error")
+                            .to_string(),
+                    )),
+                    _ => Err(Error::Generic("Unexpected response from Core".to_string())),
+                })
+                .collect::<Result<Vec<_>, _>>()?;
+        } else {
+            let requests: Vec<_> = scripts_pubkeys
+                .iter()
+                .map(|s| ImportMultiRequest {
+                    timestamp: ImportMultiRescanSince::Timestamp(0),
+                    script_pubkey: Some(ImportMultiRequestScriptPubkey::Script(s)),
+                    watchonly: Some(true),
+                    ..Default::default()
+                })
+                .collect();
+            let options = ImportMultiOptions {
+                rescan: Some(false),
+            };
+            self.client.import_multi(&requests, Some(&options))?;
+        }
+
+        loop {
+            let current_height = self.get_height()?;
+
+            // min because block invalidate may cause height to go down
+            let node_synced = self.get_node_synced_height()?.min(current_height);
+
+            let sync_up_to = node_synced.saturating_add(10_000).min(current_height);
+
+            debug!("rescan_blockchain from:{} to:{}", node_synced, sync_up_to);
+            self.client
+                .rescan_blockchain(Some(node_synced as usize), Some(sync_up_to as usize))?;
+            progress_update.update((sync_up_to as f32) / (current_height as f32), None)?;
+
+            self.set_node_synced_height(sync_up_to)?;
+
+            if sync_up_to == current_height {
+                break;
+            }
+        }
+
+        self.wallet_sync(database, progress_update)
+    }
+
+    fn wallet_sync<D: BatchDatabase>(
+        &self,
+        db: &mut D,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        let mut indexes = HashMap::new();
+        for keykind in &[KeychainKind::External, KeychainKind::Internal] {
+            indexes.insert(*keykind, db.get_last_index(*keykind)?.unwrap_or(0));
+        }
+
+        let mut known_txs: HashMap<_, _> = db
+            .iter_txs(true)?
+            .into_iter()
+            .map(|tx| (tx.txid, tx))
+            .collect();
+        let known_utxos: HashSet<_> = db.iter_utxos()?.into_iter().collect();
+
+        //TODO list_since_blocks would be more efficient
+        let current_utxo = self
+            .client
+            .list_unspent(Some(0), None, None, Some(true), None)?;
+        debug!("current_utxo len {}", current_utxo.len());
+
+        //TODO supported up to 1_000 txs, should use since_blocks or do paging
+        let list_txs = self
+            .client
+            .list_transactions(None, Some(1_000), None, Some(true))?;
+        let mut list_txs_ids = HashSet::new();
+
+        for tx_result in list_txs.iter().filter(|t| {
+            // list_txs returns all conflicting txs, we want to
+            // filter out replaced tx => unconfirmed and not in the mempool
+            t.info.confirmations > 0 || self.client.get_mempool_entry(&t.info.txid).is_ok()
+        }) {
+            let txid = tx_result.info.txid;
+            list_txs_ids.insert(txid);
+            if let Some(mut known_tx) = known_txs.get_mut(&txid) {
+                let confirmation_time =
+                    BlockTime::new(tx_result.info.blockheight, tx_result.info.blocktime);
+                if confirmation_time != known_tx.confirmation_time {
+                    // reorg may change tx height
+                    debug!(
+                        "updating tx({}) confirmation time to: {:?}",
+                        txid, confirmation_time
+                    );
+                    known_tx.confirmation_time = confirmation_time;
+                    db.set_tx(known_tx)?;
+                }
+            } else {
+                //TODO check there is already the raw tx in db?
+                let tx_result = self.client.get_transaction(&txid, Some(true))?;
+                let tx: Transaction = deserialize(&tx_result.hex)?;
+                let mut received = 0u64;
+                let mut sent = 0u64;
+                for output in tx.output.iter() {
+                    if let Ok(Some((kind, index))) =
+                        db.get_path_from_script_pubkey(&output.script_pubkey)
+                    {
+                        if index > *indexes.get(&kind).unwrap() {
+                            indexes.insert(kind, index);
+                        }
+                        received += output.value;
+                    }
+                }
+
+                for input in tx.input.iter() {
+                    if let Some(previous_output) = db.get_previous_output(&input.previous_output)? {
+                        if db.is_mine(&previous_output.script_pubkey)? {
+                            sent += previous_output.value;
+                        }
+                    }
+                }
+
+                let td = TransactionDetails {
+                    transaction: Some(tx),
+                    txid: tx_result.info.txid,
+                    confirmation_time: BlockTime::new(
+                        tx_result.info.blockheight,
+                        tx_result.info.blocktime,
+                    ),
+                    received,
+                    sent,
+                    fee: tx_result.fee.map(|f| f.as_sat().unsigned_abs()),
+                };
+                debug!(
+                    "saving tx: {} tx_result.fee:{:?} td.fees:{:?}",
+                    td.txid, tx_result.fee, td.fee
+                );
+                db.set_tx(&td)?;
+            }
+        }
+
+        for known_txid in known_txs.keys() {
+            if !list_txs_ids.contains(known_txid) {
+                debug!("removing tx: {}", known_txid);
+                db.del_tx(known_txid, false)?;
+            }
+        }
+
+        // Filter out trasactions that are for script pubkeys that aren't in this wallet.
+        let current_utxos = current_utxo
+            .into_iter()
+            .filter_map(
+                |u| match db.get_path_from_script_pubkey(&u.script_pub_key) {
+                    Err(e) => Some(Err(e)),
+                    Ok(None) => None,
+                    Ok(Some(path)) => Some(Ok(LocalUtxo {
+                        outpoint: OutPoint::new(u.txid, u.vout),
+                        keychain: path.0,
+                        txout: TxOut {
+                            value: u.amount.as_sat(),
+                            script_pubkey: u.script_pub_key,
+                        },
+                        is_spent: false,
+                    })),
+                },
+            )
+            .collect::<Result<HashSet<_>, Error>>()?;
+
+        let spent: HashSet<_> = known_utxos.difference(&current_utxos).collect();
+        for utxo in spent {
+            debug!("setting as spent utxo: {:?}", utxo);
+            let mut spent_utxo = utxo.clone();
+            spent_utxo.is_spent = true;
+            db.set_utxo(&spent_utxo)?;
+        }
+        let received: HashSet<_> = current_utxos.difference(&known_utxos).collect();
+        for utxo in received {
+            debug!("adding utxo: {:?}", utxo);
+            db.set_utxo(utxo)?;
+        }
+
+        for (keykind, index) in indexes {
+            debug!("{:?} max {}", keykind, index);
+            db.set_last_index(keykind, index)?;
+        }
+
+        Ok(())
+    }
+}
+
+impl ConfigurableBlockchain for RpcBlockchain {
+    type Config = RpcConfig;
+
+    /// Returns RpcBlockchain backend creating an RPC client to a specific wallet named as the descriptor's checksum
+    /// if it's the first time it creates the wallet in the node and upon return is granted the wallet is loaded
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        let wallet_name = config.wallet_name.clone();
+        let wallet_url = format!("{}/wallet/{}", config.url, &wallet_name);
+        debug!("connecting to {} auth:{:?}", wallet_url, config.auth);
+
+        let client = Client::new(wallet_url.as_str(), config.auth.clone().into())?;
+        let rpc_version = client.version()?;
+
+        let loaded_wallets = client.list_wallets()?;
+        if loaded_wallets.contains(&wallet_name) {
+            debug!("wallet already loaded {:?}", wallet_name);
+        } else if list_wallet_dir(&client)?.contains(&wallet_name) {
+            client.load_wallet(&wallet_name)?;
+            debug!("wallet loaded {:?}", wallet_name);
+        } else {
+            // pre-0.21 use legacy wallets
+            if rpc_version < 210_000 {
+                client.create_wallet(&wallet_name, Some(true), None, None, None)?;
+            } else {
+                // TODO: move back to api call when https://github.com/rust-bitcoin/rust-bitcoincore-rpc/issues/225 is closed
+                let args = [
+                    Value::String(wallet_name.clone()),
+                    Value::Bool(true),
+                    Value::Bool(false),
+                    Value::Null,
+                    Value::Bool(false),
+                    Value::Bool(true),
+                ];
+                let _: Value = client.call("createwallet", &args)?;
+            }
+
+            debug!("wallet created {:?}", wallet_name);
+        }
+
+        let is_descriptors = is_wallet_descriptor(&client)?;
+
+        let blockchain_info = client.get_blockchain_info()?;
+        let network = match blockchain_info.chain.as_str() {
+            "main" => Network::Bitcoin,
+            "test" => Network::Testnet,
+            "regtest" => Network::Regtest,
+            "signet" => Network::Signet,
+            _ => return Err(Error::Generic("Invalid network".to_string())),
+        };
+        if network != config.network {
+            return Err(Error::InvalidNetwork {
+                requested: config.network,
+                found: network,
+            });
+        }
+
+        let mut capabilities: HashSet<_> = vec![Capability::FullHistory].into_iter().collect();
+        if rpc_version >= 210_000 {
+            let info: HashMap<String, Value> = client.call("getindexinfo", &[]).unwrap();
+            if info.contains_key("txindex") {
+                capabilities.insert(Capability::GetAnyTx);
+                capabilities.insert(Capability::AccurateFees);
+            }
+        }
+
+        // this is just a fixed address used only to store a label containing the synced height in the node
+        let mut storage_address =
+            Address::from_str("bc1qst0rewf0wm4kw6qn6kv0e5tc56nkf9yhcxlhqv").unwrap();
+        storage_address.network = network;
+
+        Ok(RpcBlockchain {
+            client,
+            capabilities,
+            is_descriptors,
+            _storage_address: storage_address,
+            skip_blocks: config.skip_blocks,
+        })
+    }
+}
+
+/// return the wallets available in default wallet directory
+//TODO use bitcoincore_rpc method when PR #179 lands
+fn list_wallet_dir(client: &Client) -> Result<Vec<String>, Error> {
+    #[derive(Deserialize)]
+    struct Name {
+        name: String,
+    }
+    #[derive(Deserialize)]
+    struct CallResult {
+        wallets: Vec<Name>,
+    }
+
+    let result: CallResult = client.call("listwalletdir", &[])?;
+    Ok(result.wallets.into_iter().map(|n| n.name).collect())
+}
+
+/// Returns whether a wallet is legacy or descriptors by calling `getwalletinfo`.
+///
+/// This API is mapped by bitcoincore_rpc, but it doesn't have the fields we need (either
+/// "descriptors" or "format") so we have to call the RPC manually
+fn is_wallet_descriptor(client: &Client) -> Result<bool, Error> {
+    #[derive(Deserialize)]
+    struct CallResult {
+        descriptors: Option<bool>,
+    }
+
+    let result: CallResult = client.call("getwalletinfo", &[])?;
+    Ok(result.descriptors.unwrap_or(false))
+}
+
+/// Factory of [`RpcBlockchain`] instances, implements [`BlockchainFactory`]
+///
+/// Internally caches the node url and authentication params and allows getting many different [`RpcBlockchain`]
+/// objects for different wallet names and with different rescan heights.
+///
+/// ## Example
+///
+/// ```no_run
+/// # use bdk::bitcoin::Network;
+/// # use bdk::blockchain::BlockchainFactory;
+/// # use bdk::blockchain::rpc::{Auth, RpcBlockchainFactory};
+/// # fn main() -> Result<(), Box<dyn std::error::Error>> {
+/// let factory = RpcBlockchainFactory {
+///     url: "http://127.0.0.1:18332".to_string(),
+///     auth: Auth::Cookie {
+///         file: "/home/user/.bitcoin/.cookie".into(),
+///     },
+///     network: Network::Testnet,
+///     wallet_name_prefix: Some("prefix-".to_string()),
+///     default_skip_blocks: 100_000,
+/// };
+/// let main_wallet_blockchain = factory.build("main_wallet", Some(200_000))?;
+/// # Ok(())
+/// # }
+/// ```
+#[derive(Debug, Clone)]
+pub struct RpcBlockchainFactory {
+    /// The bitcoin node url
+    pub url: String,
+    /// The bitcoin node authentication mechanism
+    pub auth: Auth,
+    /// The network we are using (it will be checked the bitcoin node network matches this)
+    pub network: Network,
+    /// The optional prefix used to build the full wallet name for blockchains
+    pub wallet_name_prefix: Option<String>,
+    /// Default number of blocks to skip which will be inherited by blockchain unless overridden
+    pub default_skip_blocks: u32,
+}
+
+impl BlockchainFactory for RpcBlockchainFactory {
+    type Inner = RpcBlockchain;
+
+    fn build(
+        &self,
+        checksum: &str,
+        override_skip_blocks: Option<u32>,
+    ) -> Result<Self::Inner, Error> {
+        RpcBlockchain::from_config(&RpcConfig {
+            url: self.url.clone(),
+            auth: self.auth.clone(),
+            network: self.network,
+            wallet_name: format!(
+                "{}{}",
+                self.wallet_name_prefix.as_ref().unwrap_or(&String::new()),
+                checksum
+            ),
+            skip_blocks: Some(override_skip_blocks.unwrap_or(self.default_skip_blocks)),
+        })
+    }
+}
+
+#[cfg(test)]
+#[cfg(any(feature = "test-rpc", feature = "test-rpc-legacy"))]
+mod test {
+    use super::*;
+    use crate::testutils::blockchain_tests::TestClient;
+
+    use bitcoin::Network;
+    use bitcoincore_rpc::RpcApi;
+
+    crate::bdk_blockchain_tests! {
+        fn test_instance(test_client: &TestClient) -> RpcBlockchain {
+            let config = RpcConfig {
+                url: test_client.bitcoind.rpc_url(),
+                auth: Auth::Cookie { file: test_client.bitcoind.params.cookie_file.clone() },
+                network: Network::Regtest,
+                wallet_name: format!("client-wallet-test-{}", std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_nanos() ),
+                skip_blocks: None,
+            };
+            RpcBlockchain::from_config(&config).unwrap()
+        }
+    }
+
+    fn get_factory() -> (TestClient, RpcBlockchainFactory) {
+        let test_client = TestClient::default();
+
+        let factory = RpcBlockchainFactory {
+            url: test_client.bitcoind.rpc_url(),
+            auth: Auth::Cookie {
+                file: test_client.bitcoind.params.cookie_file.clone(),
+            },
+            network: Network::Regtest,
+            wallet_name_prefix: Some("prefix-".into()),
+            default_skip_blocks: 0,
+        };
+
+        (test_client, factory)
+    }
+
+    #[test]
+    fn test_rpc_blockchain_factory() {
+        let (_test_client, factory) = get_factory();
+
+        let a = factory.build("aaaaaa", None).unwrap();
+        assert_eq!(a.skip_blocks, Some(0));
+        assert_eq!(
+            a.client
+                .get_wallet_info()
+                .expect("Node connection isn't working")
+                .wallet_name,
+            "prefix-aaaaaa"
+        );
+
+        let b = factory.build("bbbbbb", Some(100)).unwrap();
+        assert_eq!(b.skip_blocks, Some(100));
+        assert_eq!(
+            b.client
+                .get_wallet_info()
+                .expect("Node connection isn't working")
+                .wallet_name,
+            "prefix-bbbbbb"
+        );
+    }
+}

src/blockchain/script_sync.rs

@@ -0,0 +1,433 @@
+/*!
+This models a how a sync happens where you have a server that you send your script pubkeys to and it
+returns associated transactions i.e. electrum.
+*/
+#![allow(dead_code)]
+use crate::{
+    database::{BatchDatabase, BatchOperations, DatabaseUtils},
+    wallet::time::Instant,
+    BlockTime, Error, KeychainKind, LocalUtxo, TransactionDetails,
+};
+use bitcoin::{OutPoint, Script, Transaction, TxOut, Txid};
+use log::*;
+use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque};
+
+/// A request for on-chain information
+pub enum Request<'a, D: BatchDatabase> {
+    /// A request for transactions related to script pubkeys.
+    Script(ScriptReq<'a, D>),
+    /// A request for confirmation times for some transactions.
+    Conftime(ConftimeReq<'a, D>),
+    /// A request for full transaction details of some transactions.
+    Tx(TxReq<'a, D>),
+    /// Requests are finished here's a batch database update to reflect data gathered.
+    Finish(D::Batch),
+}
+
+/// starts a sync
+pub fn start<D: BatchDatabase>(db: &D, stop_gap: usize) -> Result<Request<'_, D>, Error> {
+    use rand::seq::SliceRandom;
+    let mut keychains = vec![KeychainKind::Internal, KeychainKind::External];
+    // shuffling improve privacy, the server doesn't know my first request is from my internal or external addresses
+    keychains.shuffle(&mut rand::thread_rng());
+    let keychain = keychains.pop().unwrap();
+    let scripts_needed = db
+        .iter_script_pubkeys(Some(keychain))?
+        .into_iter()
+        .collect();
+    let state = State::new(db);
+
+    Ok(Request::Script(ScriptReq {
+        state,
+        scripts_needed,
+        script_index: 0,
+        stop_gap,
+        keychain,
+        next_keychains: keychains,
+    }))
+}
+
+pub struct ScriptReq<'a, D: BatchDatabase> {
+    state: State<'a, D>,
+    script_index: usize,
+    scripts_needed: VecDeque<Script>,
+    stop_gap: usize,
+    keychain: KeychainKind,
+    next_keychains: Vec<KeychainKind>,
+}
+
+/// The sync starts by returning script pubkeys we are interested in.
+impl<'a, D: BatchDatabase> ScriptReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Script> + Clone {
+        self.scripts_needed.iter()
+    }
+
+    pub fn satisfy(
+        mut self,
+        // we want to know the txids assoiciated with the script and their height
+        txids: Vec<Vec<(Txid, Option<u32>)>>,
+    ) -> Result<Request<'a, D>, Error> {
+        for (txid_list, script) in txids.iter().zip(self.scripts_needed.iter()) {
+            debug!(
+                "found {} transactions for script pubkey {}",
+                txid_list.len(),
+                script
+            );
+            if !txid_list.is_empty() {
+                // the address is active
+                self.state
+                    .last_active_index
+                    .insert(self.keychain, self.script_index);
+            }
+
+            for (txid, height) in txid_list {
+                // have we seen this txid already?
+                match self.state.db.get_tx(txid, true)? {
+                    Some(mut details) => {
+                        let old_height = details.confirmation_time.as_ref().map(|x| x.height);
+                        match (old_height, height) {
+                            (None, Some(_)) => {
+                                // It looks like the tx has confirmed since we last saw it -- we
+                                // need to know the confirmation time.
+                                self.state.tx_missing_conftime.insert(*txid, details);
+                            }
+                            (Some(old_height), Some(new_height)) if old_height != *new_height => {
+                                // The height of the tx has changed !? -- It's a reorg get the new confirmation time.
+                                self.state.tx_missing_conftime.insert(*txid, details);
+                            }
+                            (Some(_), None) => {
+                                // A re-org where the tx is not in the chain anymore.
+                                details.confirmation_time = None;
+                                self.state.finished_txs.push(details);
+                            }
+                            _ => self.state.finished_txs.push(details),
+                        }
+                    }
+                    None => {
+                        // we've never seen it let's get the whole thing
+                        self.state.tx_needed.insert(*txid);
+                    }
+                };
+            }
+
+            self.script_index += 1;
+        }
+
+        for _ in txids {
+            self.scripts_needed.pop_front();
+        }
+
+        let last_active_index = self
+            .state
+            .last_active_index
+            .get(&self.keychain)
+            .map(|x| x + 1)
+            .unwrap_or(0); // so no addresses active maps to 0
+
+        Ok(
+            if self.script_index > last_active_index + self.stop_gap
+                || self.scripts_needed.is_empty()
+            {
+                debug!(
+                    "finished scanning for transactions for keychain {:?} at index {}",
+                    self.keychain, last_active_index
+                );
+                // we're done here -- check if we need to do the next keychain
+                if let Some(keychain) = self.next_keychains.pop() {
+                    self.keychain = keychain;
+                    self.script_index = 0;
+                    self.scripts_needed = self
+                        .state
+                        .db
+                        .iter_script_pubkeys(Some(keychain))?
+                        .into_iter()
+                        .collect();
+                    Request::Script(self)
+                } else {
+                    Request::Tx(TxReq { state: self.state })
+                }
+            } else {
+                Request::Script(self)
+            },
+        )
+    }
+}
+
+/// Then we get full transactions
+pub struct TxReq<'a, D> {
+    state: State<'a, D>,
+}
+
+impl<'a, D: BatchDatabase> TxReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Txid> + Clone {
+        self.state.tx_needed.iter()
+    }
+
+    pub fn satisfy(
+        mut self,
+        tx_details: Vec<(Vec<Option<TxOut>>, Transaction)>,
+    ) -> Result<Request<'a, D>, Error> {
+        let tx_details: Vec<TransactionDetails> = tx_details
+            .into_iter()
+            .zip(self.state.tx_needed.iter())
+            .map(|((vout, tx), txid)| {
+                debug!("found tx_details for {}", txid);
+                assert_eq!(tx.txid(), *txid);
+                let mut sent: u64 = 0;
+                let mut received: u64 = 0;
+                let mut inputs_sum: u64 = 0;
+                let mut outputs_sum: u64 = 0;
+
+                for (txout, (_input_index, input)) in
+                    vout.into_iter().zip(tx.input.iter().enumerate())
+                {
+                    let txout = match txout {
+                        Some(txout) => txout,
+                        None => {
+                            // skip coinbase inputs
+                            debug_assert!(
+                                input.previous_output.is_null(),
+                                "prevout should only be missing for coinbase"
+                            );
+                            continue;
+                        }
+                    };
+                    // Verify this input if requested via feature flag
+                    #[cfg(feature = "verify")]
+                    {
+                        use crate::wallet::verify::VerifyError;
+                        let serialized_tx = bitcoin::consensus::serialize(&tx);
+                        bitcoinconsensus::verify(
+                            txout.script_pubkey.to_bytes().as_ref(),
+                            txout.value,
+                            &serialized_tx,
+                            _input_index,
+                        )
+                        .map_err(VerifyError::from)?;
+                    }
+                    inputs_sum += txout.value;
+                    if self.state.db.is_mine(&txout.script_pubkey)? {
+                        sent += txout.value;
+                    }
+                }
+
+                for out in &tx.output {
+                    outputs_sum += out.value;
+                    if self.state.db.is_mine(&out.script_pubkey)? {
+                        received += out.value;
+                    }
+                }
+                // we need to saturating sub since we want coinbase txs to map to 0 fee and
+                // this subtraction will be negative for coinbase txs.
+                let fee = inputs_sum.saturating_sub(outputs_sum);
+                Result::<_, Error>::Ok(TransactionDetails {
+                    txid: *txid,
+                    transaction: Some(tx),
+                    received,
+                    sent,
+                    // we're going to fill this in later
+                    confirmation_time: None,
+                    fee: Some(fee),
+                })
+            })
+            .collect::<Result<Vec<_>, _>>()?;
+
+        for tx_detail in tx_details {
+            self.state.tx_needed.remove(&tx_detail.txid);
+            self.state
+                .tx_missing_conftime
+                .insert(tx_detail.txid, tx_detail);
+        }
+
+        if !self.state.tx_needed.is_empty() {
+            Ok(Request::Tx(self))
+        } else {
+            Ok(Request::Conftime(ConftimeReq { state: self.state }))
+        }
+    }
+}
+
+/// Final step is to get confirmation times
+pub struct ConftimeReq<'a, D> {
+    state: State<'a, D>,
+}
+
+impl<'a, D: BatchDatabase> ConftimeReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Txid> + Clone {
+        self.state.tx_missing_conftime.keys()
+    }
+
+    pub fn satisfy(
+        mut self,
+        confirmation_times: Vec<Option<BlockTime>>,
+    ) -> Result<Request<'a, D>, Error> {
+        let conftime_needed = self
+            .request()
+            .cloned()
+            .take(confirmation_times.len())
+            .collect::<Vec<_>>();
+        for (confirmation_time, txid) in confirmation_times.into_iter().zip(conftime_needed.iter())
+        {
+            debug!("confirmation time for {} was {:?}", txid, confirmation_time);
+            if let Some(mut tx_details) = self.state.tx_missing_conftime.remove(txid) {
+                tx_details.confirmation_time = confirmation_time;
+                self.state.finished_txs.push(tx_details);
+            }
+        }
+
+        if self.state.tx_missing_conftime.is_empty() {
+            Ok(Request::Finish(self.state.into_db_update()?))
+        } else {
+            Ok(Request::Conftime(self))
+        }
+    }
+}
+
+struct State<'a, D> {
+    db: &'a D,
+    last_active_index: HashMap<KeychainKind, usize>,
+    /// Transactions where we need to get the full details
+    tx_needed: BTreeSet<Txid>,
+    /// Transacitions that we know everything about
+    finished_txs: Vec<TransactionDetails>,
+    /// Transactions that discovered conftimes should be inserted into
+    tx_missing_conftime: BTreeMap<Txid, TransactionDetails>,
+    /// The start of the sync
+    start_time: Instant,
+}
+
+impl<'a, D: BatchDatabase> State<'a, D> {
+    fn new(db: &'a D) -> Self {
+        State {
+            db,
+            last_active_index: HashMap::default(),
+            finished_txs: vec![],
+            tx_needed: BTreeSet::default(),
+            tx_missing_conftime: BTreeMap::default(),
+            start_time: Instant::new(),
+        }
+    }
+    fn into_db_update(self) -> Result<D::Batch, Error> {
+        debug_assert!(self.tx_needed.is_empty() && self.tx_missing_conftime.is_empty());
+        let existing_txs = self.db.iter_txs(false)?;
+        let existing_txids: HashSet<Txid> = existing_txs.iter().map(|tx| tx.txid).collect();
+        let finished_txs = make_txs_consistent(&self.finished_txs);
+        let observed_txids: HashSet<Txid> = finished_txs.iter().map(|tx| tx.txid).collect();
+        let txids_to_delete = existing_txids.difference(&observed_txids);
+
+        // Ensure `last_active_index` does not decrement database's current state.
+        let index_updates = self
+            .last_active_index
+            .iter()
+            .map(|(keychain, sync_index)| {
+                let sync_index = *sync_index as u32;
+                let index_res = match self.db.get_last_index(*keychain) {
+                    Ok(Some(db_index)) => Ok(std::cmp::max(db_index, sync_index)),
+                    Ok(None) => Ok(sync_index),
+                    Err(err) => Err(err),
+                };
+                index_res.map(|index| (*keychain, index))
+            })
+            .collect::<Result<Vec<(KeychainKind, u32)>, _>>()?;
+
+        let mut batch = self.db.begin_batch();
+
+        // Delete old txs that no longer exist
+        for txid in txids_to_delete {
+            if let Some(raw_tx) = self.db.get_raw_tx(txid)? {
+                for i in 0..raw_tx.output.len() {
+                    // Also delete any utxos from the txs that no longer exist.
+                    let _ = batch.del_utxo(&OutPoint {
+                        txid: *txid,
+                        vout: i as u32,
+                    })?;
+                }
+            } else {
+                unreachable!("we should always have the raw tx");
+            }
+            batch.del_tx(txid, true)?;
+        }
+
+        let mut spent_utxos = HashSet::new();
+
+        // track all the spent utxos
+        for finished_tx in &finished_txs {
+            let tx = finished_tx
+                .transaction
+                .as_ref()
+                .expect("transaction will always be present here");
+            for input in &tx.input {
+                spent_utxos.insert(&input.previous_output);
+            }
+        }
+
+        // set every utxo we observed, unless it's already spent
+        // we don't do this in the loop above as we want to know all the spent outputs before
+        // adding the non-spent to the batch in case there are new tranasactions
+        // that spend form each other.
+        for finished_tx in &finished_txs {
+            let tx = finished_tx
+                .transaction
+                .as_ref()
+                .expect("transaction will always be present here");
+            for (i, output) in tx.output.iter().enumerate() {
+                if let Some((keychain, _)) =
+                    self.db.get_path_from_script_pubkey(&output.script_pubkey)?
+                {
+                    // add utxos we own from the new transactions we've seen.
+                    let outpoint = OutPoint {
+                        txid: finished_tx.txid,
+                        vout: i as u32,
+                    };
+
+                    batch.set_utxo(&LocalUtxo {
+                        outpoint,
+                        txout: output.clone(),
+                        keychain,
+                        // Is this UTXO in the spent_utxos set?
+                        is_spent: spent_utxos.get(&outpoint).is_some(),
+                    })?;
+                }
+            }
+
+            batch.set_tx(finished_tx)?;
+        }
+
+        // apply index updates
+        for (keychain, new_index) in index_updates {
+            debug!("updating index ({}, {})", keychain.as_byte(), new_index);
+            batch.set_last_index(keychain, new_index)?;
+        }
+
+        info!(
+            "finished setup, elapsed {:?}ms",
+            self.start_time.elapsed().as_millis()
+        );
+        Ok(batch)
+    }
+}
+
+/// Remove conflicting transactions -- tie breaking them by fee.
+fn make_txs_consistent(txs: &[TransactionDetails]) -> Vec<&TransactionDetails> {
+    let mut utxo_index: HashMap<OutPoint, &TransactionDetails> = HashMap::default();
+    for tx in txs {
+        for input in &tx.transaction.as_ref().unwrap().input {
+            utxo_index
+                .entry(input.previous_output)
+                .and_modify(|existing| match (tx.fee, existing.fee) {
+                    (Some(fee), Some(existing_fee)) if fee > existing_fee => *existing = tx,
+                    (Some(_), None) => *existing = tx,
+                    _ => { /* leave it the same */ }
+                })
+                .or_insert(tx);
+        }
+    }
+
+    utxo_index
+        .into_iter()
+        .map(|(_, tx)| (tx.txid, tx))
+        .collect::<HashMap<_, _>>()
+        .into_iter()
+        .map(|(_, tx)| tx)
+        .collect()
+}

src/blockchain/utils.rs

@@ -1,397 +0,0 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
-//
-// Copyright (c) 2020 Magical Bitcoin
-//
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
-
-use std::collections::{HashMap, HashSet};
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
-use rand::seq::SliceRandom;
-use rand::thread_rng;
-
-use bitcoin::{BlockHeader, OutPoint, Script, Transaction, Txid};
-
-use super::*;
-use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
-use crate::error::Error;
-use crate::types::{KeychainKind, TransactionDetails, UTXO};
-use crate::wallet::time::Instant;
-use crate::wallet::utils::ChunksIterator;
-
-#[derive(Debug)]
-pub struct ELSGetHistoryRes {
-    pub height: i32,
-    pub tx_hash: Txid,
-}
-
-/// Implements the synchronization logic for an Electrum-like client.
-#[maybe_async]
-pub trait ElectrumLikeSync {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script> + Clone>(
-        &self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ELSGetHistoryRes>>, Error>;
-
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid> + Clone>(
-        &self,
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error>;
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32> + Clone>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error>;
-
-    // Provided methods down here...
-
-    fn electrum_like_setup<D: BatchDatabase, P: Progress>(
-        &self,
-        stop_gap: Option<usize>,
-        db: &mut D,
-        _progress_update: P,
-    ) -> Result<(), Error> {
-        // TODO: progress
-        let start = Instant::new();
-        debug!("start setup");
-
-        let stop_gap = stop_gap.unwrap_or(20);
-        let chunk_size = stop_gap;
-
-        let mut history_txs_id = HashSet::new();
-        let mut txid_height = HashMap::new();
-        let mut max_indexes = HashMap::new();
-
-        let mut wallet_chains = vec![KeychainKind::Internal, KeychainKind::External];
-        // shuffling improve privacy, the server doesn't know my first request is from my internal or external addresses
-        wallet_chains.shuffle(&mut thread_rng());
-        // download history of our internal and external script_pubkeys
-        for keychain in wallet_chains.iter() {
-            let script_iter = db.iter_script_pubkeys(Some(*keychain))?.into_iter();
-
-            for (i, chunk) in ChunksIterator::new(script_iter, stop_gap).enumerate() {
-                // TODO if i == last, should create another chunk of addresses in db
-                let call_result: Vec<Vec<ELSGetHistoryRes>> =
-                    maybe_await!(self.els_batch_script_get_history(chunk.iter()))?;
-                let max_index = call_result
-                    .iter()
-                    .enumerate()
-                    .filter_map(|(i, v)| v.first().map(|_| i as u32))
-                    .max();
-                if let Some(max) = max_index {
-                    max_indexes.insert(keychain, max + (i * chunk_size) as u32);
-                }
-                let flattened: Vec<ELSGetHistoryRes> = call_result.into_iter().flatten().collect();
-                debug!("#{} of {:?} results:{}", i, keychain, flattened.len());
-                if flattened.is_empty() {
-                    // Didn't find anything in the last `stop_gap` script_pubkeys, breaking
-                    break;
-                }
-
-                for el in flattened {
-                    // el.height = -1 means unconfirmed with unconfirmed parents
-                    // el.height =  0 means unconfirmed with confirmed parents
-                    // but we treat those tx the same
-                    if el.height <= 0 {
-                        txid_height.insert(el.tx_hash, None);
-                    } else {
-                        txid_height.insert(el.tx_hash, Some(el.height as u32));
-                    }
-                    history_txs_id.insert(el.tx_hash);
-                }
-            }
-        }
-
-        // saving max indexes
-        info!("max indexes are: {:?}", max_indexes);
-        for keychain in wallet_chains.iter() {
-            if let Some(index) = max_indexes.get(keychain) {
-                db.set_last_index(*keychain, *index)?;
-            }
-        }
-
-        // get db status
-        let txs_details_in_db: HashMap<Txid, TransactionDetails> = db
-            .iter_txs(false)?
-            .into_iter()
-            .map(|tx| (tx.txid, tx))
-            .collect();
-        let txs_raw_in_db: HashMap<Txid, Transaction> = db
-            .iter_raw_txs()?
-            .into_iter()
-            .map(|tx| (tx.txid(), tx))
-            .collect();
-        let utxos_deps = utxos_deps(db, &txs_raw_in_db)?;
-
-        // download new txs and headers
-        let new_txs = maybe_await!(self.download_and_save_needed_raw_txs(
-            &history_txs_id,
-            &txs_raw_in_db,
-            chunk_size,
-            db
-        ))?;
-        let new_timestamps = maybe_await!(self.download_needed_headers(
-            &txid_height,
-            &txs_details_in_db,
-            chunk_size
-        ))?;
-
-        let mut batch = db.begin_batch();
-
-        // save any tx details not in db but in history_txs_id or with different height/timestamp
-        for txid in history_txs_id.iter() {
-            let height = txid_height.get(txid).cloned().flatten();
-            let timestamp = *new_timestamps.get(txid).unwrap_or(&0u64);
-            if let Some(tx_details) = txs_details_in_db.get(txid) {
-                // check if height matches, otherwise updates it
-                if tx_details.height != height {
-                    let mut new_tx_details = tx_details.clone();
-                    new_tx_details.height = height;
-                    new_tx_details.timestamp = timestamp;
-                    batch.set_tx(&new_tx_details)?;
-                }
-            } else {
-                save_transaction_details_and_utxos(
-                    &txid,
-                    db,
-                    timestamp,
-                    height,
-                    &mut batch,
-                    &utxos_deps,
-                )?;
-            }
-        }
-
-        // remove any tx details in db but not in history_txs_id
-        for txid in txs_details_in_db.keys() {
-            if !history_txs_id.contains(txid) {
-                batch.del_tx(&txid, false)?;
-            }
-        }
-
-        // remove any spent utxo
-        for new_tx in new_txs.iter() {
-            for input in new_tx.input.iter() {
-                batch.del_utxo(&input.previous_output)?;
-            }
-        }
-
-        db.commit_batch(batch)?;
-        info!("finish setup, elapsed {:?}ms", start.elapsed().as_millis());
-
-        Ok(())
-    }
-
-    /// download txs identified by `history_txs_id` and theirs previous outputs if not already present in db
-    fn download_and_save_needed_raw_txs<D: BatchDatabase>(
-        &self,
-        history_txs_id: &HashSet<Txid>,
-        txs_raw_in_db: &HashMap<Txid, Transaction>,
-        chunk_size: usize,
-        db: &mut D,
-    ) -> Result<Vec<Transaction>, Error> {
-        let mut txs_downloaded = vec![];
-        let txids_raw_in_db: HashSet<Txid> = txs_raw_in_db.keys().cloned().collect();
-        let txids_to_download: Vec<&Txid> = history_txs_id.difference(&txids_raw_in_db).collect();
-        if !txids_to_download.is_empty() {
-            info!("got {} txs to download", txids_to_download.len());
-            txs_downloaded.extend(maybe_await!(self.download_and_save_in_chunks(
-                txids_to_download,
-                chunk_size,
-                db,
-            ))?);
-            let mut prev_txids = HashSet::new();
-            let mut txids_downloaded = HashSet::new();
-            for tx in txs_downloaded.iter() {
-                txids_downloaded.insert(tx.txid());
-                // add every previous input tx, but skip coinbase
-                for input in tx.input.iter().filter(|i| !i.previous_output.is_null()) {
-                    prev_txids.insert(input.previous_output.txid);
-                }
-            }
-            let already_present: HashSet<Txid> =
-                txids_downloaded.union(&txids_raw_in_db).cloned().collect();
-            let prev_txs_to_download: Vec<&Txid> =
-                prev_txids.difference(&already_present).collect();
-            info!("{} previous txs to download", prev_txs_to_download.len());
-            txs_downloaded.extend(maybe_await!(self.download_and_save_in_chunks(
-                prev_txs_to_download,
-                chunk_size,
-                db,
-            ))?);
-        }
-
-        Ok(txs_downloaded)
-    }
-
-    /// download headers at heights in `txid_height` if tx details not already present, returns a map Txid -> timestamp
-    fn download_needed_headers(
-        &self,
-        txid_height: &HashMap<Txid, Option<u32>>,
-        txs_details_in_db: &HashMap<Txid, TransactionDetails>,
-        chunk_size: usize,
-    ) -> Result<HashMap<Txid, u64>, Error> {
-        let mut txid_timestamp = HashMap::new();
-        let needed_txid_height: HashMap<&Txid, u32> = txid_height
-            .iter()
-            .filter(|(t, _)| txs_details_in_db.get(*t).is_none())
-            .filter_map(|(t, o)| o.map(|h| (t, h)))
-            .collect();
-        let needed_heights: HashSet<u32> = needed_txid_height.values().cloned().collect();
-        if !needed_heights.is_empty() {
-            info!("{} headers to download for timestamp", needed_heights.len());
-            let mut height_timestamp: HashMap<u32, u64> = HashMap::new();
-            for chunk in ChunksIterator::new(needed_heights.into_iter(), chunk_size) {
-                let call_result: Vec<BlockHeader> =
-                    maybe_await!(self.els_batch_block_header(chunk.clone()))?;
-                height_timestamp.extend(
-                    chunk
-                        .into_iter()
-                        .zip(call_result.iter().map(|h| h.time as u64)),
-                );
-            }
-            for (txid, height) in needed_txid_height {
-                let timestamp = height_timestamp
-                    .get(&height)
-                    .ok_or_else(|| Error::Generic("timestamp missing".to_string()))?;
-                txid_timestamp.insert(*txid, *timestamp);
-            }
-        }
-
-        Ok(txid_timestamp)
-    }
-
-    fn download_and_save_in_chunks<D: BatchDatabase>(
-        &self,
-        to_download: Vec<&Txid>,
-        chunk_size: usize,
-        db: &mut D,
-    ) -> Result<Vec<Transaction>, Error> {
-        let mut txs_downloaded = vec![];
-        for chunk in ChunksIterator::new(to_download.into_iter(), chunk_size) {
-            let call_result: Vec<Transaction> =
-                maybe_await!(self.els_batch_transaction_get(chunk))?;
-            let mut batch = db.begin_batch();
-            for new_tx in call_result.iter() {
-                batch.set_raw_tx(new_tx)?;
-            }
-            db.commit_batch(batch)?;
-            txs_downloaded.extend(call_result);
-        }
-
-        Ok(txs_downloaded)
-    }
-}
-
-fn save_transaction_details_and_utxos<D: BatchDatabase>(
-    txid: &Txid,
-    db: &mut D,
-    timestamp: u64,
-    height: Option<u32>,
-    updates: &mut dyn BatchOperations,
-    utxo_deps: &HashMap<OutPoint, OutPoint>,
-) -> Result<(), Error> {
-    let tx = db.get_raw_tx(txid)?.ok_or(Error::TransactionNotFound)?;
-
-    let mut incoming: u64 = 0;
-    let mut outgoing: u64 = 0;
-
-    let mut inputs_sum: u64 = 0;
-    let mut outputs_sum: u64 = 0;
-
-    // look for our own inputs
-    for input in tx.input.iter() {
-        // skip coinbase inputs
-        if input.previous_output.is_null() {
-            continue;
-        }
-
-        // We already downloaded all previous output txs in the previous step
-        if let Some(previous_output) = db.get_previous_output(&input.previous_output)? {
-            inputs_sum += previous_output.value;
-
-            if db.is_mine(&previous_output.script_pubkey)? {
-                outgoing += previous_output.value;
-            }
-        } else {
-            // The input is not ours, but we still need to count it for the fees
-            let tx = db
-                .get_raw_tx(&input.previous_output.txid)?
-                .ok_or(Error::TransactionNotFound)?;
-            inputs_sum += tx.output[input.previous_output.vout as usize].value;
-        }
-
-        // removes conflicting UTXO if any (generated from same inputs, like for example RBF)
-        if let Some(outpoint) = utxo_deps.get(&input.previous_output) {
-            updates.del_utxo(&outpoint)?;
-        }
-    }
-
-    for (i, output) in tx.output.iter().enumerate() {
-        // to compute the fees later
-        outputs_sum += output.value;
-
-        // this output is ours, we have a path to derive it
-        if let Some((keychain, _child)) = db.get_path_from_script_pubkey(&output.script_pubkey)? {
-            debug!("{} output #{} is mine, adding utxo", txid, i);
-            updates.set_utxo(&UTXO {
-                outpoint: OutPoint::new(tx.txid(), i as u32),
-                txout: output.clone(),
-                keychain,
-            })?;
-
-            incoming += output.value;
-        }
-    }
-
-    let tx_details = TransactionDetails {
-        txid: tx.txid(),
-        transaction: Some(tx),
-        received: incoming,
-        sent: outgoing,
-        height,
-        timestamp,
-        fees: inputs_sum.saturating_sub(outputs_sum), /* if the tx is a coinbase, fees would be negative */
-    };
-    updates.set_tx(&tx_details)?;
-
-    Ok(())
-}
-
-/// returns utxo dependency as the inputs needed for the utxo to exist
-/// `tx_raw_in_db` must contains utxo's generating txs or errors witt [crate::Error::TransactionNotFound]
-fn utxos_deps<D: BatchDatabase>(
-    db: &mut D,
-    tx_raw_in_db: &HashMap<Txid, Transaction>,
-) -> Result<HashMap<OutPoint, OutPoint>, Error> {
-    let utxos = db.iter_utxos()?;
-    let mut utxos_deps = HashMap::new();
-    for utxo in utxos {
-        let from_tx = tx_raw_in_db
-            .get(&utxo.outpoint.txid)
-            .ok_or(Error::TransactionNotFound)?;
-        for input in from_tx.input.iter() {
-            utxos_deps.insert(input.previous_output, utxo.outpoint);
-        }
-    }
-    Ok(utxos_deps)
-}

src/database/any.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Runtime-checked database types
 //!
@@ -36,12 +23,12 @@
 //! # use bdk::database::{AnyDatabase, MemoryDatabase};
 //! # use bdk::{Wallet};
 //! let memory = MemoryDatabase::default();
-//! let wallet_memory = Wallet::new_offline("...", None, Network::Testnet, memory)?;
+//! let wallet_memory = Wallet::new("...", None, Network::Testnet, memory)?;
 //!
 //! # #[cfg(feature = "key-value-db")]
 //! # {
 //! let sled = sled::open("my-database")?.open_tree("default_tree")?;
-//! let wallet_sled = Wallet::new_offline("...", None, Network::Testnet, sled)?;
+//! let wallet_sled = Wallet::new("...", None, Network::Testnet, sled)?;
 //! # }
 //! # Ok::<(), bdk::Error>(())
 //! ```
@@ -55,7 +42,7 @@
 //! # use bdk::{Wallet};
 //! let config = serde_json::from_str("...")?;
 //! let database = AnyDatabase::from_config(&config)?;
-//! let wallet = Wallet::new_offline("...", None, Network::Testnet, database)?;
+//! let wallet = Wallet::new("...", None, Network::Testnet, database)?;
 //! # Ok::<(), bdk::Error>(())
 //! ```
 
@@ -74,10 +61,13 @@ macro_rules! impl_from {
 
 macro_rules! impl_inner_method {
     ( $enum_name:ident, $self:expr, $name:ident $(, $args:expr)* ) => {
+        #[allow(deprecated)]
         match $self {
             $enum_name::Memory(inner) => inner.$name( $($args, )* ),
             #[cfg(feature = "key-value-db")]
             $enum_name::Sled(inner) => inner.$name( $($args, )* ),
+            #[cfg(feature = "sqlite")]
+            $enum_name::Sqlite(inner) => inner.$name( $($args, )* ),
         }
     }
 }
@@ -95,10 +85,15 @@ pub enum AnyDatabase {
     #[cfg_attr(docsrs, doc(cfg(feature = "key-value-db")))]
     /// Simple key-value embedded database based on [`sled`]
     Sled(sled::Tree),
+    #[cfg(feature = "sqlite")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "sqlite")))]
+    /// Sqlite embedded database using [`rusqlite`]
+    Sqlite(sqlite::SqliteDatabase),
 }
 
 impl_from!(memory::MemoryDatabase, AnyDatabase, Memory,);
 impl_from!(sled::Tree, AnyDatabase, Sled, #[cfg(feature = "key-value-db")]);
+impl_from!(sqlite::SqliteDatabase, AnyDatabase, Sqlite, #[cfg(feature = "sqlite")]);
 
 /// Type that contains any of the [`BatchDatabase::Batch`] types defined by the library
 pub enum AnyBatch {
@@ -108,6 +103,10 @@ pub enum AnyBatch {
     #[cfg_attr(docsrs, doc(cfg(feature = "key-value-db")))]
     /// Simple key-value embedded database based on [`sled`]
     Sled(<sled::Tree as BatchDatabase>::Batch),
+    #[cfg(feature = "sqlite")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "sqlite")))]
+    /// Sqlite embedded database using [`rusqlite`]
+    Sqlite(<sqlite::SqliteDatabase as BatchDatabase>::Batch),
 }
 
 impl_from!(
@@ -116,6 +115,7 @@ impl_from!(
     Memory,
 );
 impl_from!(<sled::Tree as BatchDatabase>::Batch, AnyBatch, Sled, #[cfg(feature = "key-value-db")]);
+impl_from!(<sqlite::SqliteDatabase as BatchDatabase>::Batch, AnyBatch, Sqlite, #[cfg(feature = "sqlite")]);
 
 impl BatchOperations for AnyDatabase {
     fn set_script_pubkey(
@@ -133,7 +133,7 @@ impl BatchOperations for AnyDatabase {
             child
         )
     }
-    fn set_utxo(&mut self, utxo: &UTXO) -> Result<(), Error> {
+    fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error> {
         impl_inner_method!(AnyDatabase, self, set_utxo, utxo)
     }
     fn set_raw_tx(&mut self, transaction: &Transaction) -> Result<(), Error> {
@@ -145,6 +145,9 @@ impl BatchOperations for AnyDatabase {
     fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error> {
         impl_inner_method!(AnyDatabase, self, set_last_index, keychain, value)
     }
+    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error> {
+        impl_inner_method!(AnyDatabase, self, set_sync_time, sync_time)
+    }
 
     fn del_script_pubkey_from_path(
         &mut self,
@@ -165,7 +168,7 @@ impl BatchOperations for AnyDatabase {
     ) -> Result<Option<(KeychainKind, u32)>, Error> {
         impl_inner_method!(AnyDatabase, self, del_path_from_script_pubkey, script)
     }
-    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
+    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
         impl_inner_method!(AnyDatabase, self, del_utxo, outpoint)
     }
     fn del_raw_tx(&mut self, txid: &Txid) -> Result<Option<Transaction>, Error> {
@@ -181,6 +184,9 @@ impl BatchOperations for AnyDatabase {
     fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         impl_inner_method!(AnyDatabase, self, del_last_index, keychain)
     }
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+        impl_inner_method!(AnyDatabase, self, del_sync_time)
+    }
 }
 
 impl Database for AnyDatabase {
@@ -201,7 +207,7 @@ impl Database for AnyDatabase {
     fn iter_script_pubkeys(&self, keychain: Option<KeychainKind>) -> Result<Vec<Script>, Error> {
         impl_inner_method!(AnyDatabase, self, iter_script_pubkeys, keychain)
     }
-    fn iter_utxos(&self) -> Result<Vec<UTXO>, Error> {
+    fn iter_utxos(&self) -> Result<Vec<LocalUtxo>, Error> {
         impl_inner_method!(AnyDatabase, self, iter_utxos)
     }
     fn iter_raw_txs(&self) -> Result<Vec<Transaction>, Error> {
@@ -230,7 +236,7 @@ impl Database for AnyDatabase {
     ) -> Result<Option<(KeychainKind, u32)>, Error> {
         impl_inner_method!(AnyDatabase, self, get_path_from_script_pubkey, script)
     }
-    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
+    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
         impl_inner_method!(AnyDatabase, self, get_utxo, outpoint)
     }
     fn get_raw_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
@@ -242,6 +248,9 @@ impl Database for AnyDatabase {
     fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         impl_inner_method!(AnyDatabase, self, get_last_index, keychain)
     }
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        impl_inner_method!(AnyDatabase, self, get_sync_time)
+    }
 
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
         impl_inner_method!(AnyDatabase, self, increment_last_index, keychain)
@@ -257,7 +266,7 @@ impl BatchOperations for AnyBatch {
     ) -> Result<(), Error> {
         impl_inner_method!(AnyBatch, self, set_script_pubkey, script, keychain, child)
     }
-    fn set_utxo(&mut self, utxo: &UTXO) -> Result<(), Error> {
+    fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error> {
         impl_inner_method!(AnyBatch, self, set_utxo, utxo)
     }
     fn set_raw_tx(&mut self, transaction: &Transaction) -> Result<(), Error> {
@@ -269,6 +278,9 @@ impl BatchOperations for AnyBatch {
     fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error> {
         impl_inner_method!(AnyBatch, self, set_last_index, keychain, value)
     }
+    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error> {
+        impl_inner_method!(AnyBatch, self, set_sync_time, sync_time)
+    }
 
     fn del_script_pubkey_from_path(
         &mut self,
@@ -283,7 +295,7 @@ impl BatchOperations for AnyBatch {
     ) -> Result<Option<(KeychainKind, u32)>, Error> {
         impl_inner_method!(AnyBatch, self, del_path_from_script_pubkey, script)
     }
-    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
+    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
         impl_inner_method!(AnyBatch, self, del_utxo, outpoint)
     }
     fn del_raw_tx(&mut self, txid: &Txid) -> Result<Option<Transaction>, Error> {
@@ -299,6 +311,9 @@ impl BatchOperations for AnyBatch {
     fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         impl_inner_method!(AnyBatch, self, del_last_index, keychain)
     }
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+        impl_inner_method!(AnyBatch, self, del_sync_time)
+    }
 }
 
 impl BatchDatabase for AnyDatabase {
@@ -309,27 +324,27 @@ impl BatchDatabase for AnyDatabase {
             AnyDatabase::Memory(inner) => inner.begin_batch().into(),
             #[cfg(feature = "key-value-db")]
             AnyDatabase::Sled(inner) => inner.begin_batch().into(),
+            #[cfg(feature = "sqlite")]
+            AnyDatabase::Sqlite(inner) => inner.begin_batch().into(),
         }
     }
     fn commit_batch(&mut self, batch: Self::Batch) -> Result<(), Error> {
-        // TODO: refactor once `move_ref_pattern` is stable
-        #[allow(irrefutable_let_patterns)]
         match self {
-            AnyDatabase::Memory(db) => {
-                if let AnyBatch::Memory(batch) = batch {
-                    db.commit_batch(batch)
-                } else {
-                    unimplemented!()
-                }
-            }
+            AnyDatabase::Memory(db) => match batch {
+                AnyBatch::Memory(batch) => db.commit_batch(batch),
+                #[cfg(any(feature = "key-value-db", feature = "sqlite"))]
+                _ => unimplemented!("Other batch shouldn't be used with Memory db."),
+            },
             #[cfg(feature = "key-value-db")]
-            AnyDatabase::Sled(db) => {
-                if let AnyBatch::Sled(batch) = batch {
-                    db.commit_batch(batch)
-                } else {
-                    unimplemented!()
-                }
-            }
+            AnyDatabase::Sled(db) => match batch {
+                AnyBatch::Sled(batch) => db.commit_batch(batch),
+                _ => unimplemented!("Other batch shouldn't be used with Sled db."),
+            },
+            #[cfg(feature = "sqlite")]
+            AnyDatabase::Sqlite(db) => match batch {
+                AnyBatch::Sqlite(batch) => db.commit_batch(batch),
+                _ => unimplemented!("Other batch shouldn't be used with Sqlite db."),
+            },
         }
     }
 }
@@ -353,6 +368,23 @@ impl ConfigurableDatabase for sled::Tree {
     }
 }
 
+/// Configuration type for a [`sqlite::SqliteDatabase`] database
+#[cfg(feature = "sqlite")]
+#[derive(Debug, serde::Serialize, serde::Deserialize)]
+pub struct SqliteDbConfiguration {
+    /// Main directory of the db
+    pub path: String,
+}
+
+#[cfg(feature = "sqlite")]
+impl ConfigurableDatabase for sqlite::SqliteDatabase {
+    type Config = SqliteDbConfiguration;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        Ok(sqlite::SqliteDatabase::new(config.path.clone()))
+    }
+}
+
 /// Type that can contain any of the database configurations defined by the library
 ///
 /// This allows storing a single configuration that can be loaded into an [`AnyDatabase`]
@@ -366,6 +398,10 @@ pub enum AnyDatabaseConfig {
     #[cfg_attr(docsrs, doc(cfg(feature = "key-value-db")))]
     /// Simple key-value embedded database based on [`sled`]
     Sled(SledDbConfiguration),
+    #[cfg(feature = "sqlite")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "sqlite")))]
+    /// Sqlite embedded database using [`rusqlite`]
+    Sqlite(SqliteDbConfiguration),
 }
 
 impl ConfigurableDatabase for AnyDatabase {
@@ -378,9 +414,14 @@ impl ConfigurableDatabase for AnyDatabase {
             }
             #[cfg(feature = "key-value-db")]
             AnyDatabaseConfig::Sled(inner) => AnyDatabase::Sled(sled::Tree::from_config(inner)?),
+            #[cfg(feature = "sqlite")]
+            AnyDatabaseConfig::Sqlite(inner) => {
+                AnyDatabase::Sqlite(sqlite::SqliteDatabase::from_config(inner)?)
+            }
         })
     }
 }
 
 impl_from!((), AnyDatabaseConfig, Memory,);
 impl_from!(SledDbConfiguration, AnyDatabaseConfig, Sled, #[cfg(feature = "key-value-db")]);
+impl_from!(SqliteDbConfiguration, AnyDatabaseConfig, Sqlite, #[cfg(feature = "sqlite")]);

src/database/keyvalue.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 use std::convert::TryInto;
 
@@ -31,7 +18,7 @@ use bitcoin::hash_types::Txid;
 use bitcoin::{OutPoint, Script, Transaction};
 
 use crate::database::memory::MapKey;
-use crate::database::{BatchDatabase, BatchOperations, Database};
+use crate::database::{BatchDatabase, BatchOperations, Database, SyncTime};
 use crate::error::Error;
 use crate::types::*;
 
@@ -51,11 +38,12 @@ macro_rules! impl_batch_operations {
             Ok(())
         }
 
-        fn set_utxo(&mut self, utxo: &UTXO) -> Result<(), Error> {
-            let key = MapKey::UTXO(Some(&utxo.outpoint)).as_map_key();
+        fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error> {
+            let key = MapKey::Utxo(Some(&utxo.outpoint)).as_map_key();
             let value = json!({
                 "t": utxo.txout,
                 "i": utxo.keychain,
+                "s": utxo.is_spent,
             });
             self.insert(key, serde_json::to_vec(&value)?)$($after_insert)*;
 
@@ -95,6 +83,13 @@ macro_rules! impl_batch_operations {
             Ok(())
         }
 
+        fn set_sync_time(&mut self, data: SyncTime) -> Result<(), Error> {
+            let key = MapKey::SyncTime.as_map_key();
+            self.insert(key, serde_json::to_vec(&data)?)$($after_insert)*;
+
+            Ok(())
+        }
+
         fn del_script_pubkey_from_path(&mut self, keychain: KeychainKind, path: u32) -> Result<Option<Script>, Error> {
             let key = MapKey::Path((Some(keychain), Some(path))).as_map_key();
             let res = self.remove(key);
@@ -120,8 +115,8 @@ macro_rules! impl_batch_operations {
             }
         }
 
-        fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
-            let key = MapKey::UTXO(Some(outpoint)).as_map_key();
+        fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+            let key = MapKey::Utxo(Some(outpoint)).as_map_key();
             let res = self.remove(key);
             let res = $process_delete!(res);
 
@@ -131,8 +126,9 @@ macro_rules! impl_batch_operations {
                     let mut val: serde_json::Value = serde_json::from_slice(&b)?;
                     let txout = serde_json::from_value(val["t"].take())?;
                     let keychain = serde_json::from_value(val["i"].take())?;
+                    let is_spent = val.get_mut("s").and_then(|s| s.take().as_bool()).unwrap_or(false);
 
-                    Ok(Some(UTXO { outpoint: outpoint.clone(), txout, keychain }))
+                    Ok(Some(LocalUtxo { outpoint: outpoint.clone(), txout, keychain, is_spent, }))
                 }
             }
         }
@@ -170,16 +166,17 @@ macro_rules! impl_batch_operations {
         fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
             let key = MapKey::LastIndex(keychain).as_map_key();
             let res = self.remove(key);
+            $process_delete!(res)
+            .map(ivec_to_u32)
+            .transpose()
+        }
+
+        fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+            let key = MapKey::SyncTime.as_map_key();
+            let res = self.remove(key);
             let res = $process_delete!(res);
 
-            match res {
-                None => Ok(None),
-                Some(b) => {
-                    let array: [u8; 4] = b.as_ref().try_into().map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
-                    let val = u32::from_be_bytes(array);
-                    Ok(Some(val))
-                }
-            }
+            Ok(res.map(|b| serde_json::from_slice(&b)).transpose()?)
         }
     }
 }
@@ -234,8 +231,8 @@ impl Database for Tree {
             .collect()
     }
 
-    fn iter_utxos(&self) -> Result<Vec<UTXO>, Error> {
-        let key = MapKey::UTXO(None).as_map_key();
+    fn iter_utxos(&self) -> Result<Vec<LocalUtxo>, Error> {
+        let key = MapKey::Utxo(None).as_map_key();
         self.scan_prefix(key)
             .map(|x| -> Result<_, Error> {
                 let (k, v) = x?;
@@ -244,11 +241,16 @@ impl Database for Tree {
                 let mut val: serde_json::Value = serde_json::from_slice(&v)?;
                 let txout = serde_json::from_value(val["t"].take())?;
                 let keychain = serde_json::from_value(val["i"].take())?;
+                let is_spent = val
+                    .get_mut("s")
+                    .and_then(|s| s.take().as_bool())
+                    .unwrap_or(false);
 
-                Ok(UTXO {
+                Ok(LocalUtxo {
                     outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 })
             })
             .collect()
@@ -305,18 +307,23 @@ impl Database for Tree {
             .transpose()
     }
 
-    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
-        let key = MapKey::UTXO(Some(outpoint)).as_map_key();
+    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+        let key = MapKey::Utxo(Some(outpoint)).as_map_key();
         self.get(key)?
             .map(|b| -> Result<_, Error> {
                 let mut val: serde_json::Value = serde_json::from_slice(&b)?;
                 let txout = serde_json::from_value(val["t"].take())?;
                 let keychain = serde_json::from_value(val["i"].take())?;
+                let is_spent = val
+                    .get_mut("s")
+                    .and_then(|s| s.take().as_bool())
+                    .unwrap_or(false);
 
-                Ok(UTXO {
+                Ok(LocalUtxo {
                     outpoint: *outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 })
             })
             .transpose()
@@ -333,7 +340,7 @@ impl Database for Tree {
             .map(|b| -> Result<_, Error> {
                 let mut txdetails: TransactionDetails = serde_json::from_slice(&b)?;
                 if include_raw {
-                    txdetails.transaction = self.get_raw_tx(&txid)?;
+                    txdetails.transaction = self.get_raw_tx(txid)?;
                 }
 
                 Ok(txdetails)
@@ -343,16 +350,15 @@ impl Database for Tree {
 
     fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         let key = MapKey::LastIndex(keychain).as_map_key();
-        self.get(key)?
-            .map(|b| -> Result<_, Error> {
-                let array: [u8; 4] = b
-                    .as_ref()
-                    .try_into()
-                    .map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
-                let val = u32::from_be_bytes(array);
-                Ok(val)
-            })
-            .transpose()
+        self.get(key)?.map(ivec_to_u32).transpose()
+    }
+
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        Ok(self
+            .get(key)?
+            .map(|b| serde_json::from_slice(&b))
+            .transpose()?)
     }
 
     // inserts 0 if not present
@@ -371,17 +377,19 @@ impl Database for Tree {
 
             Some(new.to_be_bytes().to_vec())
         })?
-        .map_or(Ok(0), |b| -> Result<_, Error> {
-            let array: [u8; 4] = b
-                .as_ref()
-                .try_into()
-                .map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
-            let val = u32::from_be_bytes(array);
-            Ok(val)
-        })
+        .map_or(Ok(0), ivec_to_u32)
     }
 }
 
+fn ivec_to_u32(b: sled::IVec) -> Result<u32, Error> {
+    let array: [u8; 4] = b
+        .as_ref()
+        .try_into()
+        .map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
+    let val = u32::from_be_bytes(array);
+    Ok(val)
+}
+
 impl BatchDatabase for Tree {
     type Batch = sled::Batch;
 
@@ -396,6 +404,7 @@ impl BatchDatabase for Tree {
 
 #[cfg(test)]
 mod test {
+    use lazy_static::lazy_static;
     use std::sync::{Arc, Condvar, Mutex, Once};
     use std::time::{SystemTime, UNIX_EPOCH};
 
@@ -478,4 +487,9 @@ mod test {
     fn test_last_index() {
         crate::database::test::test_last_index(get_tree());
     }
+
+    #[test]
+    fn test_sync_time() {
+        crate::database::test::test_sync_time(get_tree());
+    }
 }

src/database/memory.rs

@@ -1,32 +1,20 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! In-memory ephemeral database
 //!
 //! This module defines an in-memory database type called [`MemoryDatabase`] that is based on a
 //! [`BTreeMap`].
 
+use std::any::Any;
 use std::collections::BTreeMap;
 use std::ops::Bound::{Excluded, Included};
 
@@ -34,7 +22,7 @@ use bitcoin::consensus::encode::{deserialize, serialize};
 use bitcoin::hash_types::Txid;
 use bitcoin::{OutPoint, Script, Transaction};
 
-use crate::database::{BatchDatabase, BatchOperations, ConfigurableDatabase, Database};
+use crate::database::{BatchDatabase, BatchOperations, ConfigurableDatabase, Database, SyncTime};
 use crate::error::Error;
 use crate::types::*;
 
@@ -45,14 +33,16 @@ use crate::types::*;
 // transactions         t<txid> -> tx details
 // deriv indexes        c{i,e} -> u32
 // descriptor checksum  d{i,e} -> vec<u8>
+// last sync time       l -> { height, timestamp }
 
 pub(crate) enum MapKey<'a> {
     Path((Option<KeychainKind>, Option<u32>)),
     Script(Option<&'a Script>),
-    UTXO(Option<&'a OutPoint>),
+    Utxo(Option<&'a OutPoint>),
     RawTx(Option<&'a Txid>),
     Transaction(Option<&'a Txid>),
     LastIndex(KeychainKind),
+    SyncTime,
     DescriptorChecksum(KeychainKind),
 }
 
@@ -67,10 +57,11 @@ impl MapKey<'_> {
                 v
             }
             MapKey::Script(_) => b"s".to_vec(),
-            MapKey::UTXO(_) => b"u".to_vec(),
+            MapKey::Utxo(_) => b"u".to_vec(),
             MapKey::RawTx(_) => b"r".to_vec(),
             MapKey::Transaction(_) => b"t".to_vec(),
             MapKey::LastIndex(st) => [b"c", st.as_ref()].concat(),
+            MapKey::SyncTime => b"l".to_vec(),
             MapKey::DescriptorChecksum(st) => [b"d", st.as_ref()].concat(),
         }
     }
@@ -79,7 +70,7 @@ impl MapKey<'_> {
         match self {
             MapKey::Path((_, Some(child))) => child.to_be_bytes().to_vec(),
             MapKey::Script(Some(s)) => serialize(*s),
-            MapKey::UTXO(Some(s)) => serialize(*s),
+            MapKey::Utxo(Some(s)) => serialize(*s),
             MapKey::RawTx(Some(s)) => serialize(*s),
             MapKey::Transaction(Some(s)) => serialize(*s),
             _ => vec![],
@@ -118,12 +109,12 @@ fn after(key: &[u8]) -> Vec<u8> {
 /// Once it's dropped its content will be lost.
 ///
 /// If you are looking for a permanent storage solution, you can try with the default key-value
-/// database called [`sled`]. See the [`database`] module documentation for more defailts.
+/// database called [`sled`]. See the [`database`] module documentation for more details.
 ///
 /// [`database`]: crate::database
 #[derive(Debug, Default)]
 pub struct MemoryDatabase {
-    map: BTreeMap<Vec<u8>, Box<dyn std::any::Any>>,
+    map: BTreeMap<Vec<u8>, Box<dyn Any + Send + Sync>>,
     deleted_keys: Vec<Vec<u8>>,
 }
 
@@ -157,10 +148,12 @@ impl BatchOperations for MemoryDatabase {
         Ok(())
     }
 
-    fn set_utxo(&mut self, utxo: &UTXO) -> Result<(), Error> {
-        let key = MapKey::UTXO(Some(&utxo.outpoint)).as_map_key();
-        self.map
-            .insert(key, Box::new((utxo.txout.clone(), utxo.keychain)));
+    fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error> {
+        let key = MapKey::Utxo(Some(&utxo.outpoint)).as_map_key();
+        self.map.insert(
+            key,
+            Box::new((utxo.txout.clone(), utxo.keychain, utxo.is_spent)),
+        );
 
         Ok(())
     }
@@ -192,6 +185,12 @@ impl BatchOperations for MemoryDatabase {
 
         Ok(())
     }
+    fn set_sync_time(&mut self, data: SyncTime) -> Result<(), Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        self.map.insert(key, Box::new(data));
+
+        Ok(())
+    }
 
     fn del_script_pubkey_from_path(
         &mut self,
@@ -223,19 +222,20 @@ impl BatchOperations for MemoryDatabase {
             }
         }
     }
-    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
-        let key = MapKey::UTXO(Some(outpoint)).as_map_key();
+    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+        let key = MapKey::Utxo(Some(outpoint)).as_map_key();
         let res = self.map.remove(&key);
         self.deleted_keys.push(key);
 
         match res {
             None => Ok(None),
             Some(b) => {
-                let (txout, keychain) = b.downcast_ref().cloned().unwrap();
-                Ok(Some(UTXO {
+                let (txout, keychain, is_spent) = b.downcast_ref().cloned().unwrap();
+                Ok(Some(LocalUtxo {
                     outpoint: *outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 }))
             }
         }
@@ -282,6 +282,13 @@ impl BatchOperations for MemoryDatabase {
             Some(b) => Ok(Some(*b.downcast_ref().unwrap())),
         }
     }
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        let res = self.map.remove(&key);
+        self.deleted_keys.push(key);
+
+        Ok(res.map(|b| b.downcast_ref().cloned().unwrap()))
+    }
 }
 
 impl Database for MemoryDatabase {
@@ -316,17 +323,18 @@ impl Database for MemoryDatabase {
             .collect()
     }
 
-    fn iter_utxos(&self) -> Result<Vec<UTXO>, Error> {
-        let key = MapKey::UTXO(None).as_map_key();
+    fn iter_utxos(&self) -> Result<Vec<LocalUtxo>, Error> {
+        let key = MapKey::Utxo(None).as_map_key();
         self.map
             .range::<Vec<u8>, _>((Included(&key), Excluded(&after(&key))))
             .map(|(k, v)| {
                 let outpoint = deserialize(&k[1..]).unwrap();
-                let (txout, keychain) = v.downcast_ref().cloned().unwrap();
-                Ok(UTXO {
+                let (txout, keychain, is_spent) = v.downcast_ref().cloned().unwrap();
+                Ok(LocalUtxo {
                     outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 })
             })
             .collect()
@@ -382,14 +390,15 @@ impl Database for MemoryDatabase {
         }))
     }
 
-    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
-        let key = MapKey::UTXO(Some(outpoint)).as_map_key();
+    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+        let key = MapKey::Utxo(Some(outpoint)).as_map_key();
         Ok(self.map.get(&key).map(|b| {
-            let (txout, keychain) = b.downcast_ref().cloned().unwrap();
-            UTXO {
+            let (txout, keychain, is_spent) = b.downcast_ref().cloned().unwrap();
+            LocalUtxo {
                 outpoint: *outpoint,
                 txout,
                 keychain,
+                is_spent,
             }
         }))
     }
@@ -407,7 +416,7 @@ impl Database for MemoryDatabase {
         Ok(self.map.get(&key).map(|b| {
             let mut txdetails: TransactionDetails = b.downcast_ref().cloned().unwrap();
             if include_raw {
-                txdetails.transaction = self.get_raw_tx(&txid).unwrap();
+                txdetails.transaction = self.get_raw_tx(txid).unwrap();
             }
 
             txdetails
@@ -419,6 +428,14 @@ impl Database for MemoryDatabase {
         Ok(self.map.get(&key).map(|b| *b.downcast_ref().unwrap()))
     }
 
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        Ok(self
+            .map
+            .get(&key)
+            .map(|b| b.downcast_ref().cloned().unwrap()))
+    }
+
     // inserts 0 if not present
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
         let key = MapKey::LastIndex(keychain).as_map_key();
@@ -442,8 +459,8 @@ impl BatchDatabase for MemoryDatabase {
     }
 
     fn commit_batch(&mut self, mut batch: Self::Batch) -> Result<(), Error> {
-        for key in batch.deleted_keys {
-            self.map.remove(&key);
+        for key in batch.deleted_keys.iter() {
+            self.map.remove(key);
         }
         self.map.append(&mut batch.map);
         Ok(())
@@ -465,20 +482,29 @@ impl ConfigurableDatabase for MemoryDatabase {
 /// don't have `test` set.
 macro_rules! populate_test_db {
     ($db:expr, $tx_meta:expr, $current_height:expr$(,)?) => {{
+        $crate::populate_test_db!($db, $tx_meta, $current_height, (@coinbase false))
+    }};
+    ($db:expr, $tx_meta:expr, $current_height:expr, (@coinbase $is_coinbase:expr)$(,)?) => {{
+        use std::str::FromStr;
         use $crate::database::BatchOperations;
         let mut db = $db;
         let tx_meta = $tx_meta;
         let current_height: Option<u32> = $current_height;
-        let tx = Transaction {
+        let input = if $is_coinbase {
+            vec![$crate::bitcoin::TxIn::default()]
+        } else {
+            vec![]
+        };
+        let tx = $crate::bitcoin::Transaction {
             version: 1,
             lock_time: 0,
-            input: vec![],
+            input,
             output: tx_meta
                 .output
                 .iter()
-                .map(|out_meta| bitcoin::TxOut {
+                .map(|out_meta| $crate::bitcoin::TxOut {
                     value: out_meta.value,
-                    script_pubkey: bitcoin::Address::from_str(&out_meta.to_address)
+                    script_pubkey: $crate::bitcoin::Address::from_str(&out_meta.to_address)
                         .unwrap()
                         .script_pubkey(),
                 })
@@ -486,29 +512,33 @@ macro_rules! populate_test_db {
         };
 
         let txid = tx.txid();
-        let height = tx_meta
+        let confirmation_time = tx_meta
             .min_confirmations
-            .map(|conf| current_height.unwrap().checked_sub(conf as u32).unwrap());
+            .and_then(|v| if v == 0 { None } else { Some(v) })
+            .map(|conf| $crate::BlockTime {
+                height: current_height.unwrap().checked_sub(conf as u32).unwrap() + 1,
+                timestamp: 0,
+            });
 
-        let tx_details = TransactionDetails {
+        let tx_details = $crate::TransactionDetails {
             transaction: Some(tx.clone()),
             txid,
-            timestamp: 0,
-            height,
+            fee: Some(0),
             received: 0,
             sent: 0,
-            fees: 0,
+            confirmation_time,
         };
 
         db.set_tx(&tx_details).unwrap();
         for (vout, out) in tx.output.iter().enumerate() {
-            db.set_utxo(&UTXO {
+            db.set_utxo(&$crate::LocalUtxo {
                 txout: out.clone(),
-                outpoint: OutPoint {
+                outpoint: $crate::bitcoin::OutPoint {
                     txid,
                     vout: vout as u32,
                 },
-                keychain: KeychainKind::External,
+                keychain: $crate::KeychainKind::External,
+                is_spent: false,
             })
             .unwrap();
         }
@@ -524,7 +554,7 @@ macro_rules! doctest_wallet {
     () => {{
         use $crate::bitcoin::Network;
         use $crate::database::MemoryDatabase;
-        use testutils::testutils;
+        use $crate::testutils;
         let descriptor = "wpkh(cVpPVruEDdmutPzisEsYvtST1usBR3ntr8pXSyt6D2YYqXRyPcFW)";
         let descriptors = testutils!(@descriptors (descriptor) (descriptor));
 
@@ -537,7 +567,7 @@ macro_rules! doctest_wallet {
             Some(100),
         );
 
-        $crate::Wallet::new_offline(
+        $crate::Wallet::new(
             &descriptors.0,
             descriptors.1.as_ref(),
             Network::Regtest,
@@ -594,4 +624,9 @@ mod test {
     fn test_last_index() {
         crate::database::test::test_last_index(get_tree());
     }
+
+    #[test]
+    fn test_sync_time() {
+        crate::database::test::test_sync_time(get_tree());
+    }
 }

src/database/mod.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Database types
 //!
@@ -37,6 +24,8 @@
 //!
 //! [`Wallet`]: crate::wallet::Wallet
 
+use serde::{Deserialize, Serialize};
+
 use bitcoin::hash_types::Txid;
 use bitcoin::{OutPoint, Script, Transaction, TxOut};
 
@@ -49,9 +38,23 @@ pub use any::{AnyDatabase, AnyDatabaseConfig};
 #[cfg(feature = "key-value-db")]
 pub(crate) mod keyvalue;
 
+#[cfg(feature = "sqlite")]
+pub(crate) mod sqlite;
+#[cfg(feature = "sqlite")]
+pub use sqlite::SqliteDatabase;
+
 pub mod memory;
 pub use memory::MemoryDatabase;
 
+/// Blockchain state at the time of syncing
+///
+/// Contains only the block time and height at the moment
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct SyncTime {
+    /// Block timestamp and height at the time of sync
+    pub block_time: BlockTime,
+}
+
 /// Trait for operations that can be batched
 ///
 /// This trait defines the list of operations that must be implemented on the [`Database`] type and
@@ -64,14 +67,16 @@ pub trait BatchOperations {
         keychain: KeychainKind,
         child: u32,
     ) -> Result<(), Error>;
-    /// Store a [`UTXO`]
-    fn set_utxo(&mut self, utxo: &UTXO) -> Result<(), Error>;
+    /// Store a [`LocalUtxo`]
+    fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error>;
     /// Store a raw transaction
     fn set_raw_tx(&mut self, transaction: &Transaction) -> Result<(), Error>;
     /// Store the metadata of a transaction
     fn set_tx(&mut self, transaction: &TransactionDetails) -> Result<(), Error>;
     /// Store the last derivation index for a given keychain.
     fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error>;
+    /// Store the sync time
+    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error>;
 
     /// Delete a script_pubkey given the keychain and its child number.
     fn del_script_pubkey_from_path(
@@ -85,8 +90,8 @@ pub trait BatchOperations {
         &mut self,
         script: &Script,
     ) -> Result<Option<(KeychainKind, u32)>, Error>;
-    /// Delete a [`UTXO`] given its [`OutPoint`]
-    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error>;
+    /// Delete a [`LocalUtxo`] given its [`OutPoint`]
+    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error>;
     /// Delete a raw transaction given its [`Txid`]
     fn del_raw_tx(&mut self, txid: &Txid) -> Result<Option<Transaction>, Error>;
     /// Delete the metadata of a transaction and optionally the raw transaction itself
@@ -97,6 +102,10 @@ pub trait BatchOperations {
     ) -> Result<Option<TransactionDetails>, Error>;
     /// Delete the last derivation index for a keychain.
     fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error>;
+    /// Reset the sync time to `None`
+    ///
+    /// Returns the removed value
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error>;
 }
 
 /// Trait for reading data from a database
@@ -116,8 +125,8 @@ pub trait Database: BatchOperations {
 
     /// Return the list of script_pubkeys
     fn iter_script_pubkeys(&self, keychain: Option<KeychainKind>) -> Result<Vec<Script>, Error>;
-    /// Return the list of [`UTXO`]s
-    fn iter_utxos(&self) -> Result<Vec<UTXO>, Error>;
+    /// Return the list of [`LocalUtxo`]s
+    fn iter_utxos(&self) -> Result<Vec<LocalUtxo>, Error>;
     /// Return the list of raw transactions
     fn iter_raw_txs(&self) -> Result<Vec<Transaction>, Error>;
     /// Return the list of transactions metadata
@@ -134,14 +143,16 @@ pub trait Database: BatchOperations {
         &self,
         script: &Script,
     ) -> Result<Option<(KeychainKind, u32)>, Error>;
-    /// Fetch a [`UTXO`] given its [`OutPoint`]
-    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error>;
+    /// Fetch a [`LocalUtxo`] given its [`OutPoint`]
+    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error>;
     /// Fetch a raw transaction given its [`Txid`]
     fn get_raw_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
     /// Fetch the transaction metadata and optionally also the raw transaction
     fn get_tx(&self, txid: &Txid, include_raw: bool) -> Result<Option<TransactionDetails>, Error>;
-    /// Return the last defivation index for a keychain.
+    /// Return the last derivation index for a keychain.
     fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error>;
+    /// Return the sync time, if present
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error>;
 
     /// Increment the last derivation index for a keychain and return it
     ///
@@ -177,14 +188,13 @@ pub(crate) trait DatabaseUtils: Database {
             .map(|o| o.is_some())
     }
 
-    fn get_raw_tx_or<F>(&self, txid: &Txid, f: F) -> Result<Option<Transaction>, Error>
+    fn get_raw_tx_or<D>(&self, txid: &Txid, default: D) -> Result<Option<Transaction>, Error>
     where
-        F: FnOnce() -> Result<Option<Transaction>, Error>,
+        D: FnOnce() -> Result<Option<Transaction>, Error>,
     {
         self.get_tx(txid, true)?
-            .map(|t| t.transaction)
-            .flatten()
-            .map_or_else(f, |t| Ok(Some(t)))
+            .and_then(|t| t.transaction)
+            .map_or_else(default, |t| Ok(Some(t)))
     }
 
     fn get_previous_output(&self, outpoint: &OutPoint) -> Result<Option<TxOut>, Error> {
@@ -227,7 +237,7 @@ pub mod test {
         );
         assert_eq!(
             tree.get_path_from_script_pubkey(&script).unwrap(),
-            Some((keychain, path.clone()))
+            Some((keychain, path))
         );
     }
 
@@ -256,7 +266,7 @@ pub mod test {
         );
         assert_eq!(
             tree.get_path_from_script_pubkey(&script).unwrap(),
-            Some((keychain, path.clone()))
+            Some((keychain, path))
         );
     }
 
@@ -298,14 +308,16 @@ pub mod test {
             value: 133742,
             script_pubkey: script,
         };
-        let utxo = UTXO {
+        let utxo = LocalUtxo {
             txout,
             outpoint,
             keychain: KeychainKind::External,
+            is_spent: true,
         };
 
         tree.set_utxo(&utxo).unwrap();
-
+        tree.set_utxo(&utxo).unwrap();
+        assert_eq!(tree.iter_utxos().unwrap().len(), 1);
         assert_eq!(tree.get_utxo(&outpoint).unwrap(), Some(utxo));
     }
 
@@ -327,11 +339,13 @@ pub mod test {
         let mut tx_details = TransactionDetails {
             transaction: Some(tx),
             txid,
-            timestamp: 123456,
             received: 1337,
             sent: 420420,
-            fees: 140,
-            height: Some(1000),
+            fee: Some(140),
+            confirmation_time: Some(BlockTime {
+                timestamp: 123456,
+                height: 1000,
+            }),
         };
 
         tree.set_tx(&tx_details).unwrap();
@@ -355,6 +369,34 @@ pub mod test {
         );
     }
 
+    pub fn test_list_transaction<D: Database>(mut tree: D) {
+        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
+        let tx: Transaction = deserialize(&hex_tx).unwrap();
+        let txid = tx.txid();
+        let mut tx_details = TransactionDetails {
+            transaction: Some(tx),
+            txid,
+            received: 1337,
+            sent: 420420,
+            fee: Some(140),
+            confirmation_time: Some(BlockTime {
+                timestamp: 123456,
+                height: 1000,
+            }),
+        };
+
+        tree.set_tx(&tx_details).unwrap();
+
+        // get raw tx
+        assert_eq!(tree.iter_txs(true).unwrap(), vec![tx_details.clone()]);
+
+        // now get without raw tx
+        tx_details.transaction = None;
+
+        // get not raw tx
+        assert_eq!(tree.iter_txs(false).unwrap(), vec![tx_details.clone()]);
+    }
+
     pub fn test_last_index<D: Database>(mut tree: D) {
         tree.set_last_index(KeychainKind::External, 1337).unwrap();
 
@@ -379,5 +421,25 @@ pub mod test {
         );
     }
 
+    pub fn test_sync_time<D: Database>(mut tree: D) {
+        assert!(tree.get_sync_time().unwrap().is_none());
+
+        tree.set_sync_time(SyncTime {
+            block_time: BlockTime {
+                height: 100,
+                timestamp: 1000,
+            },
+        })
+        .unwrap();
+
+        let extracted = tree.get_sync_time().unwrap();
+        assert!(extracted.is_some());
+        assert_eq!(extracted.as_ref().unwrap().block_time.height, 100);
+        assert_eq!(extracted.as_ref().unwrap().block_time.timestamp, 1000);
+
+        tree.del_sync_time().unwrap();
+        assert!(tree.get_sync_time().unwrap().is_none());
+    }
+
     // TODO: more tests...
 }

src/descriptor/checksum.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Descriptor checksum
 //!

src/descriptor/derived.rs

@@ -1,28 +1,50 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Derived descriptor keys
+//!
+//! The [`DerivedDescriptorKey`] type is a wrapper over the standard [`DescriptorPublicKey`] which
+//! guarantees that all the extended keys have a fixed derivation path, i.e. all the wildcards have
+//! been replaced by actual derivation indexes.
+//!
+//! The [`AsDerived`] trait provides a quick way to derive descriptors to obtain a
+//! `Descriptor<DerivedDescriptorKey>` type. This, in turn, can be used to derive public
+//! keys for arbitrary derivation indexes.
+//!
+//! Combining this with [`Wallet::get_signers`], secret keys can also be derived.
+//!
+//! # Example
+//!
+//! ```
+//! # use std::str::FromStr;
+//! # use bitcoin::secp256k1::Secp256k1;
+//! use bdk::descriptor::{AsDerived, DescriptorPublicKey};
+//! use bdk::miniscript::{ToPublicKey, TranslatePk, MiniscriptKey};
+//!
+//! let secp = Secp256k1::gen_new();
+//!
+//! let key = DescriptorPublicKey::from_str("[aa600a45/84'/0'/0']tpubDCbDXFKoLTQp44wQuC12JgSn5g9CWGjZdpBHeTqyypZ4VvgYjTJmK9CkyR5bFvG9f4PutvwmvpYCLkFx2rpx25hiMs4sUgxJveW8ZzSAVAc/0/*")?;
+//! let (descriptor, _, _) = bdk::descriptor!(wpkh(key))?;
+//!
+//! // derived: wpkh([aa600a45/84'/0'/0']tpubDCbDXFKoLTQp44wQuC12JgSn5g9CWGjZdpBHeTqyypZ4VvgYjTJmK9CkyR5bFvG9f4PutvwmvpYCLkFx2rpx25hiMs4sUgxJveW8ZzSAVAc/0/42)#3ladd0t2
+//! let derived = descriptor.as_derived(42, &secp);
+//! println!("derived: {}", derived);
+//!
+//! // with_pks: wpkh(02373ecb54c5e83bd7e0d40adf78b65efaf12fafb13571f0261fc90364eee22e1e)#p4jjgvll
+//! let with_pks = derived.translate_pk_infallible(|pk| pk.to_public_key(), |pkh| pkh.to_public_key().to_pubkeyhash());
+//! println!("with_pks: {}", with_pks);
+//! # Ok::<(), Box<dyn std::error::Error>>(())
+//! ```
+//!
+//! [`Wallet::get_signers`]: crate::wallet::Wallet::get_signers
 
 use std::cmp::Ordering;
 use std::fmt;
@@ -30,13 +52,10 @@ use std::hash::{Hash, Hasher};
 use std::ops::Deref;
 
 use bitcoin::hashes::hash160;
-use bitcoin::PublicKey;
+use bitcoin::{PublicKey, XOnlyPublicKey};
 
-pub use miniscript::{
-    descriptor::KeyMap, descriptor::Wildcard, Descriptor, DescriptorPublicKey, Legacy, Miniscript,
-    ScriptContext, Segwitv0,
-};
-use miniscript::{MiniscriptKey, ToPublicKey, TranslatePk};
+use miniscript::descriptor::{DescriptorSinglePub, SinglePubKey, Wildcard};
+use miniscript::{Descriptor, DescriptorPublicKey, MiniscriptKey, ToPublicKey, TranslatePk};
 
 use crate::wallet::utils::SecpCtx;
 
@@ -109,21 +128,44 @@ impl<'s> MiniscriptKey for DerivedDescriptorKey<'s> {
     fn is_uncompressed(&self) -> bool {
         self.0.is_uncompressed()
     }
-    fn serialized_len(&self) -> usize {
-        self.0.serialized_len()
-    }
 }
 
 impl<'s> ToPublicKey for DerivedDescriptorKey<'s> {
     fn to_public_key(&self) -> PublicKey {
         match &self.0 {
-            DescriptorPublicKey::SinglePub(ref spub) => spub.key.to_public_key(),
-            DescriptorPublicKey::XPub(ref xpub) => {
+            DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+                key: SinglePubKey::XOnly(_),
+                ..
+            }) => panic!("Found x-only public key in non-tr descriptor"),
+            DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+                key: SinglePubKey::FullKey(ref pk),
+                ..
+            }) => *pk,
+            DescriptorPublicKey::XPub(ref xpub) => PublicKey::new(
                 xpub.xkey
                     .derive_pub(self.1, &xpub.derivation_path)
                     .expect("Shouldn't fail, only normal derivations")
-                    .public_key
-            }
+                    .public_key,
+            ),
+        }
+    }
+
+    fn to_x_only_pubkey(&self) -> XOnlyPublicKey {
+        match &self.0 {
+            DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+                key: SinglePubKey::XOnly(ref pk),
+                ..
+            }) => *pk,
+            DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+                key: SinglePubKey::FullKey(ref pk),
+                ..
+            }) => XOnlyPublicKey::from(pk.inner),
+            DescriptorPublicKey::XPub(ref xpub) => XOnlyPublicKey::from(
+                xpub.xkey
+                    .derive_pub(self.1, &xpub.derivation_path)
+                    .expect("Shouldn't fail, only normal derivations")
+                    .public_key,
+            ),
         }
     }
 
@@ -132,14 +174,19 @@ impl<'s> ToPublicKey for DerivedDescriptorKey<'s> {
     }
 }
 
-pub(crate) trait AsDerived {
-    // Derive a descriptor and transform all of its keys to `DerivedDescriptorKey`
+/// Utilities to derive descriptors
+///
+/// Check out the [module level] documentation for more.
+///
+/// [module level]: crate::descriptor::derived
+pub trait AsDerived {
+    /// Derive a descriptor and transform all of its keys to `DerivedDescriptorKey`
     fn as_derived<'s>(&self, index: u32, secp: &'s SecpCtx)
         -> Descriptor<DerivedDescriptorKey<'s>>;
 
-    // Transform the keys into `DerivedDescriptorKey`.
-    //
-    // Panics if the descriptor is not "fixed", i.e. if it's derivable
+    /// Transform the keys into `DerivedDescriptorKey`.
+    ///
+    /// Panics if the descriptor is not "fixed", i.e. if it's derivable
     fn as_derived_fixed<'s>(&self, secp: &'s SecpCtx) -> Descriptor<DerivedDescriptorKey<'s>>;
 }
 

src/descriptor/dsl.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Descriptors DSL
 
@@ -86,6 +73,48 @@ macro_rules! impl_top_level_pk {
     }};
 }
 
+#[doc(hidden)]
+#[macro_export]
+macro_rules! impl_top_level_tr {
+    ( $internal_key:expr, $tap_tree:expr ) => {{
+        use $crate::miniscript::descriptor::{
+            Descriptor, DescriptorPublicKey, KeyMap, TapTree, Tr,
+        };
+        use $crate::miniscript::Tap;
+
+        #[allow(unused_imports)]
+        use $crate::keys::{DescriptorKey, IntoDescriptorKey, ValidNetworks};
+
+        let secp = $crate::bitcoin::secp256k1::Secp256k1::new();
+
+        $internal_key
+            .into_descriptor_key()
+            .and_then(|key: DescriptorKey<Tap>| key.extract(&secp))
+            .map_err($crate::descriptor::DescriptorError::Key)
+            .and_then(|(pk, mut key_map, mut valid_networks)| {
+                let tap_tree = $tap_tree.map(
+                    |(tap_tree, tree_keymap, tree_networks): (
+                        TapTree<DescriptorPublicKey>,
+                        KeyMap,
+                        ValidNetworks,
+                    )| {
+                        key_map.extend(tree_keymap.into_iter());
+                        valid_networks =
+                            $crate::keys::merge_networks(&valid_networks, &tree_networks);
+
+                        tap_tree
+                    },
+                );
+
+                Ok((
+                    Descriptor::<DescriptorPublicKey>::Tr(Tr::new(pk, tap_tree)?),
+                    key_map,
+                    valid_networks,
+                ))
+            })
+    }};
+}
+
 #[doc(hidden)]
 #[macro_export]
 macro_rules! impl_leaf_opcode {
@@ -97,7 +126,7 @@ macro_rules! impl_leaf_opcode {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_minsicript()?;
+            minisc.check_miniscript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -121,7 +150,7 @@ macro_rules! impl_leaf_opcode_value {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_minsicript()?;
+            minisc.check_miniscript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -145,7 +174,7 @@ macro_rules! impl_leaf_opcode_value_two {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_minsicript()?;
+            minisc.check_miniscript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -178,7 +207,7 @@ macro_rules! impl_node_opcode_two {
                     std::sync::Arc::new(b_minisc),
                 ))?;
 
-                minisc.check_minsicript()?;
+                minisc.check_miniscript()?;
 
                 Ok((minisc, a_keymap, $crate::keys::merge_networks(&a_networks, &b_networks)))
             })
@@ -188,7 +217,7 @@ macro_rules! impl_node_opcode_two {
 #[doc(hidden)]
 #[macro_export]
 macro_rules! impl_node_opcode_three {
-    ( $terminal_variant:ident, $( $inner:tt )* ) => {
+    ( $terminal_variant:ident, $( $inner:tt )* ) => ({
         use $crate::descriptor::CheckMiniscript;
 
         let inner = $crate::fragment_internal!( @t $( $inner )* );
@@ -210,11 +239,11 @@ macro_rules! impl_node_opcode_three {
                     std::sync::Arc::new(c_minisc),
                 ))?;
 
-                minisc.check_minsicript()?;
+                minisc.check_miniscript()?;
 
                 Ok((minisc, a_keymap, networks))
             })
-    };
+    });
 }
 
 #[doc(hidden)]
@@ -228,10 +257,11 @@ macro_rules! impl_sortedmulti {
         use $crate::keys::IntoDescriptorKey;
         let secp = $crate::bitcoin::secp256k1::Secp256k1::new();
 
-        let mut keys = vec![];
-        $(
-            keys.push($key.into_descriptor_key());
-        )*
+        let keys = vec![
+            $(
+                $key.into_descriptor_key(),
+            )*
+        ];
 
         keys.into_iter().collect::<Result<Vec<_>, _>>()
             .map_err($crate::descriptor::DescriptorError::Key)
@@ -240,6 +270,62 @@ macro_rules! impl_sortedmulti {
 
 }
 
+#[doc(hidden)]
+#[macro_export]
+macro_rules! parse_tap_tree {
+    ( @merge $tree_a:expr, $tree_b:expr) => {{
+        use std::sync::Arc;
+        use $crate::miniscript::descriptor::TapTree;
+
+        $tree_a
+            .and_then(|tree_a| Ok((tree_a, $tree_b?)))
+            .and_then(|((a_tree, mut a_keymap, a_networks), (b_tree, b_keymap, b_networks))| {
+                a_keymap.extend(b_keymap.into_iter());
+                Ok((TapTree::Tree(Arc::new(a_tree), Arc::new(b_tree)), a_keymap, $crate::keys::merge_networks(&a_networks, &b_networks)))
+            })
+
+    }};
+
+    // Two sub-trees
+    ( { { $( $tree_a:tt )* }, { $( $tree_b:tt )* } } ) => {{
+        let tree_a = $crate::parse_tap_tree!( { $( $tree_a )* } );
+        let tree_b = $crate::parse_tap_tree!( { $( $tree_b )* } );
+
+        $crate::parse_tap_tree!(@merge tree_a, tree_b)
+    }};
+
+    // One leaf and a sub-tree
+    ( { $op_a:ident ( $( $minisc_a:tt )* ), { $( $tree_b:tt )* } } ) => {{
+        let tree_a = $crate::parse_tap_tree!( $op_a ( $( $minisc_a )* ) );
+        let tree_b = $crate::parse_tap_tree!( { $( $tree_b )* } );
+
+        $crate::parse_tap_tree!(@merge tree_a, tree_b)
+    }};
+    ( { { $( $tree_a:tt )* }, $op_b:ident ( $( $minisc_b:tt )* ) } ) => {{
+        let tree_a = $crate::parse_tap_tree!( { $( $tree_a )* } );
+        let tree_b = $crate::parse_tap_tree!( $op_b ( $( $minisc_b )* ) );
+
+        $crate::parse_tap_tree!(@merge tree_a, tree_b)
+    }};
+
+    // Two leaves
+    ( { $op_a:ident ( $( $minisc_a:tt )* ), $op_b:ident ( $( $minisc_b:tt )* ) } ) => {{
+        let tree_a = $crate::parse_tap_tree!( $op_a ( $( $minisc_a )* ) );
+        let tree_b = $crate::parse_tap_tree!( $op_b ( $( $minisc_b )* ) );
+
+        $crate::parse_tap_tree!(@merge tree_a, tree_b)
+    }};
+
+    // Single leaf
+    ( $op:ident ( $( $minisc:tt )* ) ) => {{
+        use std::sync::Arc;
+        use $crate::miniscript::descriptor::TapTree;
+
+        $crate::fragment!( $op ( $( $minisc )* ) )
+            .map(|(a_minisc, a_keymap, a_networks)| (TapTree::Leaf(Arc::new(a_minisc)), a_keymap, a_networks))
+    }};
+}
+
 #[doc(hidden)]
 #[macro_export]
 macro_rules! apply_modifier {
@@ -255,7 +341,7 @@ macro_rules! apply_modifier {
                     ),
                 )?;
 
-                minisc.check_minsicript()?;
+                minisc.check_miniscript()?;
 
                 Ok((minisc, keymap, networks))
             })
@@ -348,7 +434,7 @@ macro_rules! apply_modifier {
 /// syntax is more suitable for a fixed number of items known at compile time, while the other accepts a
 /// [`Vec`] of items, which makes it more suitable for writing dynamic descriptors.
 ///
-/// They both produce the descriptor: `wsh(thresh(2,pk(...),s:pk(...),sdv:older(...)))`
+/// They both produce the descriptor: `wsh(thresh(2,pk(...),s:pk(...),sndv:older(...)))`
 ///
 /// ```
 /// # use std::str::FromStr;
@@ -361,7 +447,7 @@ macro_rules! apply_modifier {
 ///
 /// let (descriptor_a, key_map_a, networks) = bdk::descriptor! {
 ///     wsh (
-///         thresh(2, pk(my_key_1), s:pk(my_key_2), s:d:v:older(my_timelock))
+///         thresh(2, pk(my_key_1), s:pk(my_key_2), s:n:d:v:older(my_timelock))
 ///     )
 /// }?;
 ///
@@ -369,7 +455,7 @@ macro_rules! apply_modifier {
 /// let b_items = vec![
 ///     bdk::fragment!(pk(my_key_1))?,
 ///     bdk::fragment!(s:pk(my_key_2))?,
-///     bdk::fragment!(s:d:v:older(my_timelock))?,
+///     bdk::fragment!(s:n:d:v:older(my_timelock))?,
 /// ];
 /// let (descriptor_b, mut key_map_b, networks) = bdk::descriptor!(wsh(thresh_vec(2, b_items)))?;
 ///
@@ -453,6 +539,15 @@ macro_rules! descriptor {
     ( wsh ( $( $minisc:tt )* ) ) => ({
         $crate::impl_top_level_sh!(Wsh, new, new_sortedmulti, Segwitv0, $( $minisc )*)
     });
+
+    ( tr ( $internal_key:expr ) ) => ({
+        $crate::impl_top_level_tr!($internal_key, None)
+    });
+    ( tr ( $internal_key:expr, $( $taptree:tt )* ) ) => ({
+        let tap_tree = $crate::parse_tap_tree!( $( $taptree )* );
+        tap_tree
+            .and_then(|tap_tree| $crate::impl_top_level_tr!($internal_key, Some(tap_tree)))
+    });
 }
 
 #[doc(hidden)]
@@ -492,6 +587,23 @@ impl<A, B, C> From<(A, (B, (C, ())))> for TupleThree<A, B, C> {
     }
 }
 
+#[doc(hidden)]
+#[macro_export]
+macro_rules! group_multi_keys {
+    ( $( $key:expr ),+ ) => {{
+        use $crate::keys::IntoDescriptorKey;
+
+        let keys = vec![
+            $(
+                $key.into_descriptor_key(),
+            )*
+        ];
+
+        keys.into_iter().collect::<Result<Vec<_>, _>>()
+            .map_err($crate::descriptor::DescriptorError::Key)
+    }};
+}
+
 #[doc(hidden)]
 #[macro_export]
 macro_rules! fragment_internal {
@@ -533,7 +645,7 @@ macro_rules! fragment_internal {
     // three operands it's (X, (X, (X, ()))), etc.
     //
     // To check that the right number of arguments has been passed we can "cast" those tuples to
-    // more convenient structures like `TupleTwo`. If the conversion succedes, the right number of
+    // more convenient structures like `TupleTwo`. If the conversion succeeds, the right number of
     // args was passed. Otherwise the compilation fails entirely.
     ( @t $op:ident ( $( $args:tt )* ) $( $tail:tt )* ) => ({
         ($crate::fragment!( $op ( $( $args )* ) ), $crate::fragment_internal!( @t $( $tail )* ))
@@ -547,9 +659,7 @@ macro_rules! fragment_internal {
     ( @t , $( $tail:tt )* ) => ({
         $crate::fragment_internal!( @t $( $tail )* )
     });
-    ( @t ) => ({
-        ()
-    });
+    ( @t ) => ({});
 
     // Fallback to calling `fragment!()`
     ( $( $tokens:tt )* ) => ({
@@ -585,8 +695,9 @@ macro_rules! fragment {
     ( pk ( $key:expr ) ) => ({
         $crate::fragment!(c:pk_k ( $key ))
     });
-    ( pk_h ( $key_hash:expr ) ) => ({
-        $crate::impl_leaf_opcode_value!(PkH, $key_hash)
+    ( pk_h ( $key:expr ) ) => ({
+        let secp = $crate::bitcoin::secp256k1::Secp256k1::new();
+        $crate::keys::make_pkh($key, &secp)
     });
     ( after ( $value:expr ) ) => ({
         $crate::impl_leaf_opcode_value!(After, $value)
@@ -615,6 +726,9 @@ macro_rules! fragment {
     ( and_or ( $( $inner:tt )* ) ) => ({
         $crate::impl_node_opcode_three!(AndOr, $( $inner )*)
     });
+    ( andor ( $( $inner:tt )* ) ) => ({
+        $crate::impl_node_opcode_three!(AndOr, $( $inner )*)
+    });
     ( or_b ( $( $inner:tt )* ) ) => ({
         $crate::impl_node_opcode_two!(OrB, $( $inner )*)
     });
@@ -650,20 +764,22 @@ macro_rules! fragment {
             .and_then(|items| $crate::fragment!(thresh_vec($thresh, items)))
     });
     ( multi_vec ( $thresh:expr, $keys:expr ) ) => ({
-        $crate::keys::make_multi($thresh, $keys)
-    });
-    ( multi ( $thresh:expr $(, $key:expr )+ ) ) => ({
-        use $crate::keys::IntoDescriptorKey;
         let secp = $crate::bitcoin::secp256k1::Secp256k1::new();
 
-        let mut keys = vec![];
-        $(
-            keys.push($key.into_descriptor_key());
-        )*
+        $crate::keys::make_multi($thresh, $crate::miniscript::Terminal::Multi, $keys, &secp)
+    });
+    ( multi ( $thresh:expr $(, $key:expr )+ ) ) => ({
+        $crate::group_multi_keys!( $( $key ),* )
+            .and_then(|keys| $crate::fragment!( multi_vec ( $thresh, keys ) ))
+    });
+    ( multi_a_vec ( $thresh:expr, $keys:expr ) ) => ({
+        let secp = $crate::bitcoin::secp256k1::Secp256k1::new();
 
-        keys.into_iter().collect::<Result<Vec<_>, _>>()
-            .map_err($crate::descriptor::DescriptorError::Key)
-            .and_then(|keys| $crate::keys::make_multi($thresh, keys, &secp))
+        $crate::keys::make_multi($thresh, $crate::miniscript::Terminal::MultiA, $keys, &secp)
+    });
+    ( multi_a ( $thresh:expr $(, $key:expr )+ ) ) => ({
+        $crate::group_multi_keys!( $( $key ),* )
+            .and_then(|keys| $crate::fragment!( multi_a_vec ( $thresh, keys ) ))
     });
 
     // `sortedmulti()` is handled separately
@@ -803,6 +919,25 @@ mod test {
         );
     }
 
+    #[test]
+    fn test_fixed_threeop_descriptors() {
+        let redeem_key = bitcoin::PublicKey::from_str(
+            "03a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd",
+        )
+        .unwrap();
+        let move_key = bitcoin::PublicKey::from_str(
+            "032e58afe51f9ed8ad3cc7897f634d881fdbe49a81564629ded8156bebd2ffd1af",
+        )
+        .unwrap();
+
+        check(
+            descriptor!(sh(wsh(and_or(pk(redeem_key), older(1000), pk(move_key))))),
+            true,
+            true,
+            &["2MypGwr5eQWAWWJtiJgUEToVxc4zuokjQRe"],
+        );
+    }
+
     #[test]
     fn test_bip32_legacy_descriptors() {
         let xprv = bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
@@ -968,7 +1103,7 @@ mod test {
     fn test_valid_networks() {
         let xprv = bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
         let path = bip32::DerivationPath::from_str("m/0").unwrap();
-        let desc_key = (xprv, path.clone()).into_descriptor_key().unwrap();
+        let desc_key = (xprv, path).into_descriptor_key().unwrap();
 
         let (_desc, _key_map, valid_networks) = descriptor!(pkh(desc_key)).unwrap();
         assert_eq!(
@@ -978,7 +1113,7 @@ mod test {
 
         let xprv = bip32::ExtendedPrivKey::from_str("xprv9s21ZrQH143K3QTDL4LXw2F7HEK3wJUD2nW2nRk4stbPy6cq3jPPqjiChkVvvNKmPGJxWUtg6LnF5kejMRNNU3TGtRBeJgk33yuGBxrMPHi").unwrap();
         let path = bip32::DerivationPath::from_str("m/10/20/30/40").unwrap();
-        let desc_key = (xprv, path.clone()).into_descriptor_key().unwrap();
+        let desc_key = (xprv, path).into_descriptor_key().unwrap();
 
         let (_desc, _key_map, valid_networks) = descriptor!(wpkh(desc_key)).unwrap();
         assert_eq!(valid_networks, [Bitcoin].iter().cloned().collect());
@@ -1005,12 +1140,9 @@ mod test {
             descriptor!(sh(wsh(multi(2, desc_key1, desc_key2, desc_key3)))).unwrap();
         assert_eq!(key_map.len(), 3);
 
-        let desc_key1: DescriptorKey<Segwitv0> =
-            (xprv1, path1.clone()).into_descriptor_key().unwrap();
-        let desc_key2: DescriptorKey<Segwitv0> =
-            (xprv2, path2.clone()).into_descriptor_key().unwrap();
-        let desc_key3: DescriptorKey<Segwitv0> =
-            (xprv3, path3.clone()).into_descriptor_key().unwrap();
+        let desc_key1: DescriptorKey<Segwitv0> = (xprv1, path1).into_descriptor_key().unwrap();
+        let desc_key2: DescriptorKey<Segwitv0> = (xprv2, path2).into_descriptor_key().unwrap();
+        let desc_key3: DescriptorKey<Segwitv0> = (xprv3, path3).into_descriptor_key().unwrap();
 
         let (key1, _key_map, _valid_networks) = desc_key1.extract(&secp).unwrap();
         let (key2, _key_map, _valid_networks) = desc_key2.extract(&secp).unwrap();
@@ -1026,7 +1158,7 @@ mod test {
         // this compiles
         let xprv = bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
         let path = bip32::DerivationPath::from_str("m/0").unwrap();
-        let desc_key: DescriptorKey<Legacy> = (xprv, path.clone()).into_descriptor_key().unwrap();
+        let desc_key: DescriptorKey<Legacy> = (xprv, path).into_descriptor_key().unwrap();
 
         let (desc, _key_map, _valid_networks) = descriptor!(pkh(desc_key)).unwrap();
         assert_eq!(desc.to_string(), "pkh(tpubD6NzVbkrYhZ4WR7a4vY1VT3khMJMeAxVsfq9TBJyJWrNk247zCJtV7AWf6UJP7rAVsn8NNKdJi3gFyKPTmWZS9iukb91xbn2HbFSMQm2igY/0/*)#yrnz9pp2");
@@ -1041,13 +1173,15 @@ mod test {
         let private_key =
             PrivateKey::from_wif("cSQPHDBwXGjVzWRqAHm6zfvQhaTuj1f2bFH58h55ghbjtFwvmeXR").unwrap();
         let (descriptor, _, _) =
-            descriptor!(wsh(thresh(2,d:v:older(1),s:pk(private_key),s:pk(private_key)))).unwrap();
+            descriptor!(wsh(thresh(2,n:d:v:older(1),s:pk(private_key),s:pk(private_key)))).unwrap();
 
-        assert_eq!(descriptor.to_string(), "wsh(thresh(2,dv:older(1),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c)))#cfdcqs3s")
+        assert_eq!(descriptor.to_string(), "wsh(thresh(2,ndv:older(1),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c)))#zzk3ux8g")
     }
 
     #[test]
-    #[should_panic(expected = "Miniscript(ContextError(CompressedOnly))")]
+    #[should_panic(
+        expected = "Miniscript(ContextError(CompressedOnly(\"04b4632d08485ff1df2db55b9dafd23347d1c47a457072a1e87be26896549a87378ec38ff91d43e8c2092ebda601780485263da089465619e0358a5c1be7ac91f4\")))"
+    )]
     fn test_dsl_miniscript_checks() {
         let mut uncompressed_pk =
             PrivateKey::from_wif("L5EZftvrYaSudiozVRzTqLcHLNDoVn7H5HSfM9BAN6tMJX8oTWz6").unwrap();
@@ -1055,4 +1189,35 @@ mod test {
 
         descriptor!(wsh(v: pk(uncompressed_pk))).unwrap();
     }
+
+    #[test]
+    fn test_dsl_tr_only_key() {
+        let private_key =
+            PrivateKey::from_wif("cSQPHDBwXGjVzWRqAHm6zfvQhaTuj1f2bFH58h55ghbjtFwvmeXR").unwrap();
+        let (descriptor, _, _) = descriptor!(tr(private_key)).unwrap();
+
+        assert_eq!(
+            descriptor.to_string(),
+            "tr(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c)#heq9m95v"
+        )
+    }
+
+    #[test]
+    fn test_dsl_tr_simple_tree() {
+        let private_key =
+            PrivateKey::from_wif("cSQPHDBwXGjVzWRqAHm6zfvQhaTuj1f2bFH58h55ghbjtFwvmeXR").unwrap();
+        let (descriptor, _, _) =
+            descriptor!(tr(private_key, { pk(private_key), pk(private_key) })).unwrap();
+
+        assert_eq!(descriptor.to_string(), "tr(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c,{pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c),pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c)})#xy5fjw6d")
+    }
+
+    #[test]
+    fn test_dsl_tr_single_leaf() {
+        let private_key =
+            PrivateKey::from_wif("cSQPHDBwXGjVzWRqAHm6zfvQhaTuj1f2bFH58h55ghbjtFwvmeXR").unwrap();
+        let (descriptor, _, _) = descriptor!(tr(private_key, pk(private_key))).unwrap();
+
+        assert_eq!(descriptor.to_string(), "tr(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c,pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c))#lzl2vmc7")
+    }
 }

src/descriptor/error.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Descriptor errors
 
@@ -28,11 +15,13 @@
 #[derive(Debug)]
 pub enum Error {
     /// Invalid HD Key path, such as having a wildcard but a length != 1
-    InvalidHDKeyPath,
+    InvalidHdKeyPath,
     /// The provided descriptor doesn't match its checksum
     InvalidDescriptorChecksum,
     /// The descriptor contains hardened derivation steps on public extended keys
     HardenedDerivationXpub,
+    /// The descriptor contains multiple keys with the same BIP32 fingerprint
+    DuplicatedKeys,
 
     /// Error thrown while working with [`keys`](crate::keys)
     Key(crate::keys::KeyError),
@@ -43,11 +32,11 @@ pub enum Error {
     InvalidDescriptorCharacter(char),
 
     /// BIP32 error
-    BIP32(bitcoin::util::bip32::Error),
+    Bip32(bitcoin::util::bip32::Error),
     /// Error during base58 decoding
     Base58(bitcoin::util::base58::Error),
     /// Key-related error
-    PK(bitcoin::util::key::Error),
+    Pk(bitcoin::util::key::Error),
     /// Miniscript error
     Miniscript(miniscript::Error),
     /// Hex decoding error
@@ -58,7 +47,7 @@ impl From<crate::keys::KeyError> for Error {
     fn from(key_error: crate::keys::KeyError) -> Error {
         match key_error {
             crate::keys::KeyError::Miniscript(inner) => Error::Miniscript(inner),
-            crate::keys::KeyError::BIP32(inner) => Error::BIP32(inner),
+            crate::keys::KeyError::Bip32(inner) => Error::Bip32(inner),
             e => Error::Key(e),
         }
     }
@@ -72,9 +61,9 @@ impl std::fmt::Display for Error {
 
 impl std::error::Error for Error {}
 
-impl_error!(bitcoin::util::bip32::Error, BIP32);
+impl_error!(bitcoin::util::bip32::Error, Bip32);
 impl_error!(bitcoin::util::base58::Error, Base58);
-impl_error!(bitcoin::util::key::Error, PK);
+impl_error!(bitcoin::util::key::Error, Pk);
 impl_error!(miniscript::Error, Miniscript);
 impl_error!(bitcoin::hashes::hex::Error, Hex);
 impl_error!(crate::descriptor::policy::PolicyError, Policy);

src/descriptor/mod.rs

@@ -1,47 +1,38 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Descriptors
 //!
 //! This module contains generic utilities to work with descriptors, plus some re-exported types
 //! from [`miniscript`].
 
-use std::collections::{BTreeMap, HashMap};
+use std::collections::{BTreeMap, HashSet};
 use std::ops::Deref;
 
-use bitcoin::util::bip32::{
-    ChildNumber, DerivationPath, ExtendedPrivKey, ExtendedPubKey, Fingerprint, KeySource,
-};
-use bitcoin::util::psbt;
-use bitcoin::{Network, PublicKey, Script, TxOut};
+use bitcoin::util::bip32::{ChildNumber, DerivationPath, ExtendedPubKey, Fingerprint, KeySource};
+use bitcoin::util::{psbt, taproot};
+use bitcoin::{secp256k1, PublicKey, XOnlyPublicKey};
+use bitcoin::{Network, Script, TxOut};
 
-use miniscript::descriptor::{DescriptorPublicKey, DescriptorType, DescriptorXKey, Wildcard};
-pub use miniscript::{descriptor::KeyMap, Descriptor, Legacy, Miniscript, ScriptContext, Segwitv0};
+use miniscript::descriptor::{DescriptorType, InnerXKey, SinglePubKey};
+pub use miniscript::{
+    descriptor::DescriptorXKey, descriptor::KeyMap, descriptor::Wildcard, Descriptor,
+    DescriptorPublicKey, Legacy, Miniscript, ScriptContext, Segwitv0,
+};
 use miniscript::{DescriptorTrait, ForEachKey, TranslatePk};
 
+use crate::descriptor::policy::BuildSatisfaction;
+
 pub mod checksum;
-pub(crate) mod derived;
+pub mod derived;
 #[doc(hidden)]
 pub mod dsl;
 pub mod error;
@@ -49,8 +40,7 @@ pub mod policy;
 pub mod template;
 
 pub use self::checksum::get_checksum;
-use self::derived::AsDerived;
-pub use self::derived::DerivedDescriptorKey;
+pub use self::derived::{AsDerived, DerivedDescriptorKey};
 pub use self::error::Error as DescriptorError;
 pub use self::policy::Policy;
 use self::template::DescriptorTemplateOut;
@@ -69,7 +59,14 @@ pub type DerivedDescriptor<'s> = Descriptor<DerivedDescriptorKey<'s>>;
 ///
 /// [`psbt::Input`]: bitcoin::util::psbt::Input
 /// [`psbt::Output`]: bitcoin::util::psbt::Output
-pub type HDKeyPaths = BTreeMap<PublicKey, KeySource>;
+pub type HdKeyPaths = BTreeMap<secp256k1::PublicKey, KeySource>;
+
+/// Alias for the type of maps that represent taproot key origins in a [`psbt::Input`] or
+/// [`psbt::Output`]
+///
+/// [`psbt::Input`]: bitcoin::util::psbt::Input
+/// [`psbt::Output`]: bitcoin::util::psbt::Output
+pub type TapKeyOrigins = BTreeMap<bitcoin::XOnlyPublicKey, (Vec<taproot::TapLeafHash>, KeySource)>;
 
 /// Trait for types which can be converted into an [`ExtendedDescriptor`] and a [`KeyMap`] usable by a wallet in a specific [`Network`]
 pub trait IntoWalletDescriptor {
@@ -137,13 +134,13 @@ impl IntoWalletDescriptor for (ExtendedDescriptor, KeyMap) {
 
         let check_key = |pk: &DescriptorPublicKey| {
             let (pk, _, networks) = if self.0.is_witness() {
-                let desciptor_key: DescriptorKey<miniscript::Segwitv0> =
+                let descriptor_key: DescriptorKey<miniscript::Segwitv0> =
                     pk.clone().into_descriptor_key()?;
-                desciptor_key.extract(&secp)?
+                descriptor_key.extract(secp)?
             } else {
-                let desciptor_key: DescriptorKey<miniscript::Legacy> =
+                let descriptor_key: DescriptorKey<miniscript::Legacy> =
                     pk.clone().into_descriptor_key()?;
-                desciptor_key.extract(&secp)?
+                descriptor_key.extract(secp)?
             };
 
             if networks.contains(&network) {
@@ -225,19 +222,37 @@ pub(crate) fn into_wallet_descriptor_checked<T: IntoWalletDescriptor>(
         return Err(DescriptorError::HardenedDerivationXpub);
     }
 
+    // Ensure that there are no duplicated keys
+    let mut found_keys = HashSet::new();
+    let descriptor_contains_duplicated_keys = descriptor.for_any_key(|k| {
+        if let DescriptorPublicKey::XPub(xkey) = k.as_key() {
+            let fingerprint = xkey.root_fingerprint(secp);
+            if found_keys.contains(&fingerprint) {
+                return true;
+            }
+
+            found_keys.insert(fingerprint);
+        }
+
+        false
+    });
+    if descriptor_contains_duplicated_keys {
+        return Err(DescriptorError::DuplicatedKeys);
+    }
+
     Ok((descriptor, keymap))
 }
 
 #[doc(hidden)]
 /// Used internally mainly by the `descriptor!()` and `fragment!()` macros
 pub trait CheckMiniscript<Ctx: miniscript::ScriptContext> {
-    fn check_minsicript(&self) -> Result<(), miniscript::Error>;
+    fn check_miniscript(&self) -> Result<(), miniscript::Error>;
 }
 
 impl<Ctx: miniscript::ScriptContext, Pk: miniscript::MiniscriptKey> CheckMiniscript<Ctx>
     for miniscript::Miniscript<Pk, Ctx>
 {
-    fn check_minsicript(&self) -> Result<(), miniscript::Error> {
+    fn check_miniscript(&self) -> Result<(), miniscript::Error> {
         Ctx::check_global_validity(self)?;
 
         Ok(())
@@ -250,6 +265,7 @@ pub trait ExtractPolicy {
     fn extract_policy(
         &self,
         signers: &SignersContainer,
+        psbt: BuildSatisfaction,
         secp: &SecpCtx,
     ) -> Result<Option<Policy>, DescriptorError>;
 }
@@ -259,41 +275,10 @@ pub(crate) trait XKeyUtils {
     fn root_fingerprint(&self, secp: &SecpCtx) -> Fingerprint;
 }
 
-// FIXME: `InnerXKey` was made private in rust-miniscript, so we have to implement this manually on
-// both `ExtendedPubKey` and `ExtendedPrivKey`.
-//
-// Revert back to using the trait once https://github.com/rust-bitcoin/rust-miniscript/pull/230 is
-// released
-impl XKeyUtils for DescriptorXKey<ExtendedPubKey> {
-    fn full_path(&self, append: &[ChildNumber]) -> DerivationPath {
-        let full_path = match self.origin {
-            Some((_, ref path)) => path
-                .into_iter()
-                .chain(self.derivation_path.into_iter())
-                .cloned()
-                .collect(),
-            None => self.derivation_path.clone(),
-        };
-
-        if self.wildcard != Wildcard::None {
-            full_path
-                .into_iter()
-                .chain(append.iter())
-                .cloned()
-                .collect()
-        } else {
-            full_path
-        }
-    }
-
-    fn root_fingerprint(&self, _: &SecpCtx) -> Fingerprint {
-        match self.origin {
-            Some((fingerprint, _)) => fingerprint,
-            None => self.xkey.fingerprint(),
-        }
-    }
-}
-impl XKeyUtils for DescriptorXKey<ExtendedPrivKey> {
+impl<T> XKeyUtils for DescriptorXKey<T>
+where
+    T: InnerXKey,
+{
     fn full_path(&self, append: &[ChildNumber]) -> DerivationPath {
         let full_path = match self.origin {
             Some((_, ref path)) => path
@@ -318,21 +303,33 @@ impl XKeyUtils for DescriptorXKey<ExtendedPrivKey> {
     fn root_fingerprint(&self, secp: &SecpCtx) -> Fingerprint {
         match self.origin {
             Some((fingerprint, _)) => fingerprint,
-            None => self.xkey.fingerprint(secp),
+            None => self.xkey.xkey_fingerprint(secp),
         }
     }
 }
 
 pub(crate) trait DerivedDescriptorMeta {
-    fn get_hd_keypaths(&self, secp: &SecpCtx) -> Result<HDKeyPaths, DescriptorError>;
+    fn get_hd_keypaths(&self, secp: &SecpCtx) -> HdKeyPaths;
+    fn get_tap_key_origins(&self, secp: &SecpCtx) -> TapKeyOrigins;
 }
 
 pub(crate) trait DescriptorMeta {
     fn is_witness(&self) -> bool;
+    fn is_taproot(&self) -> bool;
     fn get_extended_keys(&self) -> Result<Vec<DescriptorXKey<ExtendedPubKey>>, DescriptorError>;
     fn derive_from_hd_keypaths<'s>(
         &self,
-        hd_keypaths: &HDKeyPaths,
+        hd_keypaths: &HdKeyPaths,
+        secp: &'s SecpCtx,
+    ) -> Option<DerivedDescriptor<'s>>;
+    fn derive_from_tap_key_origins<'s>(
+        &self,
+        tap_key_origins: &TapKeyOrigins,
+        secp: &'s SecpCtx,
+    ) -> Option<DerivedDescriptor<'s>>;
+    fn derive_from_psbt_key_origins<'s>(
+        &self,
+        key_origins: BTreeMap<Fingerprint, (&DerivationPath, SinglePubKey)>,
         secp: &'s SecpCtx,
     ) -> Option<DerivedDescriptor<'s>>;
     fn derive_from_psbt_input<'s>(
@@ -351,23 +348,34 @@ pub(crate) trait DescriptorScripts {
 impl<'s> DescriptorScripts for DerivedDescriptor<'s> {
     fn psbt_redeem_script(&self) -> Option<Script> {
         match self.desc_type() {
-            DescriptorType::ShWpkh => Some(self.explicit_script()),
-            DescriptorType::ShWsh => Some(self.explicit_script().to_v0_p2wsh()),
-            DescriptorType::Sh => Some(self.explicit_script()),
-            DescriptorType::Bare => Some(self.explicit_script()),
-            DescriptorType::ShSortedMulti => Some(self.explicit_script()),
-            _ => None,
+            DescriptorType::ShWpkh => Some(self.explicit_script().unwrap()),
+            DescriptorType::ShWsh => Some(self.explicit_script().unwrap().to_v0_p2wsh()),
+            DescriptorType::Sh => Some(self.explicit_script().unwrap()),
+            DescriptorType::Bare => Some(self.explicit_script().unwrap()),
+            DescriptorType::ShSortedMulti => Some(self.explicit_script().unwrap()),
+            DescriptorType::ShWshSortedMulti => Some(self.explicit_script().unwrap().to_v0_p2wsh()),
+            DescriptorType::Pkh
+            | DescriptorType::Wpkh
+            | DescriptorType::Tr
+            | DescriptorType::Wsh
+            | DescriptorType::WshSortedMulti => None,
         }
     }
 
     fn psbt_witness_script(&self) -> Option<Script> {
         match self.desc_type() {
-            DescriptorType::Wsh => Some(self.explicit_script()),
-            DescriptorType::ShWsh => Some(self.explicit_script()),
+            DescriptorType::Wsh => Some(self.explicit_script().unwrap()),
+            DescriptorType::ShWsh => Some(self.explicit_script().unwrap()),
             DescriptorType::WshSortedMulti | DescriptorType::ShWshSortedMulti => {
-                Some(self.explicit_script())
+                Some(self.explicit_script().unwrap())
             }
-            _ => None,
+            DescriptorType::Bare
+            | DescriptorType::Sh
+            | DescriptorType::Pkh
+            | DescriptorType::Wpkh
+            | DescriptorType::ShSortedMulti
+            | DescriptorType::Tr
+            | DescriptorType::ShWpkh => None,
         }
     }
 }
@@ -385,6 +393,10 @@ impl DescriptorMeta for ExtendedDescriptor {
         )
     }
 
+    fn is_taproot(&self) -> bool {
+        self.desc_type() == DescriptorType::Tr
+    }
+
     fn get_extended_keys(&self) -> Result<Vec<DescriptorXKey<ExtendedPubKey>>, DescriptorError> {
         let mut answer = Vec::new();
 
@@ -399,61 +411,124 @@ impl DescriptorMeta for ExtendedDescriptor {
         Ok(answer)
     }
 
-    fn derive_from_hd_keypaths<'s>(
+    fn derive_from_psbt_key_origins<'s>(
         &self,
-        hd_keypaths: &HDKeyPaths,
+        key_origins: BTreeMap<Fingerprint, (&DerivationPath, SinglePubKey)>,
         secp: &'s SecpCtx,
     ) -> Option<DerivedDescriptor<'s>> {
-        let index: HashMap<_, _> = hd_keypaths.values().map(|(a, b)| (a, b)).collect();
+        // Ensure that deriving `xpub` with `path` yields `expected`
+        let verify_key = |xpub: &DescriptorXKey<ExtendedPubKey>,
+                          path: &DerivationPath,
+                          expected: &SinglePubKey| {
+            let derived = xpub
+                .xkey
+                .derive_pub(secp, path)
+                .expect("The path should never contain hardened derivation steps")
+                .public_key;
+
+            match expected {
+                SinglePubKey::FullKey(pk) if &PublicKey::new(derived) == pk => true,
+                SinglePubKey::XOnly(pk) if &XOnlyPublicKey::from(derived) == pk => true,
+                _ => false,
+            }
+        };
 
         let mut path_found = None;
-        self.for_each_key(|key| {
-            if path_found.is_some() {
-                // already found a matching path, we are done
-                return true;
-            }
 
+        // using `for_any_key` should make this stop as soon as we return `true`
+        self.for_any_key(|key| {
             if let DescriptorPublicKey::XPub(xpub) = key.as_key().deref() {
-                // Check if the key matches one entry in our `index`. If it does, `matches()` will
+                // Check if the key matches one entry in our `key_origins`. If it does, `matches()` will
                 // return the "prefix" that matched, so we remove that prefix from the full path
-                // found in `index` and save it in `derive_path`. We expect this to be a derivation
+                // found in `key_origins` and save it in `derive_path`. We expect this to be a derivation
                 // path of length 1 if the key is `wildcard` and an empty path otherwise.
                 let root_fingerprint = xpub.root_fingerprint(secp);
-                let derivation_path: Option<Vec<ChildNumber>> = index
+                let derive_path = key_origins
                     .get_key_value(&root_fingerprint)
-                    .and_then(|(fingerprint, path)| {
-                        xpub.matches(&(**fingerprint, (*path).clone()), secp)
+                    .and_then(|(fingerprint, (path, expected))| {
+                        xpub.matches(&(*fingerprint, (*path).clone()), secp)
+                            .zip(Some((path, expected)))
                     })
-                    .map(|prefix| {
-                        index
-                            .get(&xpub.root_fingerprint(secp))
-                            .unwrap()
+                    .and_then(|(prefix, (full_path, expected))| {
+                        let derive_path = full_path
                             .into_iter()
                             .skip(prefix.into_iter().count())
                             .cloned()
-                            .collect()
+                            .collect::<DerivationPath>();
+
+                        // `derive_path` only contains the replacement index for the wildcard, if present, or
+                        // an empty path for fixed descriptors. To verify the key we also need the normal steps
+                        // that come before the wildcard, so we take them directly from `xpub` and then append
+                        // the final index
+                        if verify_key(
+                            xpub,
+                            &xpub.derivation_path.extend(derive_path.clone()),
+                            expected,
+                        ) {
+                            Some(derive_path)
+                        } else {
+                            log::debug!(
+                                "Key `{}` derived with {} yields an unexpected key",
+                                root_fingerprint,
+                                derive_path
+                            );
+                            None
+                        }
                     });
 
-                match derivation_path {
+                match derive_path {
                     Some(path) if xpub.wildcard != Wildcard::None && path.len() == 1 => {
                         // Ignore hardened wildcards
                         if let ChildNumber::Normal { index } = path[0] {
-                            path_found = Some(index)
+                            path_found = Some(index);
+                            return true;
                         }
                     }
                     Some(path) if xpub.wildcard == Wildcard::None && path.is_empty() => {
-                        path_found = Some(0)
+                        path_found = Some(0);
+                        return true;
                     }
                     _ => {}
                 }
             }
 
-            true
+            false
         });
 
         path_found.map(|path| self.as_derived(path, secp))
     }
 
+    fn derive_from_hd_keypaths<'s>(
+        &self,
+        hd_keypaths: &HdKeyPaths,
+        secp: &'s SecpCtx,
+    ) -> Option<DerivedDescriptor<'s>> {
+        // "Convert" an hd_keypaths map to the format required by `derive_from_psbt_key_origins`
+        let key_origins = hd_keypaths
+            .iter()
+            .map(|(pk, (fingerprint, path))| {
+                (
+                    *fingerprint,
+                    (path, SinglePubKey::FullKey(PublicKey::new(*pk))),
+                )
+            })
+            .collect();
+        self.derive_from_psbt_key_origins(key_origins, secp)
+    }
+
+    fn derive_from_tap_key_origins<'s>(
+        &self,
+        tap_key_origins: &TapKeyOrigins,
+        secp: &'s SecpCtx,
+    ) -> Option<DerivedDescriptor<'s>> {
+        // "Convert" a tap_key_origins map to the format required by `derive_from_psbt_key_origins`
+        let key_origins = tap_key_origins
+            .iter()
+            .map(|(pk, (_, (fingerprint, path)))| (*fingerprint, (path, SinglePubKey::XOnly(*pk))))
+            .collect();
+        self.derive_from_psbt_key_origins(key_origins, secp)
+    }
+
     fn derive_from_psbt_input<'s>(
         &self,
         psbt_input: &psbt::Input,
@@ -463,6 +538,9 @@ impl DescriptorMeta for ExtendedDescriptor {
         if let Some(derived) = self.derive_from_hd_keypaths(&psbt_input.bip32_derivation, secp) {
             return Some(derived);
         }
+        if let Some(derived) = self.derive_from_tap_key_origins(&psbt_input.tap_key_origins, secp) {
+            return Some(derived);
+        }
         if self.is_deriveable() {
             // We can't try to bruteforce the derivation index, exit here
             return None;
@@ -471,7 +549,10 @@ impl DescriptorMeta for ExtendedDescriptor {
         let descriptor = self.as_derived_fixed(secp);
         match descriptor.desc_type() {
             // TODO: add pk() here
-            DescriptorType::Pkh | DescriptorType::Wpkh | DescriptorType::ShWpkh
+            DescriptorType::Pkh
+            | DescriptorType::Wpkh
+            | DescriptorType::ShWpkh
+            | DescriptorType::Tr
                 if utxo.is_some()
                     && descriptor.script_pubkey() == utxo.as_ref().unwrap().script_pubkey =>
             {
@@ -479,7 +560,7 @@ impl DescriptorMeta for ExtendedDescriptor {
             }
             DescriptorType::Bare | DescriptorType::Sh | DescriptorType::ShSortedMulti
                 if psbt_input.redeem_script.is_some()
-                    && &descriptor.explicit_script()
+                    && &descriptor.explicit_script().unwrap()
                         == psbt_input.redeem_script.as_ref().unwrap() =>
             {
                 Some(descriptor)
@@ -489,7 +570,7 @@ impl DescriptorMeta for ExtendedDescriptor {
             | DescriptorType::ShWshSortedMulti
             | DescriptorType::WshSortedMulti
                 if psbt_input.witness_script.is_some()
-                    && &descriptor.explicit_script()
+                    && &descriptor.explicit_script().unwrap()
                         == psbt_input.witness_script.as_ref().unwrap() =>
             {
                 Some(descriptor)
@@ -500,7 +581,7 @@ impl DescriptorMeta for ExtendedDescriptor {
 }
 
 impl<'s> DerivedDescriptorMeta for DerivedDescriptor<'s> {
-    fn get_hd_keypaths(&self, secp: &SecpCtx) -> Result<HDKeyPaths, DescriptorError> {
+    fn get_hd_keypaths(&self, secp: &SecpCtx) -> HdKeyPaths {
         let mut answer = BTreeMap::new();
         self.for_each_key(|key| {
             if let DescriptorPublicKey::XPub(xpub) = key.as_key().deref() {
@@ -518,7 +599,64 @@ impl<'s> DerivedDescriptorMeta for DerivedDescriptor<'s> {
             true
         });
 
-        Ok(answer)
+        answer
+    }
+
+    fn get_tap_key_origins(&self, secp: &SecpCtx) -> TapKeyOrigins {
+        use miniscript::ToPublicKey;
+
+        let mut answer = BTreeMap::new();
+        let mut insert_path = |pk: &DerivedDescriptorKey<'_>, lh| {
+            let key_origin = match pk.deref() {
+                DescriptorPublicKey::XPub(xpub) => {
+                    Some((xpub.root_fingerprint(secp), xpub.full_path(&[])))
+                }
+                DescriptorPublicKey::SinglePub(_) => None,
+            };
+
+            // If this is the internal key, we only insert the key origin if it's not None.
+            // For keys found in the tap tree we always insert a key origin (because the signer
+            // looks for it to know which leaves to sign for), even though it may be None
+            match (lh, key_origin) {
+                (None, Some(ko)) => {
+                    answer
+                        .entry(pk.to_x_only_pubkey())
+                        .or_insert_with(|| (vec![], ko));
+                }
+                (Some(lh), origin) => {
+                    answer
+                        .entry(pk.to_x_only_pubkey())
+                        .or_insert_with(|| (vec![], origin.unwrap_or_default()))
+                        .0
+                        .push(lh);
+                }
+                _ => {}
+            }
+        };
+
+        if let Descriptor::Tr(tr) = &self {
+            // Internal key first, then iterate the scripts
+            insert_path(tr.internal_key(), None);
+
+            for (_, ms) in tr.iter_scripts() {
+                // Assume always the same leaf version
+                let leaf_hash = taproot::TapLeafHash::from_script(
+                    &ms.encode(),
+                    taproot::LeafVersion::TapScript,
+                );
+
+                for key in ms.iter_pk_pkh() {
+                    let key = match key {
+                        miniscript::miniscript::iter::PkPkh::PlainPubkey(pk) => pk,
+                        miniscript::miniscript::iter::PkPkh::HashedPubkey(pk) => pk,
+                    };
+
+                    insert_path(&key, Some(leaf_hash));
+                }
+            }
+        }
+
+        answer
     }
 }
 
@@ -530,9 +668,10 @@ mod test {
     use bitcoin::hashes::hex::FromHex;
     use bitcoin::secp256k1::Secp256k1;
     use bitcoin::util::{bip32, psbt};
+    use bitcoin::Script;
 
     use super::*;
-    use crate::psbt::PSBTUtils;
+    use crate::psbt::PsbtUtils;
 
     #[test]
     fn test_derive_from_psbt_input_wpkh_wif() {
@@ -659,7 +798,7 @@ mod test {
 
         // make a descriptor out of it
         let desc = crate::descriptor!(wpkh(key)).unwrap();
-        // this should conver the key that supports "any_network" to the right network (testnet)
+        // this should convert the key that supports "any_network" to the right network (testnet)
         let (wallet_desc, _) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
@@ -783,5 +922,32 @@ mod test {
             result.unwrap_err(),
             DescriptorError::HardenedDerivationXpub
         ));
+
+        let descriptor = "wsh(multi(2,tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/0/*,tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/1/*))";
+        let result = into_wallet_descriptor_checked(descriptor, &secp, Network::Testnet);
+
+        assert!(result.is_err());
+        assert!(matches!(
+            result.unwrap_err(),
+            DescriptorError::DuplicatedKeys
+        ));
+    }
+
+    #[test]
+    fn test_sh_wsh_sortedmulti_redeemscript() {
+        use super::{AsDerived, DescriptorScripts};
+
+        let secp = Secp256k1::new();
+
+        let descriptor = "sh(wsh(sortedmulti(3,tpubDEsqS36T4DVsKJd9UH8pAKzrkGBYPLEt9jZMwpKtzh1G6mgYehfHt9WCgk7MJG5QGSFWf176KaBNoXbcuFcuadAFKxDpUdMDKGBha7bY3QM/0/*,tpubDF3cpwfs7fMvXXuoQbohXtLjNM6ehwYT287LWtmLsd4r77YLg6MZg4vTETx5MSJ2zkfigbYWu31VA2Z2Vc1cZugCYXgS7FQu6pE8V6TriEH/0/*,tpubDE1SKfcW76Tb2AASv5bQWMuScYNAdoqLHoexw13sNDXwmUhQDBbCD3QAedKGLhxMrWQdMDKENzYtnXPDRvexQPNuDrLj52wAjHhNEm8sJ4p/0/*,tpubDFLc6oXwJmhm3FGGzXkfJNTh2KitoY3WhmmQvuAjMhD8YbyWn5mAqckbxXfm2etM3p5J6JoTpSrMqRSTfMLtNW46poDaEZJ1kjd3csRSjwH/0/*,tpubDEWD9NBeWP59xXmdqSNt4VYdtTGwbpyP8WS962BuqpQeMZmX9Pur14dhXdZT5a7wR1pK6dPtZ9fP5WR493hPzemnBvkfLLYxnUjAKj1JCQV/0/*,tpubDEHyZkkwd7gZWCTgQuYQ9C4myF2hMEmyHsBCCmLssGqoqUxeT3gzohF5uEVURkf9TtmeepJgkSUmteac38FwZqirjApzNX59XSHLcwaTZCH/0/*,tpubDEqLouCekwnMUWN486kxGzD44qVgeyuqHyxUypNEiQt5RnUZNJe386TKPK99fqRV1vRkZjYAjtXGTECz98MCsdLcnkM67U6KdYRzVubeCgZ/0/*)))";
+        let (descriptor, _) =
+            into_wallet_descriptor_checked(descriptor, &secp, Network::Testnet).unwrap();
+
+        let descriptor = descriptor.as_derived(0, &secp);
+
+        let script = Script::from_str("5321022f533b667e2ea3b36e21961c9fe9dca340fbe0af5210173a83ae0337ab20a57621026bb53a98e810bd0ee61a0ed1164ba6c024786d76554e793e202dc6ce9c78c4ea2102d5b8a7d66a41ffdb6f4c53d61994022e886b4f45001fb158b95c9164d45f8ca3210324b75eead2c1f9c60e8adeb5e7009fec7a29afcdb30d829d82d09562fe8bae8521032d34f8932200833487bd294aa219dcbe000b9f9b3d824799541430009f0fa55121037468f8ea99b6c64788398b5ad25480cad08f4b0d65be54ce3a55fd206b5ae4722103f72d3d96663b0ea99b0aeb0d7f273cab11a8de37885f1dddc8d9112adb87169357ae").unwrap();
+
+        assert_eq!(descriptor.psbt_redeem_script(), Some(script.to_v0_p2wsh()));
+        assert_eq!(descriptor.psbt_witness_script(), Some(script));
     }
 }

src/descriptor/template.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Descriptor templates
 //!
@@ -50,21 +37,22 @@ pub type DescriptorTemplateOut = (ExtendedDescriptor, KeyMap, ValidNetworks);
 ///
 /// ```
 /// use bdk::descriptor::error::Error as DescriptorError;
-/// use bdk::keys::{KeyError, IntoDescriptorKey};
+/// use bdk::keys::{IntoDescriptorKey, KeyError};
 /// use bdk::miniscript::Legacy;
 /// use bdk::template::{DescriptorTemplate, DescriptorTemplateOut};
+/// use bitcoin::Network;
 ///
 /// struct MyP2PKH<K: IntoDescriptorKey<Legacy>>(K);
 ///
 /// impl<K: IntoDescriptorKey<Legacy>> DescriptorTemplate for MyP2PKH<K> {
-///     fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
+///     fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
 ///         Ok(bdk::descriptor!(pkh(self.0))?)
 ///     }
 /// }
 /// ```
 pub trait DescriptorTemplate {
     /// Build the complete descriptor
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError>;
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError>;
 }
 
 /// Turns a [`DescriptorTemplate`] into a valid wallet descriptor by calling its
@@ -75,7 +63,7 @@ impl<T: DescriptorTemplate> IntoWalletDescriptor for T {
         secp: &SecpCtx,
         network: Network,
     ) -> Result<(ExtendedDescriptor, KeyMap), DescriptorError> {
-        Ok(self.build()?.into_wallet_descriptor(secp, network)?)
+        self.build(network)?.into_wallet_descriptor(secp, network)
     }
 }
 
@@ -87,28 +75,29 @@ impl<T: DescriptorTemplate> IntoWalletDescriptor for T {
 /// # use bdk::bitcoin::{PrivateKey, Network};
 /// # use bdk::{Wallet};
 /// # use bdk::database::MemoryDatabase;
-/// use bdk::template::P2PKH;
+/// # use bdk::wallet::AddressIndex::New;
+/// use bdk::template::P2Pkh;
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new_offline(
-///     P2PKH(key),
+/// let wallet = Wallet::new(
+///     P2Pkh(key),
 ///     None,
 ///     Network::Testnet,
 ///     MemoryDatabase::default(),
 /// )?;
 ///
 /// assert_eq!(
-///     wallet.get_new_address()?.to_string(),
+///     wallet.get_address(New)?.to_string(),
 ///     "mwJ8hxFYW19JLuc65RCTaP4v1rzVU8cVMT"
 /// );
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
-pub struct P2PKH<K: IntoDescriptorKey<Legacy>>(pub K);
+pub struct P2Pkh<K: IntoDescriptorKey<Legacy>>(pub K);
 
-impl<K: IntoDescriptorKey<Legacy>> DescriptorTemplate for P2PKH<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        Ok(descriptor!(pkh(self.0))?)
+impl<K: IntoDescriptorKey<Legacy>> DescriptorTemplate for P2Pkh<K> {
+    fn build(self, _network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        descriptor!(pkh(self.0))
     }
 }
 
@@ -120,29 +109,30 @@ impl<K: IntoDescriptorKey<Legacy>> DescriptorTemplate for P2PKH<K> {
 /// # use bdk::bitcoin::{PrivateKey, Network};
 /// # use bdk::{Wallet};
 /// # use bdk::database::MemoryDatabase;
-/// use bdk::template::P2WPKH_P2SH;
+/// # use bdk::wallet::AddressIndex::New;
+/// use bdk::template::P2Wpkh_P2Sh;
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new_offline(
-///     P2WPKH_P2SH(key),
+/// let wallet = Wallet::new(
+///     P2Wpkh_P2Sh(key),
 ///     None,
 ///     Network::Testnet,
 ///     MemoryDatabase::default(),
 /// )?;
 ///
 /// assert_eq!(
-///     wallet.get_new_address()?.to_string(),
+///     wallet.get_address(New)?.to_string(),
 ///     "2NB4ox5VDRw1ecUv6SnT3VQHPXveYztRqk5"
 /// );
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
 #[allow(non_camel_case_types)]
-pub struct P2WPKH_P2SH<K: IntoDescriptorKey<Segwitv0>>(pub K);
+pub struct P2Wpkh_P2Sh<K: IntoDescriptorKey<Segwitv0>>(pub K);
 
-impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2WPKH_P2SH<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        Ok(descriptor!(sh(wpkh(self.0)))?)
+impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh_P2Sh<K> {
+    fn build(self, _network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        descriptor!(sh(wpkh(self.0)))
     }
 }
 
@@ -154,36 +144,37 @@ impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2WPKH_P2SH<K> {
 /// # use bdk::bitcoin::{PrivateKey, Network};
 /// # use bdk::{Wallet};
 /// # use bdk::database::MemoryDatabase;
-/// use bdk::template::P2WPKH;
+/// # use bdk::wallet::AddressIndex::New;
+/// use bdk::template::P2Wpkh;
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new_offline(
-///     P2WPKH(key),
+/// let wallet = Wallet::new(
+///     P2Wpkh(key),
 ///     None,
 ///     Network::Testnet,
 ///     MemoryDatabase::default(),
 /// )?;
 ///
 /// assert_eq!(
-///     wallet.get_new_address()?.to_string(),
+///     wallet.get_address(New)?.to_string(),
 ///     "tb1q4525hmgw265tl3drrl8jjta7ayffu6jf68ltjd"
 /// );
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
-pub struct P2WPKH<K: IntoDescriptorKey<Segwitv0>>(pub K);
+pub struct P2Wpkh<K: IntoDescriptorKey<Segwitv0>>(pub K);
 
-impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2WPKH<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        Ok(descriptor!(wpkh(self.0))?)
+impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh<K> {
+    fn build(self, _network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        descriptor!(wpkh(self.0))
     }
 }
 
-/// BIP44 template. Expands to `pkh(key/44'/0'/0'/{0,1}/*)`
+/// BIP44 template. Expands to `pkh(key/44'/{0,1}'/0'/{0,1}/*)`
 ///
 /// Since there are hardened derivation steps, this template requires a private derivable key (generally a `xprv`/`tprv`).
 ///
-/// See [`BIP44Public`] for a template that can work with a `xpub`/`tpub`.
+/// See [`Bip44Public`] for a template that can work with a `xpub`/`tpub`.
 ///
 /// ## Example
 ///
@@ -192,35 +183,36 @@ impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2WPKH<K> {
 /// # use bdk::bitcoin::{PrivateKey, Network};
 /// # use bdk::{Wallet,  KeychainKind};
 /// # use bdk::database::MemoryDatabase;
-/// use bdk::template::BIP44;
+/// # use bdk::wallet::AddressIndex::New;
+/// use bdk::template::Bip44;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new_offline(
-///     BIP44(key.clone(), KeychainKind::External),
-///     Some(BIP44(key, KeychainKind::Internal)),
+/// let wallet = Wallet::new(
+///     Bip44(key.clone(), KeychainKind::External),
+///     Some(Bip44(key, KeychainKind::Internal)),
 ///     Network::Testnet,
 ///     MemoryDatabase::default()
 /// )?;
 ///
-/// assert_eq!(wallet.get_new_address()?.to_string(), "miNG7dJTzJqNbFS19svRdTCisC65dsubtR");
-/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "pkh([c55b303f/44'/0'/0']tpubDDDzQ31JkZB7VxUr9bjvBivDdqoFLrDPyLWtLapArAi51ftfmCb2DPxwLQzX65iNcXz1DGaVvyvo6JQ6rTU73r2gqdEo8uov9QKRb7nKCSU/0/*)#xgaaevjx");
+/// assert_eq!(wallet.get_address(New)?.to_string(), "mmogjc7HJEZkrLqyQYqJmxUqFaC7i4uf89");
+/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "pkh([c55b303f/44'/1'/0']tpubDCuorCpzvYS2LCD75BR46KHE8GdDeg1wsAgNZeNr6DaB5gQK1o14uErKwKLuFmeemkQ6N2m3rNgvctdJLyr7nwu2yia7413Hhg8WWE44cgT/0/*)#5wrnv0xt");
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
-pub struct BIP44<K: DerivableKey<Legacy>>(pub K, pub KeychainKind);
+pub struct Bip44<K: DerivableKey<Legacy>>(pub K, pub KeychainKind);
 
-impl<K: DerivableKey<Legacy>> DescriptorTemplate for BIP44<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        Ok(P2PKH(legacy::make_bipxx_private(44, self.0, self.1)?).build()?)
+impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44<K> {
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Pkh(legacy::make_bipxx_private(44, self.0, self.1, network)?).build(network)
     }
 }
 
 /// BIP44 public template. Expands to `pkh(key/{0,1}/*)`
 ///
-/// This assumes that the key used has already been derived with `m/44'/0'/0'`.
+/// This assumes that the key used has already been derived with `m/44'/0'/0'` for Mainnet or `m/44'/1'/0'` for Testnet.
 ///
 /// This template requires the parent fingerprint to populate correctly the metadata of PSBTs.
 ///
-/// See [`BIP44`] for a template that does the full derivation, but requires private data
+/// See [`Bip44`] for a template that does the full derivation, but requires private data
 /// for the key.
 ///
 /// ## Example
@@ -230,34 +222,35 @@ impl<K: DerivableKey<Legacy>> DescriptorTemplate for BIP44<K> {
 /// # use bdk::bitcoin::{PrivateKey, Network};
 /// # use bdk::{Wallet,  KeychainKind};
 /// # use bdk::database::MemoryDatabase;
-/// use bdk::template::BIP44Public;
+/// # use bdk::wallet::AddressIndex::New;
+/// use bdk::template::Bip44Public;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDDDzQ31JkZB7VxUr9bjvBivDdqoFLrDPyLWtLapArAi51ftfmCb2DPxwLQzX65iNcXz1DGaVvyvo6JQ6rTU73r2gqdEo8uov9QKRb7nKCSU")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new_offline(
-///     BIP44Public(key.clone(), fingerprint, KeychainKind::External),
-///     Some(BIP44Public(key, fingerprint, KeychainKind::Internal)),
+/// let wallet = Wallet::new(
+///     Bip44Public(key.clone(), fingerprint, KeychainKind::External),
+///     Some(Bip44Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,
 ///     MemoryDatabase::default()
 /// )?;
 ///
-/// assert_eq!(wallet.get_new_address()?.to_string(), "miNG7dJTzJqNbFS19svRdTCisC65dsubtR");
+/// assert_eq!(wallet.get_address(New)?.to_string(), "miNG7dJTzJqNbFS19svRdTCisC65dsubtR");
 /// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "pkh([c55b303f/44'/0'/0']tpubDDDzQ31JkZB7VxUr9bjvBivDdqoFLrDPyLWtLapArAi51ftfmCb2DPxwLQzX65iNcXz1DGaVvyvo6JQ6rTU73r2gqdEo8uov9QKRb7nKCSU/0/*)#xgaaevjx");
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
-pub struct BIP44Public<K: DerivableKey<Legacy>>(pub K, pub bip32::Fingerprint, pub KeychainKind);
+pub struct Bip44Public<K: DerivableKey<Legacy>>(pub K, pub bip32::Fingerprint, pub KeychainKind);
 
-impl<K: DerivableKey<Legacy>> DescriptorTemplate for BIP44Public<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        Ok(P2PKH(legacy::make_bipxx_public(44, self.0, self.1, self.2)?).build()?)
+impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44Public<K> {
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Pkh(legacy::make_bipxx_public(44, self.0, self.1, self.2)?).build(network)
     }
 }
 
-/// BIP49 template. Expands to `sh(wpkh(key/49'/0'/0'/{0,1}/*))`
+/// BIP49 template. Expands to `sh(wpkh(key/49'/{0,1}'/0'/{0,1}/*))`
 ///
 /// Since there are hardened derivation steps, this template requires a private derivable key (generally a `xprv`/`tprv`).
 ///
-/// See [`BIP49Public`] for a template that can work with a `xpub`/`tpub`.
+/// See [`Bip49Public`] for a template that can work with a `xpub`/`tpub`.
 ///
 /// ## Example
 ///
@@ -266,25 +259,26 @@ impl<K: DerivableKey<Legacy>> DescriptorTemplate for BIP44Public<K> {
 /// # use bdk::bitcoin::{PrivateKey, Network};
 /// # use bdk::{Wallet,  KeychainKind};
 /// # use bdk::database::MemoryDatabase;
-/// use bdk::template::BIP49;
+/// # use bdk::wallet::AddressIndex::New;
+/// use bdk::template::Bip49;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new_offline(
-///     BIP49(key.clone(), KeychainKind::External),
-///     Some(BIP49(key, KeychainKind::Internal)),
+/// let wallet = Wallet::new(
+///     Bip49(key.clone(), KeychainKind::External),
+///     Some(Bip49(key, KeychainKind::Internal)),
 ///     Network::Testnet,
 ///     MemoryDatabase::default()
 /// )?;
 ///
-/// assert_eq!(wallet.get_new_address()?.to_string(), "2N3K4xbVAHoiTQSwxkZjWDfKoNC27pLkYnt");
-/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "sh(wpkh([c55b303f/49\'/0\'/0\']tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L/0/*))#gsmdv4xr");
+/// assert_eq!(wallet.get_address(New)?.to_string(), "2N4zkWAoGdUv4NXhSsU8DvS5MB36T8nKHEB");
+/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "sh(wpkh([c55b303f/49'/1'/0']tpubDDYr4kdnZgjjShzYNjZUZXUUtpXaofdkMaipyS8ThEh45qFmhT4hKYways7UXmg6V7het1QiFo9kf4kYUXyDvV4rHEyvSpys9pjCB3pukxi/0/*))#s9vxlc8e");
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
-pub struct BIP49<K: DerivableKey<Segwitv0>>(pub K, pub KeychainKind);
+pub struct Bip49<K: DerivableKey<Segwitv0>>(pub K, pub KeychainKind);
 
-impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for BIP49<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        Ok(P2WPKH_P2SH(segwit_v0::make_bipxx_private(49, self.0, self.1)?).build()?)
+impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49<K> {
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Wpkh_P2Sh(segwit_v0::make_bipxx_private(49, self.0, self.1, network)?).build(network)
     }
 }
 
@@ -294,7 +288,7 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for BIP49<K> {
 ///
 /// This template requires the parent fingerprint to populate correctly the metadata of PSBTs.
 ///
-/// See [`BIP49`] for a template that does the full derivation, but requires private data
+/// See [`Bip49`] for a template that does the full derivation, but requires private data
 /// for the key.
 ///
 /// ## Example
@@ -304,34 +298,35 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for BIP49<K> {
 /// # use bdk::bitcoin::{PrivateKey, Network};
 /// # use bdk::{Wallet,  KeychainKind};
 /// # use bdk::database::MemoryDatabase;
-/// use bdk::template::BIP49Public;
+/// # use bdk::wallet::AddressIndex::New;
+/// use bdk::template::Bip49Public;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new_offline(
-///     BIP49Public(key.clone(), fingerprint, KeychainKind::External),
-///     Some(BIP49Public(key, fingerprint, KeychainKind::Internal)),
+/// let wallet = Wallet::new(
+///     Bip49Public(key.clone(), fingerprint, KeychainKind::External),
+///     Some(Bip49Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,
 ///     MemoryDatabase::default()
 /// )?;
 ///
-/// assert_eq!(wallet.get_new_address()?.to_string(), "2N3K4xbVAHoiTQSwxkZjWDfKoNC27pLkYnt");
-/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "sh(wpkh([c55b303f/49\'/0\'/0\']tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L/0/*))#gsmdv4xr");
+/// assert_eq!(wallet.get_address(New)?.to_string(), "2N3K4xbVAHoiTQSwxkZjWDfKoNC27pLkYnt");
+/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "sh(wpkh([c55b303f/49'/0'/0']tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L/0/*))#gsmdv4xr");
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
-pub struct BIP49Public<K: DerivableKey<Segwitv0>>(pub K, pub bip32::Fingerprint, pub KeychainKind);
+pub struct Bip49Public<K: DerivableKey<Segwitv0>>(pub K, pub bip32::Fingerprint, pub KeychainKind);
 
-impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for BIP49Public<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        Ok(P2WPKH_P2SH(segwit_v0::make_bipxx_public(49, self.0, self.1, self.2)?).build()?)
+impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49Public<K> {
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Wpkh_P2Sh(segwit_v0::make_bipxx_public(49, self.0, self.1, self.2)?).build(network)
     }
 }
 
-/// BIP84 template. Expands to `wpkh(key/84'/0'/0'/{0,1}/*)`
+/// BIP84 template. Expands to `wpkh(key/84'/{0,1}'/0'/{0,1}/*)`
 ///
 /// Since there are hardened derivation steps, this template requires a private derivable key (generally a `xprv`/`tprv`).
 ///
-/// See [`BIP84Public`] for a template that can work with a `xpub`/`tpub`.
+/// See [`Bip84Public`] for a template that can work with a `xpub`/`tpub`.
 ///
 /// ## Example
 ///
@@ -340,25 +335,26 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for BIP49Public<K> {
 /// # use bdk::bitcoin::{PrivateKey, Network};
 /// # use bdk::{Wallet,  KeychainKind};
 /// # use bdk::database::MemoryDatabase;
-/// use bdk::template::BIP84;
+/// # use bdk::wallet::AddressIndex::New;
+/// use bdk::template::Bip84;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new_offline(
-///     BIP84(key.clone(), KeychainKind::External),
-///     Some(BIP84(key, KeychainKind::Internal)),
+/// let wallet = Wallet::new(
+///     Bip84(key.clone(), KeychainKind::External),
+///     Some(Bip84(key, KeychainKind::Internal)),
 ///     Network::Testnet,
 ///     MemoryDatabase::default()
 /// )?;
 ///
-/// assert_eq!(wallet.get_new_address()?.to_string(), "tb1qedg9fdlf8cnnqfd5mks6uz5w4kgpk2pr6y4qc7");
-/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "wpkh([c55b303f/84\'/0\'/0\']tpubDC2Qwo2TFsaNC4ju8nrUJ9mqVT3eSgdmy1yPqhgkjwmke3PRXutNGRYAUo6RCHTcVQaDR3ohNU9we59brGHuEKPvH1ags2nevW5opEE9Z5Q/0/*)#nkk5dtkg");
+/// assert_eq!(wallet.get_address(New)?.to_string(), "tb1qhl85z42h7r4su5u37rvvw0gk8j2t3n9y7zsg4n");
+/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "wpkh([c55b303f/84'/1'/0']tpubDDc5mum24DekpNw92t6fHGp8Gr2JjF9J7i4TZBtN6Vp8xpAULG5CFaKsfugWa5imhrQQUZKXe261asP5koDHo5bs3qNTmf3U3o4v9SaB8gg/0/*)#6kfecsmr");
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
-pub struct BIP84<K: DerivableKey<Segwitv0>>(pub K, pub KeychainKind);
+pub struct Bip84<K: DerivableKey<Segwitv0>>(pub K, pub KeychainKind);
 
-impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for BIP84<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        Ok(P2WPKH(segwit_v0::make_bipxx_private(84, self.0, self.1)?).build()?)
+impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip84<K> {
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Wpkh(segwit_v0::make_bipxx_private(84, self.0, self.1, network)?).build(network)
     }
 }
 
@@ -368,7 +364,7 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for BIP84<K> {
 ///
 /// This template requires the parent fingerprint to populate correctly the metadata of PSBTs.
 ///
-/// See [`BIP84`] for a template that does the full derivation, but requires private data
+/// See [`Bip84`] for a template that does the full derivation, but requires private data
 /// for the key.
 ///
 /// ## Example
@@ -378,26 +374,27 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for BIP84<K> {
 /// # use bdk::bitcoin::{PrivateKey, Network};
 /// # use bdk::{Wallet,  KeychainKind};
 /// # use bdk::database::MemoryDatabase;
-/// use bdk::template::BIP84Public;
+/// # use bdk::wallet::AddressIndex::New;
+/// use bdk::template::Bip84Public;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC2Qwo2TFsaNC4ju8nrUJ9mqVT3eSgdmy1yPqhgkjwmke3PRXutNGRYAUo6RCHTcVQaDR3ohNU9we59brGHuEKPvH1ags2nevW5opEE9Z5Q")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new_offline(
-///     BIP84Public(key.clone(), fingerprint, KeychainKind::External),
-///     Some(BIP84Public(key, fingerprint, KeychainKind::Internal)),
+/// let wallet = Wallet::new(
+///     Bip84Public(key.clone(), fingerprint, KeychainKind::External),
+///     Some(Bip84Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,
 ///     MemoryDatabase::default()
 /// )?;
 ///
-/// assert_eq!(wallet.get_new_address()?.to_string(), "tb1qedg9fdlf8cnnqfd5mks6uz5w4kgpk2pr6y4qc7");
+/// assert_eq!(wallet.get_address(New)?.to_string(), "tb1qedg9fdlf8cnnqfd5mks6uz5w4kgpk2pr6y4qc7");
 /// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "wpkh([c55b303f/84\'/0\'/0\']tpubDC2Qwo2TFsaNC4ju8nrUJ9mqVT3eSgdmy1yPqhgkjwmke3PRXutNGRYAUo6RCHTcVQaDR3ohNU9we59brGHuEKPvH1ags2nevW5opEE9Z5Q/0/*)#nkk5dtkg");
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
-pub struct BIP84Public<K: DerivableKey<Segwitv0>>(pub K, pub bip32::Fingerprint, pub KeychainKind);
+pub struct Bip84Public<K: DerivableKey<Segwitv0>>(pub K, pub bip32::Fingerprint, pub KeychainKind);
 
-impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for BIP84Public<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        Ok(P2WPKH(segwit_v0::make_bipxx_public(84, self.0, self.1, self.2)?).build()?)
+impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip84Public<K> {
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Wpkh(segwit_v0::make_bipxx_public(84, self.0, self.1, self.2)?).build(network)
     }
 }
 
@@ -410,10 +407,19 @@ macro_rules! expand_make_bipxx {
                 bip: u32,
                 key: K,
                 keychain: KeychainKind,
+                network: Network,
             ) -> Result<impl IntoDescriptorKey<$ctx>, DescriptorError> {
                 let mut derivation_path = Vec::with_capacity(4);
                 derivation_path.push(bip32::ChildNumber::from_hardened_idx(bip)?);
-                derivation_path.push(bip32::ChildNumber::from_hardened_idx(0)?);
+
+                match network {
+                    Network::Bitcoin => {
+                        derivation_path.push(bip32::ChildNumber::from_hardened_idx(0)?);
+                    }
+                    _ => {
+                        derivation_path.push(bip32::ChildNumber::from_hardened_idx(1)?);
+                    }
+                }
                 derivation_path.push(bip32::ChildNumber::from_hardened_idx(0)?);
 
                 match keychain {
@@ -440,11 +446,11 @@ macro_rules! expand_make_bipxx {
                     KeychainKind::Internal => vec![bip32::ChildNumber::from_normal_idx(1)?].into(),
                 };
 
-                let mut source_path = Vec::with_capacity(3);
-                source_path.push(bip32::ChildNumber::from_hardened_idx(bip)?);
-                source_path.push(bip32::ChildNumber::from_hardened_idx(0)?);
-                source_path.push(bip32::ChildNumber::from_hardened_idx(0)?);
-                let source_path: bip32::DerivationPath = source_path.into();
+                let source_path = bip32::DerivationPath::from(vec![
+                    bip32::ChildNumber::from_hardened_idx(bip)?,
+                    bip32::ChildNumber::from_hardened_idx(0)?,
+                    bip32::ChildNumber::from_hardened_idx(0)?,
+                ]);
 
                 Ok((key, (parent_fingerprint, source_path), derivation_path))
             }
@@ -459,16 +465,51 @@ expand_make_bipxx!(segwit_v0, Segwitv0);
 mod test {
     // test existing descriptor templates, make sure they are expanded to the right descriptors
 
+    use std::str::FromStr;
+
     use super::*;
     use crate::descriptor::derived::AsDerived;
     use crate::descriptor::{DescriptorError, DescriptorMeta};
     use crate::keys::ValidNetworks;
-    use bitcoin::hashes::core::str::FromStr;
     use bitcoin::network::constants::Network::Regtest;
     use bitcoin::secp256k1::Secp256k1;
     use miniscript::descriptor::{DescriptorPublicKey, DescriptorTrait, KeyMap};
     use miniscript::Descriptor;
 
+    // BIP44 `pkh(key/44'/{0,1}'/0'/{0,1}/*)`
+    #[test]
+    fn test_bip44_template_cointype() {
+        use bitcoin::util::bip32::ChildNumber::{self, Hardened};
+
+        let xprvkey = bitcoin::util::bip32::ExtendedPrivKey::from_str("xprv9s21ZrQH143K2fpbqApQL69a4oKdGVnVN52R82Ft7d1pSqgKmajF62acJo3aMszZb6qQ22QsVECSFxvf9uyxFUvFYQMq3QbtwtRSMjLAhMf").unwrap();
+        assert_eq!(Network::Bitcoin, xprvkey.network);
+        let xdesc = Bip44(xprvkey, KeychainKind::Internal)
+            .build(Network::Bitcoin)
+            .unwrap();
+
+        if let ExtendedDescriptor::Pkh(pkh) = xdesc.0 {
+            let path: Vec<ChildNumber> = pkh.into_inner().full_derivation_path().into();
+            let purpose = path.get(0).unwrap();
+            assert!(matches!(purpose, Hardened { index: 44 }));
+            let coin_type = path.get(1).unwrap();
+            assert!(matches!(coin_type, Hardened { index: 0 }));
+        }
+
+        let tprvkey = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
+        assert_eq!(Network::Testnet, tprvkey.network);
+        let tdesc = Bip44(tprvkey, KeychainKind::Internal)
+            .build(Network::Testnet)
+            .unwrap();
+
+        if let ExtendedDescriptor::Pkh(pkh) = tdesc.0 {
+            let path: Vec<ChildNumber> = pkh.into_inner().full_derivation_path().into();
+            let purpose = path.get(0).unwrap();
+            assert!(matches!(purpose, Hardened { index: 44 }));
+            let coin_type = path.get(1).unwrap();
+            assert!(matches!(coin_type, Hardened { index: 1 }));
+        }
+    }
+
     // verify template descriptor generates expected address(es)
     fn check(
         desc: Result<(Descriptor<DescriptorPublicKey>, KeyMap, ValidNetworks), DescriptorError>,
@@ -500,7 +541,7 @@ mod test {
             bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")
                 .unwrap();
         check(
-            P2PKH(prvkey).build(),
+            P2Pkh(prvkey).build(Network::Bitcoin),
             false,
             true,
             &["mwJ8hxFYW19JLuc65RCTaP4v1rzVU8cVMT"],
@@ -511,7 +552,7 @@ mod test {
         )
         .unwrap();
         check(
-            P2PKH(pubkey).build(),
+            P2Pkh(pubkey).build(Network::Bitcoin),
             false,
             true,
             &["muZpTpBYhxmRFuCjLc7C6BBDF32C8XVJUi"],
@@ -525,7 +566,7 @@ mod test {
             bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")
                 .unwrap();
         check(
-            P2WPKH_P2SH(prvkey).build(),
+            P2Wpkh_P2Sh(prvkey).build(Network::Bitcoin),
             true,
             true,
             &["2NB4ox5VDRw1ecUv6SnT3VQHPXveYztRqk5"],
@@ -536,7 +577,7 @@ mod test {
         )
         .unwrap();
         check(
-            P2WPKH_P2SH(pubkey).build(),
+            P2Wpkh_P2Sh(pubkey).build(Network::Bitcoin),
             true,
             true,
             &["2N5LiC3CqzxDamRTPG1kiNv1FpNJQ7x28sb"],
@@ -550,7 +591,7 @@ mod test {
             bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")
                 .unwrap();
         check(
-            P2WPKH(prvkey).build(),
+            P2Wpkh(prvkey).build(Network::Bitcoin),
             true,
             true,
             &["bcrt1q4525hmgw265tl3drrl8jjta7ayffu6jfcwxx9y"],
@@ -561,7 +602,7 @@ mod test {
         )
         .unwrap();
         check(
-            P2WPKH(pubkey).build(),
+            P2Wpkh(pubkey).build(Network::Bitcoin),
             true,
             true,
             &["bcrt1qngw83fg8dz0k749cg7k3emc7v98wy0c7azaa6h"],
@@ -573,7 +614,7 @@ mod test {
     fn test_bip44_template() {
         let prvkey = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
         check(
-            BIP44(prvkey, KeychainKind::External).build(),
+            Bip44(prvkey, KeychainKind::External).build(Network::Bitcoin),
             false,
             false,
             &[
@@ -583,7 +624,7 @@ mod test {
             ],
         );
         check(
-            BIP44(prvkey, KeychainKind::Internal).build(),
+            Bip44(prvkey, KeychainKind::Internal).build(Network::Bitcoin),
             false,
             false,
             &[
@@ -600,7 +641,7 @@ mod test {
         let pubkey = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDDDzQ31JkZB7VxUr9bjvBivDdqoFLrDPyLWtLapArAi51ftfmCb2DPxwLQzX65iNcXz1DGaVvyvo6JQ6rTU73r2gqdEo8uov9QKRb7nKCSU").unwrap();
         let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f").unwrap();
         check(
-            BIP44Public(pubkey, fingerprint, KeychainKind::External).build(),
+            Bip44Public(pubkey, fingerprint, KeychainKind::External).build(Network::Bitcoin),
             false,
             false,
             &[
@@ -610,7 +651,7 @@ mod test {
             ],
         );
         check(
-            BIP44Public(pubkey, fingerprint, KeychainKind::Internal).build(),
+            Bip44Public(pubkey, fingerprint, KeychainKind::Internal).build(Network::Bitcoin),
             false,
             false,
             &[
@@ -626,7 +667,7 @@ mod test {
     fn test_bip49_template() {
         let prvkey = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
         check(
-            BIP49(prvkey, KeychainKind::External).build(),
+            Bip49(prvkey, KeychainKind::External).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -636,7 +677,7 @@ mod test {
             ],
         );
         check(
-            BIP49(prvkey, KeychainKind::Internal).build(),
+            Bip49(prvkey, KeychainKind::Internal).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -653,7 +694,7 @@ mod test {
         let pubkey = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L").unwrap();
         let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f").unwrap();
         check(
-            BIP49Public(pubkey, fingerprint, KeychainKind::External).build(),
+            Bip49Public(pubkey, fingerprint, KeychainKind::External).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -663,7 +704,7 @@ mod test {
             ],
         );
         check(
-            BIP49Public(pubkey, fingerprint, KeychainKind::Internal).build(),
+            Bip49Public(pubkey, fingerprint, KeychainKind::Internal).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -679,7 +720,7 @@ mod test {
     fn test_bip84_template() {
         let prvkey = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
         check(
-            BIP84(prvkey, KeychainKind::External).build(),
+            Bip84(prvkey, KeychainKind::External).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -689,7 +730,7 @@ mod test {
             ],
         );
         check(
-            BIP84(prvkey, KeychainKind::Internal).build(),
+            Bip84(prvkey, KeychainKind::Internal).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -706,7 +747,7 @@ mod test {
         let pubkey = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC2Qwo2TFsaNC4ju8nrUJ9mqVT3eSgdmy1yPqhgkjwmke3PRXutNGRYAUo6RCHTcVQaDR3ohNU9we59brGHuEKPvH1ags2nevW5opEE9Z5Q").unwrap();
         let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f").unwrap();
         check(
-            BIP84Public(pubkey, fingerprint, KeychainKind::External).build(),
+            Bip84Public(pubkey, fingerprint, KeychainKind::External).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -716,7 +757,7 @@ mod test {
             ],
         );
         check(
-            BIP84Public(pubkey, fingerprint, KeychainKind::Internal).build(),
+            Bip84Public(pubkey, fingerprint, KeychainKind::Internal).build(Network::Bitcoin),
             true,
             false,
             &[

src/doctest.rs

@@ -1,3 +1,14 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
 #[doc(include = "../README.md")]
 #[cfg(doctest)]
 pub struct ReadmeDoctests;

src/error.rs

@@ -1,29 +1,17 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 use std::fmt;
 
+use crate::bitcoin::Network;
 use crate::{descriptor, wallet, wallet::address_validator};
 use bitcoin::OutPoint;
 
@@ -36,10 +24,6 @@ pub enum Error {
     Generic(String),
     /// This error is thrown when trying to convert Bare and Public key script to address
     ScriptDoesntHaveAddressForm,
-    /// Found multiple outputs when `single_recipient` option has been specified
-    SingleRecipientMultipleOutputs,
-    /// `single_recipient` option is selected but neither `drain_wallet` nor `manually_selected_only` are
-    SingleRecipientNoInputs,
     /// Cannot build a tx without recipients
     NoRecipients,
     /// `manually_selected_only` option is selected but no utxo has been passed
@@ -60,7 +44,7 @@ pub enum Error {
     /// the desired outputs plus fee, if there is not such combination this error is thrown
     BnBNoExactMatch,
     /// Happens when trying to spend an UTXO that is not in the internal database
-    UnknownUTXO,
+    UnknownUtxo,
     /// Thrown when a tx is not found in the internal database
     TransactionNotFound,
     /// Happens when trying to bump a transaction that is already confirmed
@@ -77,6 +61,8 @@ pub enum Error {
         /// Required fee absolute value (satoshi)
         required: u64,
     },
+    /// Node doesn't have data to estimate a fee rate
+    FeeRateUnavailable,
     /// In order to use the [`TxBuilder::add_global_xpubs`] option every extended
     /// key in the descriptor must either be a master key itself (having depth = 0) or have an
     /// explicit origin provided
@@ -93,6 +79,16 @@ pub enum Error {
     InvalidPolicyPathError(crate::descriptor::policy::PolicyError),
     /// Signing error
     Signer(crate::wallet::signer::SignerError),
+    /// Invalid network
+    InvalidNetwork {
+        /// requested network, for example what is given as bdk-cli option
+        requested: Network,
+        /// found network, for example the network of the bitcoin node
+        found: Network,
+    },
+    #[cfg(feature = "verify")]
+    /// Transaction verification error
+    Verification(crate::wallet::verify::VerifyError),
 
     /// Progress value must be between `0.0` (included) and `100.0` (included)
     InvalidProgressValue(f32),
@@ -110,15 +106,17 @@ pub enum Error {
     /// Miniscript error
     Miniscript(miniscript::Error),
     /// BIP32 error
-    BIP32(bitcoin::util::bip32::Error),
+    Bip32(bitcoin::util::bip32::Error),
     /// An ECDSA error
     Secp256k1(bitcoin::secp256k1::Error),
     /// Error serializing or deserializing JSON data
-    JSON(serde_json::Error),
+    Json(serde_json::Error),
     /// Hex decoding error
     Hex(bitcoin::hashes::hex::Error),
     /// Partially signed bitcoin transaction error
-    PSBT(bitcoin::util::psbt::Error),
+    Psbt(bitcoin::util::psbt::Error),
+    /// Partially signed bitcoin transaction parse error
+    PsbtParse(bitcoin::util::psbt::PsbtParseError),
 
     //KeyMismatch(bitcoin::secp256k1::PublicKey, bitcoin::secp256k1::PublicKey),
     //MissingInputUTXO(usize),
@@ -132,13 +130,19 @@ pub enum Error {
     Electrum(electrum_client::Error),
     #[cfg(feature = "esplora")]
     /// Esplora client error
-    Esplora(crate::blockchain::esplora::EsploraError),
+    Esplora(Box<crate::blockchain::esplora::EsploraError>),
     #[cfg(feature = "compact_filters")]
     /// Compact filters client error)
     CompactFilters(crate::blockchain::compact_filters::CompactFiltersError),
     #[cfg(feature = "key-value-db")]
     /// Sled database error
     Sled(sled::Error),
+    #[cfg(feature = "rpc")]
+    /// Rpc client error
+    Rpc(bitcoincore_rpc::Error),
+    #[cfg(feature = "sqlite")]
+    /// Rusqlite client error
+    Rusqlite(rusqlite::Error),
 }
 
 impl fmt::Display for Error {
@@ -171,7 +175,7 @@ impl From<crate::keys::KeyError> for Error {
     fn from(key_error: crate::keys::KeyError) -> Error {
         match key_error {
             crate::keys::KeyError::Miniscript(inner) => Error::Miniscript(inner),
-            crate::keys::KeyError::BIP32(inner) => Error::BIP32(inner),
+            crate::keys::KeyError::Bip32(inner) => Error::Bip32(inner),
             crate::keys::KeyError::InvalidChecksum => Error::ChecksumMismatch,
             e => Error::Key(e),
         }
@@ -180,18 +184,21 @@ impl From<crate::keys::KeyError> for Error {
 
 impl_error!(bitcoin::consensus::encode::Error, Encode);
 impl_error!(miniscript::Error, Miniscript);
-impl_error!(bitcoin::util::bip32::Error, BIP32);
+impl_error!(bitcoin::util::bip32::Error, Bip32);
 impl_error!(bitcoin::secp256k1::Error, Secp256k1);
-impl_error!(serde_json::Error, JSON);
+impl_error!(serde_json::Error, Json);
 impl_error!(bitcoin::hashes::hex::Error, Hex);
-impl_error!(bitcoin::util::psbt::Error, PSBT);
+impl_error!(bitcoin::util::psbt::Error, Psbt);
+impl_error!(bitcoin::util::psbt::PsbtParseError, PsbtParse);
 
 #[cfg(feature = "electrum")]
 impl_error!(electrum_client::Error, Electrum);
-#[cfg(feature = "esplora")]
-impl_error!(crate::blockchain::esplora::EsploraError, Esplora);
 #[cfg(feature = "key-value-db")]
 impl_error!(sled::Error, Sled);
+#[cfg(feature = "rpc")]
+impl_error!(bitcoincore_rpc::Error, Rpc);
+#[cfg(feature = "sqlite")]
+impl_error!(rusqlite::Error, Rusqlite);
 
 #[cfg(feature = "compact_filters")]
 impl From<crate::blockchain::compact_filters::CompactFiltersError> for Error {
@@ -202,3 +209,20 @@ impl From<crate::blockchain::compact_filters::CompactFiltersError> for Error {
         }
     }
 }
+
+#[cfg(feature = "verify")]
+impl From<crate::wallet::verify::VerifyError> for Error {
+    fn from(other: crate::wallet::verify::VerifyError) -> Self {
+        match other {
+            crate::wallet::verify::VerifyError::Global(inner) => *inner,
+            err => Error::Verification(err),
+        }
+    }
+}
+
+#[cfg(feature = "esplora")]
+impl From<crate::blockchain::esplora::EsploraError> for Error {
+    fn from(other: crate::blockchain::esplora::EsploraError) -> Self {
+        Error::Esplora(Box::new(other))
+    }
+}

src/keys/bip39.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! BIP-0039
 
@@ -32,7 +19,23 @@ use bitcoin::Network;
 
 use miniscript::ScriptContext;
 
-pub use bip39::{Language, Mnemonic, MnemonicType, Seed};
+pub use bip39::{Error, Language, Mnemonic};
+
+type Seed = [u8; 64];
+
+/// Type describing entropy length (aka word count) in the mnemonic
+pub enum WordCount {
+    /// 12 words mnemonic (128 bits entropy)
+    Words12 = 128,
+    /// 15 words mnemonic (160 bits entropy)
+    Words15 = 160,
+    /// 18 words mnemonic (192 bits entropy)
+    Words18 = 192,
+    /// 21 words mnemonic (224 bits entropy)
+    Words21 = 224,
+    /// 24 words mnemonic (256 bits entropy)
+    Words24 = 256,
+}
 
 use super::{
     any_network, DerivableKey, DescriptorKey, ExtendedKey, GeneratableKey, GeneratedKey, KeyError,
@@ -53,7 +56,7 @@ pub type MnemonicWithPassphrase = (Mnemonic, Option<String>);
 #[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
 impl<Ctx: ScriptContext> DerivableKey<Ctx> for Seed {
     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
-        Ok(bip32::ExtendedPrivKey::new_master(Network::Bitcoin, &self.as_bytes())?.into())
+        Ok(bip32::ExtendedPrivKey::new_master(Network::Bitcoin, &self[..])?.into())
     }
 
     fn into_descriptor_key(
@@ -73,7 +76,7 @@ impl<Ctx: ScriptContext> DerivableKey<Ctx> for Seed {
 impl<Ctx: ScriptContext> DerivableKey<Ctx> for MnemonicWithPassphrase {
     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
         let (mnemonic, passphrase) = self;
-        let seed = Seed::new(&mnemonic, passphrase.as_deref().unwrap_or(""));
+        let seed: Seed = mnemonic.to_seed(passphrase.as_deref().unwrap_or(""));
 
         seed.into_extended_key()
     }
@@ -91,6 +94,23 @@ impl<Ctx: ScriptContext> DerivableKey<Ctx> for MnemonicWithPassphrase {
     }
 }
 
+#[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
+impl<Ctx: ScriptContext> DerivableKey<Ctx> for (GeneratedKey<Mnemonic, Ctx>, Option<String>) {
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        let (mnemonic, passphrase) = self;
+        (mnemonic.into_key(), passphrase).into_extended_key()
+    }
+
+    fn into_descriptor_key(
+        self,
+        source: Option<bip32::KeySource>,
+        derivation_path: bip32::DerivationPath,
+    ) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let (mnemonic, passphrase) = self;
+        (mnemonic.into_key(), passphrase).into_descriptor_key(source, derivation_path)
+    }
+}
+
 #[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
 impl<Ctx: ScriptContext> DerivableKey<Ctx> for Mnemonic {
     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
@@ -114,15 +134,15 @@ impl<Ctx: ScriptContext> DerivableKey<Ctx> for Mnemonic {
 impl<Ctx: ScriptContext> GeneratableKey<Ctx> for Mnemonic {
     type Entropy = [u8; 32];
 
-    type Options = (MnemonicType, Language);
-    type Error = Option<bip39::ErrorKind>;
+    type Options = (WordCount, Language);
+    type Error = Option<bip39::Error>;
 
     fn generate_with_entropy(
-        (mnemonic_type, language): Self::Options,
+        (word_count, language): Self::Options,
         entropy: Self::Entropy,
     ) -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
-        let entropy = &entropy.as_ref()[..(mnemonic_type.entropy_bits() / 8)];
-        let mnemonic = Mnemonic::from_entropy(entropy, language).map_err(|e| e.downcast().ok())?;
+        let entropy = &entropy.as_ref()[..(word_count as usize / 8)];
+        let mnemonic = Mnemonic::from_entropy_in(language, entropy)?;
 
         Ok(GeneratedKey::new(mnemonic, any_network()))
     }
@@ -134,15 +154,17 @@ mod test {
 
     use bitcoin::util::bip32;
 
-    use bip39::{Language, Mnemonic, MnemonicType};
+    use bip39::{Language, Mnemonic};
 
     use crate::keys::{any_network, GeneratableKey, GeneratedKey};
 
+    use super::WordCount;
+
     #[test]
     fn test_keys_bip39_mnemonic() {
         let mnemonic =
             "aim bunker wash balance finish force paper analyst cabin spoon stable organ";
-        let mnemonic = Mnemonic::from_phrase(mnemonic, Language::English).unwrap();
+        let mnemonic = Mnemonic::parse_in(Language::English, mnemonic).unwrap();
         let path = bip32::DerivationPath::from_str("m/44'/0'/0'/0").unwrap();
 
         let key = (mnemonic, path);
@@ -156,7 +178,7 @@ mod test {
     fn test_keys_bip39_mnemonic_passphrase() {
         let mnemonic =
             "aim bunker wash balance finish force paper analyst cabin spoon stable organ";
-        let mnemonic = Mnemonic::from_phrase(mnemonic, Language::English).unwrap();
+        let mnemonic = Mnemonic::parse_in(Language::English, mnemonic).unwrap();
         let path = bip32::DerivationPath::from_str("m/44'/0'/0'/0").unwrap();
 
         let key = ((mnemonic, Some("passphrase".into())), path);
@@ -170,7 +192,7 @@ mod test {
     fn test_keys_generate_bip39() {
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
             Mnemonic::generate_with_entropy(
-                (MnemonicType::Words12, Language::English),
+                (WordCount::Words12, Language::English),
                 crate::keys::test::TEST_ENTROPY,
             )
             .unwrap();
@@ -182,7 +204,7 @@ mod test {
 
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
             Mnemonic::generate_with_entropy(
-                (MnemonicType::Words24, Language::English),
+                (WordCount::Words24, Language::English),
                 crate::keys::test::TEST_ENTROPY,
             )
             .unwrap();
@@ -193,11 +215,11 @@ mod test {
     #[test]
     fn test_keys_generate_bip39_random() {
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
-            Mnemonic::generate((MnemonicType::Words12, Language::English)).unwrap();
+            Mnemonic::generate((WordCount::Words12, Language::English)).unwrap();
         assert_eq!(generated_mnemonic.valid_networks, any_network());
 
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
-            Mnemonic::generate((MnemonicType::Words24, Language::English)).unwrap();
+            Mnemonic::generate((WordCount::Words24, Language::English)).unwrap();
         assert_eq!(generated_mnemonic.valid_networks, any_network());
     }
 }

src/keys/mod.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Key formats
 
@@ -33,12 +20,12 @@ use std::str::FromStr;
 use bitcoin::secp256k1::{self, Secp256k1, Signing};
 
 use bitcoin::util::bip32;
-use bitcoin::{Network, PrivateKey, PublicKey};
+use bitcoin::{Network, PrivateKey, PublicKey, XOnlyPublicKey};
 
 use miniscript::descriptor::{Descriptor, DescriptorXKey, Wildcard};
 pub use miniscript::descriptor::{
     DescriptorPublicKey, DescriptorSecretKey, DescriptorSinglePriv, DescriptorSinglePub, KeyMap,
-    SortedMultiVec,
+    SinglePubKey, SortedMultiVec,
 };
 pub use miniscript::ScriptContext;
 use miniscript::{Miniscript, Terminal};
@@ -140,6 +127,8 @@ pub enum ScriptContextEnum {
     Legacy,
     /// Segwitv0 scripts
     Segwitv0,
+    /// Taproot scripts
+    Tap,
 }
 
 impl ScriptContextEnum {
@@ -152,6 +141,11 @@ impl ScriptContextEnum {
     pub fn is_segwit_v0(&self) -> bool {
         self == &ScriptContextEnum::Segwitv0
     }
+
+    /// Returns whether the script context is [`ScriptContextEnum::Tap`]
+    pub fn is_taproot(&self) -> bool {
+        self == &ScriptContextEnum::Tap
+    }
 }
 
 /// Trait that adds extra useful methods to [`ScriptContext`]s
@@ -168,6 +162,11 @@ pub trait ExtScriptContext: ScriptContext {
     fn is_segwit_v0() -> bool {
         Self::as_enum().is_segwit_v0()
     }
+
+    /// Returns whether the script context is [`Tap`](miniscript::Tap), aka Taproot or Segwit V1
+    fn is_taproot() -> bool {
+        Self::as_enum().is_taproot()
+    }
 }
 
 impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
@@ -175,6 +174,7 @@ impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
         match TypeId::of::<Ctx>() {
             t if t == TypeId::of::<miniscript::Legacy>() => ScriptContextEnum::Legacy,
             t if t == TypeId::of::<miniscript::Segwitv0>() => ScriptContextEnum::Segwitv0,
+            t if t == TypeId::of::<miniscript::Tap>() => ScriptContextEnum::Tap,
             _ => unimplemented!("Unknown ScriptContext type"),
         }
     }
@@ -205,7 +205,7 @@ impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
 /// ```
 /// use bdk::bitcoin::PublicKey;
 ///
-/// use bdk::keys::{DescriptorKey, KeyError, ScriptContext, IntoDescriptorKey};
+/// use bdk::keys::{DescriptorKey, IntoDescriptorKey, KeyError, ScriptContext};
 ///
 /// pub struct MyKeyType {
 ///     pubkey: PublicKey,
@@ -224,8 +224,8 @@ impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
 /// use bdk::bitcoin::PublicKey;
 ///
 /// use bdk::keys::{
-///     mainnet_network, DescriptorKey, DescriptorPublicKey, DescriptorSinglePub, KeyError,
-///     ScriptContext, IntoDescriptorKey,
+///     mainnet_network, DescriptorKey, DescriptorPublicKey, DescriptorSinglePub,
+///     IntoDescriptorKey, KeyError, ScriptContext, SinglePubKey,
 /// };
 ///
 /// pub struct MyKeyType {
@@ -237,7 +237,7 @@ impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
 ///         Ok(DescriptorKey::from_public(
 ///             DescriptorPublicKey::SinglePub(DescriptorSinglePub {
 ///                 origin: None,
-///                 key: self.pubkey,
+///                 key: SinglePubKey::FullKey(self.pubkey),
 ///             }),
 ///             mainnet_network(),
 ///         ))
@@ -250,7 +250,7 @@ impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
 /// ```
 /// use bdk::bitcoin::PublicKey;
 ///
-/// use bdk::keys::{DescriptorKey, ExtScriptContext, KeyError, ScriptContext, IntoDescriptorKey};
+/// use bdk::keys::{DescriptorKey, ExtScriptContext, IntoDescriptorKey, KeyError, ScriptContext};
 ///
 /// pub struct MyKeyType {
 ///     is_legacy: bool,
@@ -279,7 +279,7 @@ impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
 /// use bdk::bitcoin::PublicKey;
 /// use std::str::FromStr;
 ///
-/// use bdk::keys::{DescriptorKey, KeyError, IntoDescriptorKey};
+/// use bdk::keys::{DescriptorKey, IntoDescriptorKey, KeyError};
 ///
 /// pub struct MySegwitOnlyKeyType {
 ///     pubkey: PublicKey,
@@ -346,7 +346,7 @@ impl<Ctx: ScriptContext> ExtendedKey<Ctx> {
         secp: &Secp256k1<C>,
     ) -> bip32::ExtendedPubKey {
         let mut xpub = match self {
-            ExtendedKey::Private((xprv, _)) => bip32::ExtendedPubKey::from_private(secp, &xprv),
+            ExtendedKey::Private((xprv, _)) => bip32::ExtendedPubKey::from_priv(secp, &xprv),
             ExtendedKey::Public((xpub, _)) => xpub,
         };
 
@@ -369,7 +369,7 @@ impl<Ctx: ScriptContext> From<bip32::ExtendedPrivKey> for ExtendedKey<Ctx> {
 
 /// Trait for keys that can be derived.
 ///
-/// When extra metadata are provided, a [`DerivableKey`] can be transofrmed into a
+/// When extra metadata are provided, a [`DerivableKey`] can be transformed into a
 /// [`DescriptorKey`]: the trait [`IntoDescriptorKey`] is automatically implemented
 /// for `(DerivableKey, DerivationPath)` and
 /// `(DerivableKey, KeySource, DerivationPath)` tuples.
@@ -400,7 +400,7 @@ impl<Ctx: ScriptContext> From<bip32::ExtendedPrivKey> for ExtendedKey<Ctx> {
 ///             network: self.network,
 ///             depth: 0,
 ///             parent_fingerprint: bip32::Fingerprint::default(),
-///             private_key: self.key_data,
+///             private_key: self.key_data.inner,
 ///             chain_code: bip32::ChainCode::from(self.chain_code.as_ref()),
 ///             child_number: bip32::ChildNumber::Normal { index: 0 },
 ///         };
@@ -432,7 +432,7 @@ impl<Ctx: ScriptContext> From<bip32::ExtendedPrivKey> for ExtendedKey<Ctx> {
 ///             network: bitcoin::Network::Bitcoin, // pick an arbitrary network here
 ///             depth: 0,
 ///             parent_fingerprint: bip32::Fingerprint::default(),
-///             private_key: self.key_data,
+///             private_key: self.key_data.inner,
 ///             chain_code: bip32::ChainCode::from(self.chain_code.as_ref()),
 ///             child_number: bip32::ChildNumber::Normal { index: 0 },
 ///         };
@@ -473,9 +473,9 @@ use bdk::keys::bip39::{Mnemonic, Language};
 
 # fn main() -> Result<(), Box<dyn std::error::Error>> {
 let xkey: ExtendedKey =
-    Mnemonic::from_phrase(
+    Mnemonic::parse_in(
+        Language::English,
         "jelly crash boy whisper mouse ecology tuna soccer memory million news short",
-        Language::English
     )?
     .into_extended_key()?;
 let xprv = xkey.into_xprv(Network::Bitcoin).unwrap();
@@ -560,6 +560,16 @@ impl<K, Ctx: ScriptContext> Deref for GeneratedKey<K, Ctx> {
     }
 }
 
+impl<K: Clone, Ctx: ScriptContext> Clone for GeneratedKey<K, Ctx> {
+    fn clone(&self) -> GeneratedKey<K, Ctx> {
+        GeneratedKey {
+            key: self.key.clone(),
+            valid_networks: self.valid_networks.clone(),
+            phantom: self.phantom,
+        }
+    }
+}
+
 // Make generated "derivable" keys themselves "derivable". Also make sure they are assigned the
 // right `valid_networks`.
 impl<Ctx, K> DerivableKey<Ctx> for GeneratedKey<K, Ctx>
@@ -700,11 +710,11 @@ impl<Ctx: ScriptContext> GeneratableKey<Ctx> for PrivateKey {
         entropy: Self::Entropy,
     ) -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
         // pick a arbitrary network here, but say that we support all of them
-        let key = secp256k1::SecretKey::from_slice(&entropy)?;
+        let inner = secp256k1::SecretKey::from_slice(&entropy)?;
         let private_key = PrivateKey {
             compressed: options.compressed,
             network: Network::Bitcoin,
-            key,
+            inner,
         };
 
         Ok(GeneratedKey::new(private_key, any_network()))
@@ -733,7 +743,7 @@ fn expand_multi_keys<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
 ) -> Result<(Vec<DescriptorPublicKey>, KeyMap, ValidNetworks), KeyError> {
     let (pks, key_maps_networks): (Vec<_>, Vec<_>) = pks
         .into_iter()
-        .map(|key| Ok::<_, KeyError>(key.into_descriptor_key()?.extract(secp)?))
+        .map(|key| key.into_descriptor_key()?.extract(secp))
         .collect::<Result<Vec<_>, _>>()?
         .into_iter()
         .map(|(a, b, c)| (a, (b, c)))
@@ -761,22 +771,41 @@ pub fn make_pk<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
     let (key, key_map, valid_networks) = descriptor_key.into_descriptor_key()?.extract(secp)?;
     let minisc = Miniscript::from_ast(Terminal::PkK(key))?;
 
-    minisc.check_minsicript()?;
+    minisc.check_miniscript()?;
+
+    Ok((minisc, key_map, valid_networks))
+}
+
+// Used internally by `bdk::fragment!` to build `pk_h()` fragments
+#[doc(hidden)]
+pub fn make_pkh<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
+    descriptor_key: Pk,
+    secp: &SecpCtx,
+) -> Result<(Miniscript<DescriptorPublicKey, Ctx>, KeyMap, ValidNetworks), DescriptorError> {
+    let (key, key_map, valid_networks) = descriptor_key.into_descriptor_key()?.extract(secp)?;
+    let minisc = Miniscript::from_ast(Terminal::PkH(key))?;
+
+    minisc.check_miniscript()?;
 
     Ok((minisc, key_map, valid_networks))
 }
 
 // Used internally by `bdk::fragment!` to build `multi()` fragments
 #[doc(hidden)]
-pub fn make_multi<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
+pub fn make_multi<
+    Pk: IntoDescriptorKey<Ctx>,
+    Ctx: ScriptContext,
+    V: Fn(usize, Vec<DescriptorPublicKey>) -> Terminal<DescriptorPublicKey, Ctx>,
+>(
     thresh: usize,
+    variant: V,
     pks: Vec<Pk>,
     secp: &SecpCtx,
 ) -> Result<(Miniscript<DescriptorPublicKey, Ctx>, KeyMap, ValidNetworks), DescriptorError> {
     let (pks, key_map, valid_networks) = expand_multi_keys(pks, secp)?;
-    let minisc = Miniscript::from_ast(Terminal::Multi(thresh, pks))?;
+    let minisc = Miniscript::from_ast(variant(thresh, pks))?;
 
-    minisc.check_minsicript()?;
+    minisc.check_miniscript()?;
 
     Ok((minisc, key_map, valid_networks))
 }
@@ -829,7 +858,17 @@ impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for DescriptorPublicKey {
 impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for PublicKey {
     fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
         DescriptorPublicKey::SinglePub(DescriptorSinglePub {
-            key: self,
+            key: SinglePubKey::FullKey(self),
+            origin: None,
+        })
+        .into_descriptor_key()
+    }
+}
+
+impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for XOnlyPublicKey {
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+            key: SinglePubKey::XOnly(self),
             origin: None,
         })
         .into_descriptor_key()
@@ -886,13 +925,13 @@ pub enum KeyError {
     Message(String),
 
     /// BIP32 error
-    BIP32(bitcoin::util::bip32::Error),
+    Bip32(bitcoin::util::bip32::Error),
     /// Miniscript error
     Miniscript(miniscript::Error),
 }
 
 impl_error!(miniscript::Error, Miniscript, KeyError);
-impl_error!(bitcoin::util::bip32::Error, BIP32, KeyError);
+impl_error!(bitcoin::util::bip32::Error, Bip32, KeyError);
 
 impl std::fmt::Display for KeyError {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
@@ -930,4 +969,19 @@ pub mod test {
             "L2wTu6hQrnDMiFNWA5na6jB12ErGQqtXwqpSL7aWquJaZG8Ai3ch"
         );
     }
+
+    #[cfg(feature = "keys-bip39")]
+    #[test]
+    fn test_keys_wif_network_bip39() {
+        let xkey: ExtendedKey = bip39::Mnemonic::parse_in(
+            bip39::Language::English,
+            "jelly crash boy whisper mouse ecology tuna soccer memory million news short",
+        )
+        .unwrap()
+        .into_extended_key()
+        .unwrap();
+        let xprv = xkey.into_xprv(Network::Testnet).unwrap();
+
+        assert_eq!(xprv.network, Network::Testnet);
+    }
 }

src/lib.rs

@@ -1,35 +1,23 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 // rustdoc will warn if there are missing docs
 #![warn(missing_docs)]
 // only enables the `doc_cfg` feature when
 // the `docsrs` configuration attribute is defined
 #![cfg_attr(docsrs, feature(doc_cfg))]
-// only enables the nightly `external_doc` feature when
-// `test-md-docs` is enabled
-#![cfg_attr(feature = "test-md-docs", feature(external_doc))]
+#![cfg_attr(
+    docsrs,
+    doc(html_logo_url = "https://github.com/bitcoindevkit/bdk/raw/master/static/bdk.png")
+)]
 
 //! A modern, lightweight, descriptor-based wallet library written in Rust.
 //!
@@ -56,112 +44,121 @@
 //! interact with the bitcoin P2P network.
 //!
 //! ```toml
-//! bdk = "0.4.0"
+//! bdk = "0.20.0"
 //! ```
 //!
-//! ## Sync the balance of a descriptor
-//!
-//! ### Example
-//! ```ignore
-//! use bdk::Wallet;
-//! use bdk::database::MemoryDatabase;
-//! use bdk::blockchain::{noop_progress, ElectrumBlockchain};
-//!
-//! use bdk::electrum_client::Client;
-//!
-//! fn main() -> Result<(), bdk::Error> {
-//!     let client = Client::new("ssl://electrum.blockstream.info:60002")?;
-//!     let wallet = Wallet::new(
-//!         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
-//!         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
-//!         bitcoin::Network::Testnet,
-//!         MemoryDatabase::default(),
-//!         ElectrumBlockchain::from(client)
-//!     )?;
-//!
-//!     wallet.sync(noop_progress(), None)?;
-//!
-//!     println!("Descriptor balance: {} SAT", wallet.get_balance()?);
-//!
-//!     Ok(())
-//! }
-//! ```
+//! # Examples
+#![cfg_attr(
+    feature = "electrum",
+    doc = r##"
+## Sync the balance of a descriptor
+
+```no_run
+use bdk::{Wallet, SyncOptions};
+use bdk::database::MemoryDatabase;
+use bdk::blockchain::ElectrumBlockchain;
+use bdk::electrum_client::Client;
+
+fn main() -> Result<(), bdk::Error> {
+    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let blockchain = ElectrumBlockchain::from(client);
+    let wallet = Wallet::new(
+        "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
+        Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
+        bitcoin::Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!("Descriptor balance: {} SAT", wallet.get_balance()?);
+
+    Ok(())
+}
+```
+"##
+)]
 //!
 //! ## Generate a few addresses
 //!
-//! ### Example
 //! ```
 //! use bdk::{Wallet};
 //! use bdk::database::MemoryDatabase;
+//! use bdk::wallet::AddressIndex::New;
 //!
 //! fn main() -> Result<(), bdk::Error> {
-//!     let wallet = Wallet::new_offline(
+//! let wallet = Wallet::new(
 //!         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
 //!         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
 //!         bitcoin::Network::Testnet,
 //!         MemoryDatabase::default(),
 //!     )?;
 //!
-//!     println!("Address #0: {}", wallet.get_new_address()?);
-//!     println!("Address #1: {}", wallet.get_new_address()?);
-//!     println!("Address #2: {}", wallet.get_new_address()?);
-//!
-//!     Ok(())
-//! }
-//! ```
-//!
-//! ## Create a transaction
-//!
-//! ### Example
-//! ```ignore
-//! use base64::decode;
-//! use bdk::{FeeRate, Wallet};
-//! use bdk::database::MemoryDatabase;
-//! use bdk::blockchain::{noop_progress, ElectrumBlockchain};
-//!
-//! use bdk::electrum_client::Client;
-//!
-//! use bitcoin::consensus::serialize;
-//!
-//! fn main() -> Result<(), bdk::Error> {
-//!     let client = Client::new("ssl://electrum.blockstream.info:60002")?;
-//!     let wallet = Wallet::new(
-//!         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
-//!         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
-//!         bitcoin::Network::Testnet,
-//!         MemoryDatabase::default(),
-//!         ElectrumBlockchain::from(client)
-//!     )?;
-//!
-//!     wallet.sync(noop_progress(), None)?;
-//!
-//!     let send_to = wallet.get_new_address()?;
-//!     let (psbt, details) = wallet.build_tx()
-//!         .add_recipient(send_to.script_pubkey(), 50_000)
-//!         .enable_rbf()
-//!         .do_not_spend_change()
-//!         .fee_rate(FeeRate::from_sat_per_vb(5.0))
-//!         .finish()?;
-//!
-//!     println!("Transaction details: {:#?}", details);
-//!     println!("Unsigned PSBT: {}", base64::encode(&serialize(&psbt)));
+//!     println!("Address #0: {}", wallet.get_address(New)?);
+//!     println!("Address #1: {}", wallet.get_address(New)?);
+//!     println!("Address #2: {}", wallet.get_address(New)?);
 //!
 //!     Ok(())
 //! }
 //! ```
+#![cfg_attr(
+    feature = "electrum",
+    doc = r##"
+## Create a transaction
+
+```no_run
+use bdk::{FeeRate, Wallet, SyncOptions};
+use bdk::database::MemoryDatabase;
+use bdk::blockchain::ElectrumBlockchain;
+use bdk::electrum_client::Client;
+
+use bitcoin::consensus::serialize;
+use bdk::wallet::AddressIndex::New;
+
+fn main() -> Result<(), bdk::Error> {
+    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let wallet = Wallet::new(
+        "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
+        Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
+        bitcoin::Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+    let blockchain = ElectrumBlockchain::from(client);
+
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    let send_to = wallet.get_address(New)?;
+    let (psbt, details) = {
+        let mut builder =  wallet.build_tx();
+        builder
+            .add_recipient(send_to.script_pubkey(), 50_000)
+            .enable_rbf()
+            .do_not_spend_change()
+            .fee_rate(FeeRate::from_sat_per_vb(5.0));
+        builder.finish()?
+    };
+
+    println!("Transaction details: {:#?}", details);
+    println!("Unsigned PSBT: {}", &psbt);
+
+    Ok(())
+}
+```
+"##
+)]
 //!
 //! ## Sign a transaction
 //!
-//! ### Example
-//! ```ignore
-//! use base64::decode;
-//! use bdk::{Wallet};
+//! ```no_run
+//! use std::str::FromStr;
+//!
+//! use bitcoin::util::psbt::PartiallySignedTransaction as Psbt;
+//!
+//! use bdk::{Wallet, SignOptions};
 //! use bdk::database::MemoryDatabase;
 //!
-//! use bitcoin::consensus::deserialize;
-//!
 //! fn main() -> Result<(), bdk::Error> {
-//!     let wallet = Wallet::new_offline(
+//!     let wallet = Wallet::new(
 //!         "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
 //!         Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
 //!         bitcoin::Network::Testnet,
@@ -169,9 +166,9 @@
 //!     )?;
 //!
 //!     let psbt = "...";
-//!     let psbt = deserialize(&base64::decode(psbt).unwrap())?;
+//!     let mut psbt = Psbt::from_str(psbt)?;
 //!
-//!     let (signed_psbt, finalized) = wallet.sign(psbt, None)?;
+//!     let finalized = wallet.sign(&mut psbt, SignOptions::default())?;
 //!
 //!     Ok(())
 //! }
@@ -193,7 +190,7 @@
 //! * `async-interface`: async functions in bdk traits
 //! * `keys-bip39`: [BIP-39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) mnemonic codes for generating deterministic keys
 //!
-//! ## Internal features
+//! # Internal features
 //!
 //! These features do not expose any new API, but influence internal implementation aspects of
 //! BDK.
@@ -210,11 +207,24 @@ extern crate serde;
 #[macro_use]
 extern crate serde_json;
 
+#[cfg(all(feature = "reqwest", feature = "ureq"))]
+compile_error!("Features reqwest and ureq are mutually exclusive and cannot be enabled together");
+
 #[cfg(all(feature = "async-interface", feature = "electrum"))]
 compile_error!(
     "Features async-interface and electrum are mutually exclusive and cannot be enabled together"
 );
 
+#[cfg(all(feature = "async-interface", feature = "ureq"))]
+compile_error!(
+    "Features async-interface and ureq are mutually exclusive and cannot be enabled together"
+);
+
+#[cfg(all(feature = "async-interface", feature = "compact_filters"))]
+compile_error!(
+    "Features async-interface and compact_filters are mutually exclusive and cannot be enabled together"
+);
+
 #[cfg(feature = "keys-bip39")]
 extern crate bip39;
 
@@ -225,31 +235,29 @@ extern crate async_trait;
 extern crate bdk_macros;
 
 #[cfg(feature = "compact_filters")]
-#[macro_use]
 extern crate lazy_static;
 
+#[cfg(feature = "rpc")]
+pub extern crate bitcoincore_rpc;
+
 #[cfg(feature = "electrum")]
 pub extern crate electrum_client;
 
-#[cfg(feature = "esplora")]
-pub extern crate reqwest;
-
 #[cfg(feature = "key-value-db")]
 pub extern crate sled;
 
-#[allow(unused_imports)]
-#[cfg(test)]
-#[macro_use]
-extern crate testutils;
-#[allow(unused_imports)]
-#[cfg(test)]
-#[macro_use]
-extern crate testutils_macros;
-#[allow(unused_imports)]
-#[cfg(test)]
-#[macro_use]
-extern crate serial_test;
+#[cfg(feature = "sqlite")]
+pub extern crate rusqlite;
 
+// We should consider putting this under a feature flag but we need the macro in doctests so we need
+// to wait until https://github.com/rust-lang/rust/issues/67295 is fixed.
+//
+// Stuff in here is too rough to document atm
+#[doc(hidden)]
+#[macro_use]
+pub mod testutils;
+
+#[allow(unused_imports)]
 #[macro_use]
 pub(crate) mod error;
 pub mod blockchain;
@@ -263,12 +271,14 @@ pub(crate) mod types;
 pub mod wallet;
 
 pub use descriptor::template;
-pub use descriptor::HDKeyPaths;
+pub use descriptor::HdKeyPaths;
 pub use error::Error;
 pub use types::*;
 pub use wallet::address_validator;
 pub use wallet::signer;
+pub use wallet::signer::SignOptions;
 pub use wallet::tx_builder::TxBuilder;
+pub use wallet::SyncOptions;
 pub use wallet::Wallet;
 
 /// Get the version of BDK at runtime

src/psbt/mod.rs

@@ -1,37 +1,25 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
-use bitcoin::util::psbt::PartiallySignedTransaction as PSBT;
+use bitcoin::util::psbt::PartiallySignedTransaction as Psbt;
 use bitcoin::TxOut;
 
-pub trait PSBTUtils {
+pub trait PsbtUtils {
     fn get_utxo_for(&self, input_index: usize) -> Option<TxOut>;
 }
 
-impl PSBTUtils for PSBT {
+impl PsbtUtils for Psbt {
+    #[allow(clippy::all)] // We want to allow `manual_map` but it is too new.
     fn get_utxo_for(&self, input_index: usize) -> Option<TxOut> {
-        let tx = &self.global.unsigned_tx;
+        let tx = &self.unsigned_tx;
 
         if input_index >= tx.input.len() {
             return None;
@@ -50,3 +38,84 @@ impl PSBTUtils for PSBT {
         }
     }
 }
+
+#[cfg(test)]
+mod test {
+    use crate::bitcoin::TxIn;
+    use crate::psbt::Psbt;
+    use crate::wallet::AddressIndex;
+    use crate::wallet::{get_funded_wallet, test::get_test_wpkh};
+    use crate::SignOptions;
+    use std::str::FromStr;
+
+    // from bip 174
+    const PSBT_STR: &str = "cHNidP8BAKACAAAAAqsJSaCMWvfEm4IS9Bfi8Vqz9cM9zxU4IagTn4d6W3vkAAAAAAD+////qwlJoIxa98SbghL0F+LxWrP1wz3PFTghqBOfh3pbe+QBAAAAAP7///8CYDvqCwAAAAAZdqkUdopAu9dAy+gdmI5x3ipNXHE5ax2IrI4kAAAAAAAAGXapFG9GILVT+glechue4O/p+gOcykWXiKwAAAAAAAEHakcwRAIgR1lmF5fAGwNrJZKJSGhiGDR9iYZLcZ4ff89X0eURZYcCIFMJ6r9Wqk2Ikf/REf3xM286KdqGbX+EhtdVRs7tr5MZASEDXNxh/HupccC1AaZGoqg7ECy0OIEhfKaC3Ibi1z+ogpIAAQEgAOH1BQAAAAAXqRQ1RebjO4MsRwUPJNPuuTycA5SLx4cBBBYAFIXRNTfy4mVAWjTbr6nj3aAfuCMIAAAA";
+
+    #[test]
+    #[should_panic(expected = "InputIndexOutOfRange")]
+    fn test_psbt_malformed_psbt_input_legacy() {
+        let psbt_bip = Psbt::from_str(PSBT_STR).unwrap();
+        let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        let send_to = wallet.get_address(AddressIndex::New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.add_recipient(send_to.script_pubkey(), 10_000);
+        let (mut psbt, _) = builder.finish().unwrap();
+        psbt.inputs.push(psbt_bip.inputs[0].clone());
+        let options = SignOptions {
+            trust_witness_utxo: true,
+            ..Default::default()
+        };
+        let _ = wallet.sign(&mut psbt, options).unwrap();
+    }
+
+    #[test]
+    #[should_panic(expected = "InputIndexOutOfRange")]
+    fn test_psbt_malformed_psbt_input_segwit() {
+        let psbt_bip = Psbt::from_str(PSBT_STR).unwrap();
+        let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        let send_to = wallet.get_address(AddressIndex::New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.add_recipient(send_to.script_pubkey(), 10_000);
+        let (mut psbt, _) = builder.finish().unwrap();
+        psbt.inputs.push(psbt_bip.inputs[1].clone());
+        let options = SignOptions {
+            trust_witness_utxo: true,
+            ..Default::default()
+        };
+        let _ = wallet.sign(&mut psbt, options).unwrap();
+    }
+
+    #[test]
+    #[should_panic(expected = "InputIndexOutOfRange")]
+    fn test_psbt_malformed_tx_input() {
+        let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        let send_to = wallet.get_address(AddressIndex::New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.add_recipient(send_to.script_pubkey(), 10_000);
+        let (mut psbt, _) = builder.finish().unwrap();
+        psbt.unsigned_tx.input.push(TxIn::default());
+        let options = SignOptions {
+            trust_witness_utxo: true,
+            ..Default::default()
+        };
+        let _ = wallet.sign(&mut psbt, options).unwrap();
+    }
+
+    #[test]
+    fn test_psbt_sign_with_finalized() {
+        let psbt_bip = Psbt::from_str(PSBT_STR).unwrap();
+        let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        let send_to = wallet.get_address(AddressIndex::New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.add_recipient(send_to.script_pubkey(), 10_000);
+        let (mut psbt, _) = builder.finish().unwrap();
+
+        // add a finalized input
+        psbt.inputs.push(psbt_bip.inputs[0].clone());
+        psbt.unsigned_tx
+            .input
+            .push(psbt_bip.unsigned_tx.input[0].clone());
+
+        let _ = wallet.sign(&mut psbt, SignOptions::default()).unwrap();
+    }
+}

src/testutils/configurable_blockchain_tests.rs

@@ -0,0 +1,140 @@
+use bitcoin::Network;
+
+use crate::{
+    blockchain::ConfigurableBlockchain, database::MemoryDatabase, testutils, wallet::AddressIndex,
+    Wallet,
+};
+
+use super::blockchain_tests::TestClient;
+
+/// Trait for testing [`ConfigurableBlockchain`] implementations.
+pub trait ConfigurableBlockchainTester<B: ConfigurableBlockchain>: Sized {
+    /// Blockchain name for logging.
+    const BLOCKCHAIN_NAME: &'static str;
+
+    /// Generates a blockchain config with a given stop_gap.
+    ///
+    /// If this returns [`Option::None`], then the associated tests will not run.
+    fn config_with_stop_gap(
+        &self,
+        _test_client: &mut TestClient,
+        _stop_gap: usize,
+    ) -> Option<B::Config> {
+        None
+    }
+
+    /// Runs all avaliable tests.
+    fn run(&self) {
+        let test_client = &mut TestClient::default();
+
+        if self.config_with_stop_gap(test_client, 0).is_some() {
+            test_wallet_sync_with_stop_gaps(test_client, self);
+        } else {
+            println!(
+                "{}: Skipped tests requiring config_with_stop_gap.",
+                Self::BLOCKCHAIN_NAME
+            );
+        }
+    }
+}
+
+/// Test whether blockchain implementation syncs with expected behaviour given different `stop_gap`
+/// parameters.
+///
+/// For each test vector:
+/// * Fill wallet's derived addresses with balances (as specified by test vector).
+///    * [0..addrs_before]          => 1000sats for each address
+///    * [addrs_before..actual_gap] => empty addresses
+///    * [actual_gap..addrs_after]  => 1000sats for each address
+/// * Then, perform wallet sync and obtain wallet balance
+/// * Check balance is within expected range (we can compare `stop_gap` and `actual_gap` to
+///    determine this).
+fn test_wallet_sync_with_stop_gaps<T, B>(test_client: &mut TestClient, tester: &T)
+where
+    T: ConfigurableBlockchainTester<B>,
+    B: ConfigurableBlockchain,
+{
+    // Generates wallet descriptor
+    let descriptor_of_account = |account_index: usize| -> String {
+        format!("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/{account_index}/*)")
+    };
+
+    // Amount (in satoshis) provided to a single address (which expects to have a balance)
+    const AMOUNT_PER_TX: u64 = 1000;
+
+    // [stop_gap, actual_gap, addrs_before, addrs_after]
+    //
+    // [0]     stop_gap: Passed to [`ElectrumBlockchainConfig`]
+    // [1]   actual_gap: Range size of address indexes without a balance
+    // [2] addrs_before: Range size of address indexes (before gap) which contains a balance
+    // [3]  addrs_after: Range size of address indexes (after gap) which contains a balance
+    let test_vectors: Vec<[u64; 4]> = vec![
+        [0, 0, 0, 5],
+        [0, 0, 5, 5],
+        [0, 1, 5, 5],
+        [0, 2, 5, 5],
+        [1, 0, 5, 5],
+        [1, 1, 5, 5],
+        [1, 2, 5, 5],
+        [2, 1, 5, 5],
+        [2, 2, 5, 5],
+        [2, 3, 5, 5],
+    ];
+
+    for (account_index, vector) in test_vectors.into_iter().enumerate() {
+        let [stop_gap, actual_gap, addrs_before, addrs_after] = vector;
+        let descriptor = descriptor_of_account(account_index);
+
+        let blockchain = B::from_config(
+            &tester
+                .config_with_stop_gap(test_client, stop_gap as _)
+                .unwrap(),
+        )
+        .unwrap();
+
+        let wallet =
+            Wallet::new(&descriptor, None, Network::Regtest, MemoryDatabase::new()).unwrap();
+
+        // fill server-side with txs to specified address indexes
+        // return the max balance of the wallet (also the actual balance)
+        let max_balance = (0..addrs_before)
+            .chain(addrs_before + actual_gap..addrs_before + actual_gap + addrs_after)
+            .fold(0_u64, |sum, i| {
+                let address = wallet.get_address(AddressIndex::Peek(i as _)).unwrap();
+                test_client.receive(testutils! {
+                    @tx ( (@addr address.address) => AMOUNT_PER_TX )
+                });
+                sum + AMOUNT_PER_TX
+            });
+
+        // minimum allowed balance of wallet (based on stop gap)
+        let min_balance = if actual_gap > stop_gap {
+            addrs_before * AMOUNT_PER_TX
+        } else {
+            max_balance
+        };
+
+        // perform wallet sync
+        wallet.sync(&blockchain, Default::default()).unwrap();
+
+        let wallet_balance = wallet.get_balance().unwrap();
+
+        let details = format!(
+            "test_vector: [stop_gap: {}, actual_gap: {}, addrs_before: {}, addrs_after: {}]",
+            stop_gap, actual_gap, addrs_before, addrs_after,
+        );
+        assert!(
+            wallet_balance <= max_balance,
+            "wallet balance is greater than received amount: {}",
+            details
+        );
+        assert!(
+            wallet_balance >= min_balance,
+            "wallet balance is smaller than expected: {}",
+            details
+        );
+
+        // generate block to confirm new transactions
+        test_client.generate(1, None);
+    }
+}

src/testutils/mod.rs

@@ -0,0 +1,236 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+#![allow(missing_docs)]
+
+#[cfg(test)]
+#[cfg(feature = "test-blockchains")]
+pub mod blockchain_tests;
+
+#[cfg(test)]
+#[cfg(feature = "test-blockchains")]
+pub mod configurable_blockchain_tests;
+
+use bitcoin::{Address, Txid};
+
+#[derive(Clone, Debug)]
+pub struct TestIncomingInput {
+    pub txid: Txid,
+    pub vout: u32,
+    pub sequence: Option<u32>,
+}
+
+impl TestIncomingInput {
+    pub fn new(txid: Txid, vout: u32, sequence: Option<u32>) -> Self {
+        Self {
+            txid,
+            vout,
+            sequence,
+        }
+    }
+
+    #[cfg(feature = "test-blockchains")]
+    pub fn into_raw_tx_input(self) -> bitcoincore_rpc::json::CreateRawTransactionInput {
+        bitcoincore_rpc::json::CreateRawTransactionInput {
+            txid: self.txid,
+            vout: self.vout,
+            sequence: self.sequence,
+        }
+    }
+}
+
+#[derive(Clone, Debug)]
+pub struct TestIncomingOutput {
+    pub value: u64,
+    pub to_address: String,
+}
+
+impl TestIncomingOutput {
+    pub fn new(value: u64, to_address: Address) -> Self {
+        Self {
+            value,
+            to_address: to_address.to_string(),
+        }
+    }
+}
+
+#[derive(Clone, Debug)]
+pub struct TestIncomingTx {
+    pub input: Vec<TestIncomingInput>,
+    pub output: Vec<TestIncomingOutput>,
+    pub min_confirmations: Option<u64>,
+    pub locktime: Option<i64>,
+    pub replaceable: Option<bool>,
+}
+
+impl TestIncomingTx {
+    pub fn new(
+        input: Vec<TestIncomingInput>,
+        output: Vec<TestIncomingOutput>,
+        min_confirmations: Option<u64>,
+        locktime: Option<i64>,
+        replaceable: Option<bool>,
+    ) -> Self {
+        Self {
+            input,
+            output,
+            min_confirmations,
+            locktime,
+            replaceable,
+        }
+    }
+
+    pub fn add_input(&mut self, input: TestIncomingInput) {
+        self.input.push(input);
+    }
+
+    pub fn add_output(&mut self, output: TestIncomingOutput) {
+        self.output.push(output);
+    }
+}
+
+#[doc(hidden)]
+#[macro_export]
+macro_rules! testutils {
+    ( @external $descriptors:expr, $child:expr ) => ({
+        use $crate::bitcoin::secp256k1::Secp256k1;
+        use $crate::miniscript::descriptor::{Descriptor, DescriptorPublicKey, DescriptorTrait};
+
+        use $crate::descriptor::AsDerived;
+
+        let secp = Secp256k1::new();
+
+        let parsed = Descriptor::<DescriptorPublicKey>::parse_descriptor(&secp, &$descriptors.0).expect("Failed to parse descriptor in `testutils!(@external)`").0;
+        parsed.as_derived($child, &secp).address(bitcoin::Network::Regtest).expect("No address form")
+    });
+    ( @internal $descriptors:expr, $child:expr ) => ({
+        use $crate::bitcoin::secp256k1::Secp256k1;
+        use $crate::miniscript::descriptor::{Descriptor, DescriptorPublicKey, DescriptorTrait};
+
+        use $crate::descriptor::AsDerived;
+
+        let secp = Secp256k1::new();
+
+        let parsed = Descriptor::<DescriptorPublicKey>::parse_descriptor(&secp, &$descriptors.1.expect("Missing internal descriptor")).expect("Failed to parse descriptor in `testutils!(@internal)`").0;
+        parsed.as_derived($child, &secp).address($crate::bitcoin::Network::Regtest).expect("No address form")
+    });
+    ( @e $descriptors:expr, $child:expr ) => ({ testutils!(@external $descriptors, $child) });
+    ( @i $descriptors:expr, $child:expr ) => ({ testutils!(@internal $descriptors, $child) });
+    ( @addr $addr:expr ) => ({ $addr });
+
+    ( @tx ( $( ( $( $addr:tt )* ) => $amount:expr ),+ ) $( ( @inputs $( ($txid:expr, $vout:expr) ),+ ) )? $( ( @locktime $locktime:expr ) )? $( ( @confirmations $confirmations:expr ) )? $( ( @replaceable $replaceable:expr ) )? ) => ({
+        let outs = vec![$( $crate::testutils::TestIncomingOutput::new($amount, testutils!( $($addr)* ))),+];
+        let _ins: Vec<$crate::testutils::TestIncomingInput> = vec![];
+        $(
+            let _ins = vec![$( $crate::testutils::TestIncomingInput { txid: $txid, vout: $vout, sequence: None }),+];
+        )?
+
+        let locktime = None::<i64>$(.or(Some($locktime)))?;
+
+        let min_confirmations = None::<u64>$(.or(Some($confirmations)))?;
+        let replaceable = None::<bool>$(.or(Some($replaceable)))?;
+
+        $crate::testutils::TestIncomingTx::new(_ins, outs, min_confirmations, locktime, replaceable)
+    });
+
+    ( @literal $key:expr ) => ({
+        let key = $key.to_string();
+        (key, None::<String>, None::<String>)
+    });
+    ( @generate_xprv $( $external_path:expr )? $( ,$internal_path:expr )? ) => ({
+        use rand::Rng;
+
+        let mut seed = [0u8; 32];
+        rand::thread_rng().fill(&mut seed[..]);
+
+        let key = $crate::bitcoin::util::bip32::ExtendedPrivKey::new_master(
+            $crate::bitcoin::Network::Testnet,
+            &seed,
+        );
+
+        let external_path = None::<String>$(.or(Some($external_path.to_string())))?;
+        let internal_path = None::<String>$(.or(Some($internal_path.to_string())))?;
+
+        (key.unwrap().to_string(), external_path, internal_path)
+    });
+    ( @generate_wif ) => ({
+        use rand::Rng;
+
+        let mut key = [0u8; $crate::bitcoin::secp256k1::constants::SECRET_KEY_SIZE];
+        rand::thread_rng().fill(&mut key[..]);
+
+        ($crate::bitcoin::PrivateKey {
+            compressed: true,
+            network: $crate::bitcoin::Network::Testnet,
+            key: $crate::bitcoin::secp256k1::SecretKey::from_slice(&key).unwrap(),
+        }.to_string(), None::<String>, None::<String>)
+    });
+
+    ( @keys ( $( $alias:expr => ( $( $key_type:tt )* ) ),+ ) ) => ({
+        let mut map = std::collections::HashMap::new();
+        $(
+            let alias: &str = $alias;
+            map.insert(alias, testutils!( $($key_type)* ));
+        )+
+
+        map
+    });
+
+    ( @descriptors ( $external_descriptor:expr ) $( ( $internal_descriptor:expr ) )? $( ( @keys $( $keys:tt )* ) )* ) => ({
+        use std::str::FromStr;
+        use std::collections::HashMap;
+        use $crate::miniscript::descriptor::Descriptor;
+        use $crate::miniscript::TranslatePk;
+
+        #[allow(unused_assignments, unused_mut)]
+        let mut keys: HashMap<&'static str, (String, Option<String>, Option<String>)> = HashMap::new();
+        $(
+            keys = testutils!{ @keys $( $keys )* };
+        )*
+
+        let external: Descriptor<String> = FromStr::from_str($external_descriptor).unwrap();
+        let external: Descriptor<String> = external.translate_pk_infallible::<_, _>(|k| {
+            if let Some((key, ext_path, _)) = keys.get(&k.as_str()) {
+                format!("{}{}", key, ext_path.as_ref().unwrap_or(&"".into()))
+            } else {
+                k.clone()
+            }
+        }, |kh| {
+            if let Some((key, ext_path, _)) = keys.get(&kh.as_str()) {
+                format!("{}{}", key, ext_path.as_ref().unwrap_or(&"".into()))
+            } else {
+                kh.clone()
+            }
+
+        });
+        let external = external.to_string();
+
+        let internal = None::<String>$(.or({
+            let string_internal: Descriptor<String> = FromStr::from_str($internal_descriptor).unwrap();
+
+            let string_internal: Descriptor<String> = string_internal.translate_pk_infallible::<_, _>(|k| {
+                if let Some((key, _, int_path)) = keys.get(&k.as_str()) {
+                    format!("{}{}", key, int_path.as_ref().unwrap_or(&"".into()))
+                } else {
+                    k.clone()
+                }
+            }, |kh| {
+                if let Some((key, _, int_path)) = keys.get(&kh.as_str()) {
+                    format!("{}{}", key, int_path.as_ref().unwrap_or(&"".into()))
+                } else {
+                    kh.clone()
+                }
+            });
+            Some(string_internal.to_string())
+        }))?;
+
+        (external, internal)
+    })
+}

src/types.rs

@@ -1,31 +1,19 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 use std::convert::AsRef;
+use std::ops::Sub;
 
 use bitcoin::blockdata::transaction::{OutPoint, Transaction, TxOut};
-use bitcoin::hash_types::Txid;
+use bitcoin::{hash_types::Txid, util::psbt};
 
 use serde::{Deserialize, Serialize};
 
@@ -69,19 +57,40 @@ impl FeeRate {
     }
 
     /// Create a new instance of [`FeeRate`] given a float fee rate in satoshi/vbyte
-    pub fn from_sat_per_vb(sat_per_vb: f32) -> Self {
+    pub const fn from_sat_per_vb(sat_per_vb: f32) -> Self {
         FeeRate(sat_per_vb)
     }
 
     /// Create a new [`FeeRate`] with the default min relay fee value
-    pub fn default_min_relay_fee() -> Self {
+    pub const fn default_min_relay_fee() -> Self {
         FeeRate(1.0)
     }
 
+    /// Calculate fee rate from `fee` and weight units (`wu`).
+    pub fn from_wu(fee: u64, wu: usize) -> FeeRate {
+        Self::from_vb(fee, wu.vbytes())
+    }
+
+    /// Calculate fee rate from `fee` and `vbytes`.
+    pub fn from_vb(fee: u64, vbytes: usize) -> FeeRate {
+        let rate = fee as f32 / vbytes as f32;
+        Self::from_sat_per_vb(rate)
+    }
+
     /// Return the value as satoshi/vbyte
     pub fn as_sat_vb(&self) -> f32 {
         self.0
     }
+
+    /// Calculate absolute fee in Satoshis using size in weight units.
+    pub fn fee_wu(&self, wu: usize) -> u64 {
+        self.fee_vb(wu.vbytes())
+    }
+
+    /// Calculate absolute fee in Satoshis using size in virtual bytes.
+    pub fn fee_vb(&self, vbytes: usize) -> u64 {
+        (self.as_sat_vb() * vbytes as f32).ceil() as u64
+    }
 }
 
 impl std::default::Default for FeeRate {
@@ -90,15 +99,98 @@ impl std::default::Default for FeeRate {
     }
 }
 
-/// A wallet unspent output
-#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
-pub struct UTXO {
+impl Sub for FeeRate {
+    type Output = Self;
+
+    fn sub(self, other: FeeRate) -> Self::Output {
+        FeeRate(self.0 - other.0)
+    }
+}
+
+/// Trait implemented by types that can be used to measure weight units.
+pub trait Vbytes {
+    /// Convert weight units to virtual bytes.
+    fn vbytes(self) -> usize;
+}
+
+impl Vbytes for usize {
+    fn vbytes(self) -> usize {
+        // ref: https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki#transaction-size-calculations
+        (self as f32 / 4.0).ceil() as usize
+    }
+}
+
+/// An unspent output owned by a [`Wallet`].
+///
+/// [`Wallet`]: crate::Wallet
+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, Hash)]
+pub struct LocalUtxo {
     /// Reference to a transaction output
     pub outpoint: OutPoint,
     /// Transaction output
     pub txout: TxOut,
     /// Type of keychain
     pub keychain: KeychainKind,
+    /// Whether this UTXO is spent or not
+    pub is_spent: bool,
+}
+
+/// A [`Utxo`] with its `satisfaction_weight`.
+#[derive(Debug, Clone, PartialEq)]
+pub struct WeightedUtxo {
+    /// The weight of the witness data and `scriptSig` expressed in [weight units]. This is used to
+    /// properly maintain the feerate when adding this input to a transaction during coin selection.
+    ///
+    /// [weight units]: https://en.bitcoin.it/wiki/Weight_units
+    pub satisfaction_weight: usize,
+    /// The UTXO
+    pub utxo: Utxo,
+}
+
+#[derive(Debug, Clone, PartialEq)]
+/// An unspent transaction output (UTXO).
+pub enum Utxo {
+    /// A UTXO owned by the local wallet.
+    Local(LocalUtxo),
+    /// A UTXO owned by another wallet.
+    Foreign {
+        /// The location of the output.
+        outpoint: OutPoint,
+        /// The information about the input we require to add it to a PSBT.
+        // Box it to stop the type being too big.
+        psbt_input: Box<psbt::Input>,
+    },
+}
+
+impl Utxo {
+    /// Get the location of the UTXO
+    pub fn outpoint(&self) -> OutPoint {
+        match &self {
+            Utxo::Local(local) => local.outpoint,
+            Utxo::Foreign { outpoint, .. } => *outpoint,
+        }
+    }
+
+    /// Get the `TxOut` of the UTXO
+    pub fn txout(&self) -> &TxOut {
+        match &self {
+            Utxo::Local(local) => &local.txout,
+            Utxo::Foreign {
+                outpoint,
+                psbt_input,
+            } => {
+                if let Some(prev_tx) = &psbt_input.non_witness_utxo {
+                    return &prev_tx.output[outpoint.vout as usize];
+                }
+
+                if let Some(txout) = &psbt_input.witness_utxo {
+                    return txout;
+                }
+
+                unreachable!("Foreign UTXOs will always have one of these set")
+            }
+        }
+    }
 }
 
 /// A wallet transaction
@@ -108,14 +200,55 @@ pub struct TransactionDetails {
     pub transaction: Option<Transaction>,
     /// Transaction id
     pub txid: Txid,
-    /// Timestamp
-    pub timestamp: u64,
+
     /// Received value (sats)
+    /// Sum of owned outputs of this transaction.
     pub received: u64,
     /// Sent value (sats)
+    /// Sum of owned inputs of this transaction.
     pub sent: u64,
-    /// Fee value (sats)
-    pub fees: u64,
-    /// Confirmed in block height, `None` means unconfirmed
-    pub height: Option<u32>,
+    /// Fee value (sats) if available.
+    /// The availability of the fee depends on the backend. It's never `None` with an Electrum
+    /// Server backend, but it could be `None` with a Bitcoin RPC node without txindex that receive
+    /// funds while offline.
+    pub fee: Option<u64>,
+    /// If the transaction is confirmed, contains height and timestamp of the block containing the
+    /// transaction, unconfirmed transaction contains `None`.
+    pub confirmation_time: Option<BlockTime>,
+}
+
+/// Block height and timestamp of a block
+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, Default)]
+pub struct BlockTime {
+    /// confirmation block height
+    pub height: u32,
+    /// confirmation block timestamp
+    pub timestamp: u64,
+}
+
+/// **DEPRECATED**: Confirmation time of a transaction
+///
+/// The structure has been renamed to `BlockTime`
+#[deprecated(note = "This structure has been renamed to `BlockTime`")]
+pub type ConfirmationTime = BlockTime;
+
+impl BlockTime {
+    /// Returns `Some` `BlockTime` if both `height` and `timestamp` are `Some`
+    pub fn new(height: Option<u32>, timestamp: Option<u64>) -> Option<Self> {
+        match (height, timestamp) {
+            (Some(height), Some(timestamp)) => Some(BlockTime { height, timestamp }),
+            _ => None,
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn can_store_feerate_in_const() {
+        const _MY_RATE: FeeRate = FeeRate::from_sat_per_vb(10.0);
+        const _MIN_RELAY: FeeRate = FeeRate::default_min_relay_fee();
+    }
 }

src/wallet/address_validator.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Address validation callbacks
 //!
@@ -33,7 +20,7 @@
 //! An address validator can be attached to a [`Wallet`](super::Wallet) by using the
 //! [`Wallet::add_address_validator`](super::Wallet::add_address_validator) method, and
 //! whenever a new address is generated (either explicitly by the user with
-//! [`Wallet::get_new_address`](super::Wallet::get_new_address) or internally to create a change
+//! [`Wallet::get_address`](super::Wallet::get_address) or internally to create a change
 //! address) all the attached validators will be polled, in sequence. All of them must complete
 //! successfully to continue.
 //!
@@ -45,6 +32,7 @@
 //! # use bdk::address_validator::*;
 //! # use bdk::database::*;
 //! # use bdk::*;
+//! # use bdk::wallet::AddressIndex::New;
 //! #[derive(Debug)]
 //! struct PrintAddressAndContinue;
 //!
@@ -52,7 +40,7 @@
 //!     fn validate(
 //!         &self,
 //!         keychain: KeychainKind,
-//!         hd_keypaths: &HDKeyPaths,
+//!         hd_keypaths: &HdKeyPaths,
 //!         script: &Script
 //!     ) -> Result<(), AddressValidatorError> {
 //!         let address = Address::from_script(script, Network::Testnet)
@@ -67,10 +55,10 @@
 //! }
 //!
 //! let descriptor = "wpkh(tpubD6NzVbkrYhZ4Xferm7Pz4VnjdcDPFyjVu5K4iZXQ4pVN8Cks4pHVowTBXBKRhX64pkRyJZJN5xAKj4UDNnLPb5p2sSKXhewoYx5GbTdUFWq/*)";
-//! let mut wallet = Wallet::new_offline(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
+//! let mut wallet = Wallet::new(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
 //! wallet.add_address_validator(Arc::new(PrintAddressAndContinue));
 //!
-//! let address = wallet.get_new_address()?;
+//! let address = wallet.get_address(New)?;
 //! println!("Address: {}", address);
 //! # Ok::<(), bdk::Error>(())
 //! ```
@@ -79,7 +67,7 @@ use std::fmt;
 
 use bitcoin::Script;
 
-use crate::descriptor::HDKeyPaths;
+use crate::descriptor::HdKeyPaths;
 use crate::types::KeychainKind;
 
 /// Errors that can be returned to fail the validation of an address
@@ -112,12 +100,13 @@ impl std::error::Error for AddressValidatorError {}
 /// validator will be propagated up to the original caller that triggered the address generation.
 ///
 /// For a usage example see [this module](crate::address_validator)'s documentation.
+#[deprecated = "AddressValidator was rarely used. Address validation can occur outside of BDK"]
 pub trait AddressValidator: Send + Sync + fmt::Debug {
     /// Validate or inspect an address
     fn validate(
         &self,
         keychain: KeychainKind,
-        hd_keypaths: &HDKeyPaths,
+        hd_keypaths: &HdKeyPaths,
         script: &Script,
     ) -> Result<(), AddressValidatorError>;
 }
@@ -127,15 +116,17 @@ mod test {
     use std::sync::Arc;
 
     use super::*;
-    use crate::wallet::test::{get_funded_wallet, get_test_wpkh};
+    use crate::wallet::AddressIndex::New;
+    use crate::wallet::{get_funded_wallet, test::get_test_wpkh};
 
     #[derive(Debug)]
     struct TestValidator;
+    #[allow(deprecated)]
     impl AddressValidator for TestValidator {
         fn validate(
             &self,
             _keychain: KeychainKind,
-            _hd_keypaths: &HDKeyPaths,
+            _hd_keypaths: &HdKeyPaths,
             _script: &bitcoin::Script,
         ) -> Result<(), AddressValidatorError> {
             Err(AddressValidatorError::InvalidScript)
@@ -146,18 +137,20 @@ mod test {
     #[should_panic(expected = "InvalidScript")]
     fn test_address_validator_external() {
         let (mut wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        #[allow(deprecated)]
         wallet.add_address_validator(Arc::new(TestValidator));
 
-        wallet.get_new_address().unwrap();
+        wallet.get_address(New).unwrap();
     }
 
     #[test]
     #[should_panic(expected = "InvalidScript")]
     fn test_address_validator_internal() {
         let (mut wallet, descriptors, _) = get_funded_wallet(get_test_wpkh());
+        #[allow(deprecated)]
         wallet.add_address_validator(Arc::new(TestValidator));
 
-        let addr = testutils!(@external descriptors, 10);
+        let addr = crate::testutils!(@external descriptors, 10);
         let mut builder = wallet.build_tx();
         builder.add_recipient(addr.script_pubkey(), 25_000);
         builder.finish().unwrap();

src/wallet/export.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Wallet export
 //!
@@ -42,8 +29,8 @@
 //!     "label":"testnet"
 //! }"#;
 //!
-//! let import = WalletExport::from_str(import)?;
-//! let wallet = Wallet::new_offline(
+//! let import = FullyNodedExport::from_str(import)?;
+//! let wallet = Wallet::new(
 //!     &import.descriptor(),
 //!     import.change_descriptor().as_ref(),
 //!     Network::Testnet,
@@ -58,13 +45,13 @@
 //! # use bdk::database::*;
 //! # use bdk::wallet::export::*;
 //! # use bdk::*;
-//! let wallet = Wallet::new_offline(
+//! let wallet = Wallet::new(
 //!     "wpkh([c258d2e4/84h/1h/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe/0/*)",
 //!     Some("wpkh([c258d2e4/84h/1h/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe/1/*)"),
 //!     Network::Testnet,
 //!     MemoryDatabase::default()
 //! )?;
-//! let export = WalletExport::export_wallet(&wallet, "exported wallet", true)
+//! let export = FullyNodedExport::export_wallet(&wallet, "exported wallet", true)
 //!     .map_err(ToString::to_string)
 //!     .map_err(bdk::Error::Generic)?;
 //!
@@ -77,16 +64,21 @@ use std::str::FromStr;
 use serde::{Deserialize, Serialize};
 
 use miniscript::descriptor::{ShInner, WshInner};
-use miniscript::{Descriptor, DescriptorPublicKey, ScriptContext, Terminal};
+use miniscript::{Descriptor, ScriptContext, Terminal};
 
 use crate::database::BatchDatabase;
+use crate::types::KeychainKind;
 use crate::wallet::Wallet;
 
+/// Alias for [`FullyNodedExport`]
+#[deprecated(since = "0.18.0", note = "Please use [`FullyNodedExport`] instead")]
+pub type WalletExport = FullyNodedExport;
+
 /// Structure that contains the export of a wallet
 ///
 /// For a usage example see [this module](crate::wallet::export)'s documentation.
 #[derive(Debug, Serialize, Deserialize)]
-pub struct WalletExport {
+pub struct FullyNodedExport {
     descriptor: String,
     /// Earliest block to rescan when looking for the wallet's transactions
     pub blockheight: u32,
@@ -94,13 +86,13 @@ pub struct WalletExport {
     pub label: String,
 }
 
-impl ToString for WalletExport {
+impl ToString for FullyNodedExport {
     fn to_string(&self) -> String {
         serde_json::to_string(self).unwrap()
     }
 }
 
-impl FromStr for WalletExport {
+impl FromStr for FullyNodedExport {
     type Err = serde_json::Error;
 
     fn from_str(s: &str) -> Result<Self, Self::Err> {
@@ -109,10 +101,10 @@ impl FromStr for WalletExport {
 }
 
 fn remove_checksum(s: String) -> String {
-    s.splitn(2, '#').next().map(String::from).unwrap()
+    s.split_once('#').map(|(a, _)| String::from(a)).unwrap()
 }
 
-impl WalletExport {
+impl FullyNodedExport {
     /// Export a wallet
     ///
     /// This function returns an error if it determines that the `wallet`'s descriptor(s) are not
@@ -124,14 +116,18 @@ impl WalletExport {
     ///
     /// If the database is empty or `include_blockheight` is false, the `blockheight` field
     /// returned will be `0`.
-    pub fn export_wallet<B, D: BatchDatabase>(
-        wallet: &Wallet<B, D>,
+    pub fn export_wallet<D: BatchDatabase>(
+        wallet: &Wallet<D>,
         label: &str,
         include_blockheight: bool,
     ) -> Result<Self, &'static str> {
         let descriptor = wallet
-            .descriptor
-            .to_string_with_secret(&wallet.signers.as_key_map(wallet.secp_ctx()));
+            .get_descriptor_for_keychain(KeychainKind::External)
+            .to_string_with_secret(
+                &wallet
+                    .get_signers(KeychainKind::External)
+                    .as_key_map(wallet.secp_ctx()),
+            );
         let descriptor = remove_checksum(descriptor);
         Self::is_compatible_with_core(&descriptor)?;
 
@@ -141,7 +137,7 @@ impl WalletExport {
             Ok(txs) => {
                 let mut heights = txs
                     .into_iter()
-                    .map(|tx| tx.height.unwrap_or(0))
+                    .map(|tx| tx.confirmation_time.map(|c| c.height).unwrap_or(0))
                     .collect::<Vec<_>>();
                 heights.sort_unstable();
 
@@ -149,18 +145,30 @@ impl WalletExport {
             }
         };
 
-        let export = WalletExport {
+        let export = FullyNodedExport {
             descriptor,
             label: label.into(),
             blockheight,
         };
 
-        let desc_to_string = |d: &Descriptor<DescriptorPublicKey>| {
-            let descriptor =
-                d.to_string_with_secret(&wallet.change_signers.as_key_map(wallet.secp_ctx()));
-            remove_checksum(descriptor)
+        let change_descriptor = match wallet
+            .public_descriptor(KeychainKind::Internal)
+            .map_err(|_| "Invalid change descriptor")?
+            .is_some()
+        {
+            false => None,
+            true => {
+                let descriptor = wallet
+                    .get_descriptor_for_keychain(KeychainKind::Internal)
+                    .to_string_with_secret(
+                        &wallet
+                            .get_signers(KeychainKind::Internal)
+                            .as_key_map(wallet.secp_ctx()),
+                    );
+                Some(remove_checksum(descriptor))
+            }
         };
-        if export.change_descriptor() != wallet.change_descriptor.as_ref().map(desc_to_string) {
+        if export.change_descriptor() != change_descriptor {
             return Err("Incompatible change descriptor");
         }
 
@@ -225,6 +233,7 @@ mod test {
     use crate::database::{memory::MemoryDatabase, BatchOperations};
     use crate::types::TransactionDetails;
     use crate::wallet::Wallet;
+    use crate::BlockTime;
 
     fn get_test_db() -> MemoryDatabase {
         let mut db = MemoryDatabase::new();
@@ -234,11 +243,14 @@ mod test {
                 "4ddff1fa33af17f377f62b72357b43107c19110a8009b36fb832af505efed98a",
             )
             .unwrap(),
-            timestamp: 12345678,
+
             received: 100_000,
             sent: 0,
-            fees: 500,
-            height: Some(5000),
+            fee: Some(500),
+            confirmation_time: Some(BlockTime {
+                timestamp: 12345678,
+                height: 5000,
+            }),
         })
         .unwrap();
 
@@ -250,14 +262,14 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
             get_test_db(),
         )
         .unwrap();
-        let export = WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
 
         assert_eq!(export.descriptor(), descriptor);
         assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));
@@ -274,9 +286,8 @@ mod test {
 
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
 
-        let wallet =
-            Wallet::new_offline(descriptor, None, Network::Bitcoin, get_test_db()).unwrap();
-        WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let wallet = Wallet::new(descriptor, None, Network::Bitcoin, get_test_db()).unwrap();
+        FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
     }
 
     #[test]
@@ -288,14 +299,14 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/50'/0'/1/*)";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
             get_test_db(),
         )
         .unwrap();
-        WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
     }
 
     #[test]
@@ -311,14 +322,14 @@ mod test {
                                        [c98b1535/48'/0'/0'/2']tpubDCDi5W4sP6zSnzJeowy8rQDVhBdRARaPhK1axABi8V1661wEPeanpEXj4ZLAUEoikVtoWcyK26TKKJSecSfeKxwHCcRrge9k1ybuiL71z4a/1/*\
                                  ))";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Testnet,
             get_test_db(),
         )
         .unwrap();
-        let export = WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
 
         assert_eq!(export.descriptor(), descriptor);
         assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));
@@ -331,14 +342,14 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
             get_test_db(),
         )
         .unwrap();
-        let export = WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
 
         assert_eq!(export.to_string(), "{\"descriptor\":\"wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44\'/0\'/0\'/0/*)\",\"blockheight\":5000,\"label\":\"Test Label\"}");
     }
@@ -349,7 +360,7 @@ mod test {
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
         let import_str = "{\"descriptor\":\"wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44\'/0\'/0\'/0/*)\",\"blockheight\":5000,\"label\":\"Test Label\"}";
-        let export = WalletExport::from_str(import_str).unwrap();
+        let export = FullyNodedExport::from_str(import_str).unwrap();
 
         assert_eq!(export.descriptor(), descriptor);
         assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));

src/wallet/time.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Cross-platform time
 //!

src/wallet/tx_builder.rs

@@ -1,26 +1,13 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
 //! Transaction builder
 //!
@@ -54,15 +41,15 @@ use std::collections::HashSet;
 use std::default::Default;
 use std::marker::PhantomData;
 
-use bitcoin::util::psbt::PartiallySignedTransaction as PSBT;
-use bitcoin::{OutPoint, Script, SigHashType, Transaction};
+use bitcoin::util::psbt::{self, PartiallySignedTransaction as Psbt};
+use bitcoin::{OutPoint, Script, Transaction};
 
 use miniscript::descriptor::DescriptorTrait;
 
 use super::coin_selection::{CoinSelectionAlgorithm, DefaultCoinSelectionAlgorithm};
-use crate::{database::BatchDatabase, Error, Wallet};
+use crate::{database::BatchDatabase, Error, Utxo, Wallet};
 use crate::{
-    types::{FeeRate, KeychainKind, UTXO},
+    types::{FeeRate, KeychainKind, LocalUtxo, WeightedUtxo},
     TransactionDetails,
 };
 /// Context in which the [`TxBuilder`] is valid
@@ -100,9 +87,9 @@ impl TxBuilderContext for BumpFee {}
 /// let (psbt1, details) = {
 ///     let mut builder = wallet.build_tx();
 ///     builder
-///        .ordering(TxOrdering::Untouched)
-///        .add_recipient(addr1.script_pubkey(), 50_000)
-///        .add_recipient(addr2.script_pubkey(), 50_000);
+///         .ordering(TxOrdering::Untouched)
+///         .add_recipient(addr1.script_pubkey(), 50_000)
+///         .add_recipient(addr2.script_pubkey(), 50_000);
 ///     builder.finish()?
 /// };
 ///
@@ -116,7 +103,7 @@ impl TxBuilderContext for BumpFee {}
 ///     builder.finish()?
 /// };
 ///
-/// assert_eq!(psbt1.global.unsigned_tx.output[..2], psbt2.global.unsigned_tx.output[..2]);
+/// assert_eq!(psbt1.unsigned_tx.output[..2], psbt2.unsigned_tx.output[..2]);
 /// # Ok::<(), bdk::Error>(())
 /// ```
 ///
@@ -129,12 +116,9 @@ impl TxBuilderContext for BumpFee {}
 /// [`build_fee_bump`]: Wallet::build_fee_bump
 /// [`finish`]: Self::finish
 /// [`coin_selection`]: Self::coin_selection
-#[derive(Clone, Debug)]
-pub struct TxBuilder<'a, B, D, Cs, Ctx> {
-    pub(crate) wallet: &'a Wallet<B, D>,
-    // params and coin_selection are Options not becasue they are optionally set (they are always
-    // there) but because `.finish()` uses `Option::take` to get an owned value from a &mut self.
-    // They are only `None` after `.finish()` is called.
+#[derive(Debug)]
+pub struct TxBuilder<'a, D, Cs, Ctx> {
+    pub(crate) wallet: &'a Wallet<D>,
     pub(crate) params: TxParams,
     pub(crate) coin_selection: Cs,
     pub(crate) phantom: PhantomData<Ctx>,
@@ -146,23 +130,24 @@ pub struct TxBuilder<'a, B, D, Cs, Ctx> {
 pub(crate) struct TxParams {
     pub(crate) recipients: Vec<(Script, u64)>,
     pub(crate) drain_wallet: bool,
-    pub(crate) single_recipient: Option<Script>,
+    pub(crate) drain_to: Option<Script>,
     pub(crate) fee_policy: Option<FeePolicy>,
     pub(crate) internal_policy_path: Option<BTreeMap<String, Vec<usize>>>,
     pub(crate) external_policy_path: Option<BTreeMap<String, Vec<usize>>>,
-    pub(crate) utxos: Vec<(UTXO, usize)>,
+    pub(crate) utxos: Vec<WeightedUtxo>,
     pub(crate) unspendable: HashSet<OutPoint>,
     pub(crate) manually_selected_only: bool,
-    pub(crate) sighash: Option<SigHashType>,
+    pub(crate) sighash: Option<psbt::PsbtSighashType>,
     pub(crate) ordering: TxOrdering,
     pub(crate) locktime: Option<u32>,
-    pub(crate) rbf: Option<RBFValue>,
+    pub(crate) rbf: Option<RbfValue>,
     pub(crate) version: Option<Version>,
     pub(crate) change_policy: ChangeSpendPolicy,
-    pub(crate) force_non_witness_utxo: bool,
+    pub(crate) only_witness_utxo: bool,
     pub(crate) add_global_xpubs: bool,
     pub(crate) include_output_redeem_witness_script: bool,
     pub(crate) bumping_fee: Option<PreviousFee>,
+    pub(crate) current_height: Option<u32>,
 }
 
 #[derive(Clone, Copy, Debug)]
@@ -183,9 +168,20 @@ impl std::default::Default for FeePolicy {
     }
 }
 
+impl<'a, Cs: Clone, Ctx, D> Clone for TxBuilder<'a, D, Cs, Ctx> {
+    fn clone(&self) -> Self {
+        TxBuilder {
+            wallet: self.wallet,
+            params: self.params.clone(),
+            coin_selection: self.coin_selection.clone(),
+            phantom: PhantomData,
+        }
+    }
+}
+
 // methods supported by both contexts, for any CoinSelectionAlgorithm
-impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
-    TxBuilder<'a, B, D, Cs, Ctx>
+impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
+    TxBuilder<'a, D, Cs, Ctx>
 {
     /// Set a custom fee rate
     pub fn fee_rate(&mut self, fee_rate: FeeRate) -> &mut Self {
@@ -251,7 +247,8 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     /// let mut path = BTreeMap::new();
     /// path.insert("aabbccdd".to_string(), vec![0, 1]);
     ///
-    /// let builder = wallet.build_tx()
+    /// let builder = wallet
+    ///     .build_tx()
     ///     .add_recipient(to_address.script_pubkey(), 50_000)
     ///     .policy_path(path, KeychainKind::External);
     ///
@@ -280,13 +277,16 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, Error> {
         let utxos = outpoints
             .iter()
-            .map(|outpoint| self.wallet.get_utxo(*outpoint)?.ok_or(Error::UnknownUTXO))
+            .map(|outpoint| self.wallet.get_utxo(*outpoint)?.ok_or(Error::UnknownUtxo))
             .collect::<Result<Vec<_>, _>>()?;
 
         for utxo in utxos {
             let descriptor = self.wallet.get_descriptor_for_keychain(utxo.keychain);
             let satisfaction_weight = descriptor.max_satisfaction_weight().unwrap();
-            self.params.utxos.push((utxo, satisfaction_weight));
+            self.params.utxos.push(WeightedUtxo {
+                satisfaction_weight,
+                utxo: Utxo::Local(utxo),
+            });
         }
 
         Ok(self)
@@ -300,6 +300,85 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
         self.add_utxos(&[outpoint])
     }
 
+    /// Add a foreign UTXO i.e. a UTXO not owned by this wallet.
+    ///
+    /// At a minimum to add a foreign UTXO we need:
+    ///
+    /// 1. `outpoint`: To add it to the raw transaction.
+    /// 2. `psbt_input`: To know the value.
+    /// 3. `satisfaction_weight`: To know how much weight/vbytes the input will add to the transaction for fee calculation.
+    ///
+    /// There are several security concerns about adding foreign UTXOs that application
+    /// developers should consider. First, how do you know the value of the input is correct? If a
+    /// `non_witness_utxo` is provided in the `psbt_input` then this method implicitly verifies the
+    /// value by checking it against the transaction. If only a `witness_utxo` is provided then this
+    /// method doesn't verify the value but just takes it as a given -- it is up to you to check
+    /// that whoever sent you the `input_psbt` was not lying!
+    ///
+    /// Secondly, you must somehow provide `satisfaction_weight` of the input. Depending on your
+    /// application it may be important that this be known precisely. If not, a malicious
+    /// counterparty may fool you into putting in a value that is too low, giving the transaction a
+    /// lower than expected feerate. They could also fool you into putting a value that is too high
+    /// causing you to pay a fee that is too high. The party who is broadcasting the transaction can
+    /// of course check the real input weight matches the expected weight prior to broadcasting.
+    ///
+    /// To guarantee the `satisfaction_weight` is correct, you can require the party providing the
+    /// `psbt_input` provide a miniscript descriptor for the input so you can check it against the
+    /// `script_pubkey` and then ask it for the [`max_satisfaction_weight`].
+    ///
+    /// This is an **EXPERIMENTAL** feature, API and other major changes are expected.
+    ///
+    /// # Errors
+    ///
+    /// This method returns errors in the following circumstances:
+    ///
+    /// 1. The `psbt_input` does not contain a `witness_utxo` or `non_witness_utxo`.
+    /// 2. The data in `non_witness_utxo` does not match what is in `outpoint`.
+    ///
+    /// Note unless you set [`only_witness_utxo`] any non-taproot `psbt_input` you pass to this
+    /// method must have `non_witness_utxo` set otherwise you will get an error when [`finish`]
+    /// is called.
+    ///
+    /// [`only_witness_utxo`]: Self::only_witness_utxo
+    /// [`finish`]: Self::finish
+    /// [`max_satisfaction_weight`]: miniscript::Descriptor::max_satisfaction_weight
+    pub fn add_foreign_utxo(
+        &mut self,
+        outpoint: OutPoint,
+        psbt_input: psbt::Input,
+        satisfaction_weight: usize,
+    ) -> Result<&mut Self, Error> {
+        if psbt_input.witness_utxo.is_none() {
+            match psbt_input.non_witness_utxo.as_ref() {
+                Some(tx) => {
+                    if tx.txid() != outpoint.txid {
+                        return Err(Error::Generic(
+                            "Foreign utxo outpoint does not match PSBT input".into(),
+                        ));
+                    }
+                    if tx.output.len() <= outpoint.vout as usize {
+                        return Err(Error::InvalidOutpoint(outpoint));
+                    }
+                }
+                None => {
+                    return Err(Error::Generic(
+                        "Foreign utxo missing witness_utxo or non_witness_utxo".into(),
+                    ))
+                }
+            }
+        }
+
+        self.params.utxos.push(WeightedUtxo {
+            satisfaction_weight,
+            utxo: Utxo::Foreign {
+                outpoint,
+                psbt_input: Box::new(psbt_input),
+            },
+        });
+
+        Ok(self)
+    }
+
     /// Only spend utxos added by [`add_utxo`].
     ///
     /// The wallet will **not** add additional utxos to the transaction even if they are needed to
@@ -332,7 +411,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     /// Sign with a specific sig hash
     ///
     /// **Use this option very carefully**
-    pub fn sighash(&mut self, sighash: SigHashType) -> &mut Self {
+    pub fn sighash(&mut self, sighash: psbt::PsbtSighashType) -> &mut Self {
         self.params.sighash = Some(sighash);
         self
     }
@@ -385,12 +464,13 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
         self
     }
 
-    /// Fill-in the [`psbt::Input::non_witness_utxo`](bitcoin::util::psbt::Input::non_witness_utxo) field even if the wallet only has SegWit
-    /// descriptors.
+    /// Only Fill-in the [`psbt::Input::witness_utxo`](bitcoin::util::psbt::Input::witness_utxo) field when spending from
+    /// SegWit descriptors.
     ///
-    /// This is useful for signers which always require it, like Trezor hardware wallets.
-    pub fn force_non_witness_utxo(&mut self) -> &mut Self {
-        self.params.force_non_witness_utxo = true;
+    /// This reduces the size of the PSBT, but some signers might reject them due to the lack of
+    /// the `non_witness_utxo`.
+    pub fn only_witness_utxo(&mut self) -> &mut Self {
+        self.params.only_witness_utxo = true;
         self
     }
 
@@ -427,7 +507,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     pub fn coin_selection<P: CoinSelectionAlgorithm<D>>(
         self,
         coin_selection: P,
-    ) -> TxBuilder<'a, B, D, P, Ctx> {
+    ) -> TxBuilder<'a, D, P, Ctx> {
         TxBuilder {
             wallet: self.wallet,
             params: self.params,
@@ -441,12 +521,48 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     /// Returns the [`BIP174`] "PSBT" and summary details about the transaction.
     ///
     /// [`BIP174`]: https://github.com/bitcoin/bips/blob/master/bip-0174.mediawiki
-    pub fn finish(self) -> Result<(PSBT, TransactionDetails), Error> {
+    pub fn finish(self) -> Result<(Psbt, TransactionDetails), Error> {
         self.wallet.create_tx(self.coin_selection, self.params)
     }
+
+    /// Enable signaling RBF
+    ///
+    /// This will use the default nSequence value of `0xFFFFFFFD`.
+    pub fn enable_rbf(&mut self) -> &mut Self {
+        self.params.rbf = Some(RbfValue::Default);
+        self
+    }
+
+    /// Enable signaling RBF with a specific nSequence value
+    ///
+    /// This can cause conflicts if the wallet's descriptors contain an "older" (OP_CSV) operator
+    /// and the given `nsequence` is lower than the CSV value.
+    ///
+    /// If the `nsequence` is higher than `0xFFFFFFFD` an error will be thrown, since it would not
+    /// be a valid nSequence to signal RBF.
+    pub fn enable_rbf_with_sequence(&mut self, nsequence: u32) -> &mut Self {
+        self.params.rbf = Some(RbfValue::Value(nsequence));
+        self
+    }
+
+    /// Set the current blockchain height.
+    ///
+    /// This will be used to:
+    /// 1. Set the nLockTime for preventing fee sniping.
+    /// **Note**: This will be ignored if you manually specify a nlocktime using [`TxBuilder::nlocktime`].
+    /// 2. Decide whether coinbase outputs are mature or not. If the coinbase outputs are not
+    ///    mature at `current_height`, we ignore them in the coin selection.
+    ///    If you want to create a transaction that spends immature coinbase inputs, manually
+    ///    add them using [`TxBuilder::add_utxos`].
+    ///
+    /// In both cases, if you don't provide a current height, we use the last sync height.
+    pub fn current_height(&mut self, height: u32) -> &mut Self {
+        self.params.current_height = Some(height);
+        self
+    }
 }
 
-impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, B, D, Cs, CreateTx> {
+impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, D, Cs, CreateTx> {
     /// Replace the recipients already added with a new list
     pub fn set_recipients(&mut self, recipients: Vec<(Script, u64)>) -> &mut Self {
         self.params.recipients = recipients;
@@ -459,69 +575,92 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, B, D,
         self
     }
 
-    /// Set a single recipient that will get all the selected funds minus the fee. No change will
-    /// be created
-    ///
-    /// This method overrides any recipient set with [`set_recipients`](Self::set_recipients) or
-    /// [`add_recipient`](Self::add_recipient).
-    ///
-    /// It can only be used in conjunction with [`drain_wallet`](Self::drain_wallet) to send the
-    /// entire content of the wallet (minus filters) to a single recipient or with a
-    /// list of manually selected UTXOs by enabling [`manually_selected_only`](Self::manually_selected_only)
-    /// and selecting them with or [`add_utxo`](Self::add_utxo).
-    ///
-    /// When bumping the fees of a transaction made with this option, the user should remeber to
-    /// add [`maintain_single_recipient`](Self::maintain_single_recipient) to correctly update the
-    /// single output instead of adding one more for the change.
-    pub fn set_single_recipient(&mut self, recipient: Script) -> &mut Self {
-        self.params.single_recipient = Some(recipient);
-        self.params.recipients.clear();
-
+    /// Add data as an output, using OP_RETURN
+    pub fn add_data(&mut self, data: &[u8]) -> &mut Self {
+        let script = Script::new_op_return(data);
+        self.add_recipient(script, 0u64);
         self
     }
 
-    /// Enable signaling RBF
+    /// Sets the address to *drain* excess coins to.
     ///
-    /// This will use the default nSequence value of `0xFFFFFFFD`.
-    pub fn enable_rbf(&mut self) -> &mut Self {
-        self.params.rbf = Some(RBFValue::Default);
-        self
-    }
-
-    /// Enable signaling RBF with a specific nSequence value
+    /// Usually, when there are excess coins they are sent to a change address generated by the
+    /// wallet. This option replaces the usual change address with an arbitrary `script_pubkey` of
+    /// your choosing. Just as with a change output, if the drain output is not needed (the excess
+    /// coins are too small) it will not be included in the resulting transaction. The only
+    /// difference is that it is valid to use `drain_to` without setting any ordinary recipients
+    /// with [`add_recipient`] (but it is perfectly fine to add recipients as well).
     ///
-    /// This can cause conflicts if the wallet's descriptors contain an "older" (OP_CSV) operator
-    /// and the given `nsequence` is lower than the CSV value.
+    /// If you choose not to set any recipients, you should either provide the utxos that the
+    /// transaction should spend via [`add_utxos`], or set [`drain_wallet`] to spend all of them.
     ///
-    /// If the `nsequence` is higher than `0xFFFFFFFD` an error will be thrown, since it would not
-    /// be a valid nSequence to signal RBF.
-    pub fn enable_rbf_with_sequence(&mut self, nsequence: u32) -> &mut Self {
-        self.params.rbf = Some(RBFValue::Value(nsequence));
+    /// When bumping the fees of a transaction made with this option, you probably want to
+    /// use [`allow_shrinking`] to allow this output to be reduced to pay for the extra fees.
+    ///
+    /// # Example
+    ///
+    /// `drain_to` is very useful for draining all the coins in a wallet with [`drain_wallet`] to a
+    /// single address.
+    ///
+    /// ```
+    /// # use std::str::FromStr;
+    /// # use bitcoin::*;
+    /// # use bdk::*;
+    /// # use bdk::wallet::tx_builder::CreateTx;
+    /// # let to_address = Address::from_str("2N4eQYCbKUHCCTUjBJeHcJp9ok6J2GZsTDt").unwrap();
+    /// # let wallet = doctest_wallet!();
+    /// let mut tx_builder = wallet.build_tx();
+    ///
+    /// tx_builder
+    ///     // Spend all outputs in this wallet.
+    ///     .drain_wallet()
+    ///     // Send the excess (which is all the coins minus the fee) to this address.
+    ///     .drain_to(to_address.script_pubkey())
+    ///     .fee_rate(FeeRate::from_sat_per_vb(5.0))
+    ///     .enable_rbf();
+    /// let (psbt, tx_details) = tx_builder.finish()?;
+    /// # Ok::<(), bdk::Error>(())
+    /// ```
+    ///
+    /// [`allow_shrinking`]: Self::allow_shrinking
+    /// [`add_recipient`]: Self::add_recipient
+    /// [`add_utxos`]: Self::add_utxos
+    /// [`drain_wallet`]: Self::drain_wallet
+    pub fn drain_to(&mut self, script_pubkey: Script) -> &mut Self {
+        self.params.drain_to = Some(script_pubkey);
         self
     }
 }
 
 // methods supported only by bump_fee
-impl<'a, B, D: BatchDatabase> TxBuilder<'a, B, D, DefaultCoinSelectionAlgorithm, BumpFee> {
-    /// Bump the fees of a transaction made with [`set_single_recipient`](Self::set_single_recipient)
+impl<'a, D: BatchDatabase> TxBuilder<'a, D, DefaultCoinSelectionAlgorithm, BumpFee> {
+    /// Explicitly tells the wallet that it is allowed to reduce the amount of the output matching this
+    /// `script_pubkey` in order to bump the transaction fee. Without specifying this the wallet
+    /// will attempt to find a change output to shrink instead.
     ///
-    /// Unless extra inputs are specified with [`add_utxo`], this flag will make
-    /// `bump_fee` reduce the value of the existing output, or fail if it would be consumed
-    /// entirely given the higher new fee rate.
+    /// **Note** that the output may shrink to below the dust limit and therefore be removed. If it is
+    /// preserved then it is currently not guaranteed to be in the same position as it was
+    /// originally.
     ///
-    /// If extra inputs are added and they are not entirely consumed in fees, a change output will not
-    /// be added; the existing output will simply grow in value.
-    ///
-    /// Fails if the transaction has more than one outputs.
-    ///
-    /// [`add_utxo`]: Self::add_utxo
-    pub fn maintain_single_recipient(&mut self) -> Result<&mut Self, Error> {
-        let mut recipients = self.params.recipients.drain(..).collect::<Vec<_>>();
-        if recipients.len() != 1 {
-            return Err(Error::SingleRecipientMultipleOutputs);
+    /// Returns an `Err` if `script_pubkey` can't be found among the recipients of the
+    /// transaction we are bumping.
+    pub fn allow_shrinking(&mut self, script_pubkey: Script) -> Result<&mut Self, Error> {
+        match self
+            .params
+            .recipients
+            .iter()
+            .position(|(recipient_script, _)| *recipient_script == script_pubkey)
+        {
+            Some(position) => {
+                self.params.recipients.remove(position);
+                self.params.drain_to = Some(script_pubkey);
+                Ok(self)
+            }
+            None => Err(Error::Generic(format!(
+                "{} was not in the original transaction",
+                script_pubkey
+            ))),
         }
-        self.params.single_recipient = Some(recipients.pop().unwrap().0);
-        Ok(self)
     }
 }
 
@@ -533,7 +672,7 @@ pub enum TxOrdering {
     /// Unchanged
     Untouched,
     /// BIP69 / Lexicographic
-    BIP69Lexicographic,
+    Bip69Lexicographic,
 }
 
 impl Default for TxOrdering {
@@ -559,7 +698,7 @@ impl TxOrdering {
 
                 tx.output.shuffle(&mut rng);
             }
-            TxOrdering::BIP69Lexicographic => {
+            TxOrdering::Bip69Lexicographic => {
                 tx.input.sort_unstable_by_key(|txin| {
                     (txin.previous_output.txid, txin.previous_output.vout)
                 });
@@ -586,16 +725,16 @@ impl Default for Version {
 ///
 /// Has a default value of `0xFFFFFFFD`
 #[derive(Debug, Ord, PartialOrd, Eq, PartialEq, Hash, Clone, Copy)]
-pub(crate) enum RBFValue {
+pub(crate) enum RbfValue {
     Default,
     Value(u32),
 }
 
-impl RBFValue {
+impl RbfValue {
     pub(crate) fn get_value(&self) -> u32 {
         match self {
-            RBFValue::Default => 0xFFFFFFFD,
-            RBFValue::Value(v) => *v,
+            RbfValue::Default => 0xFFFFFFFD,
+            RbfValue::Value(v) => *v,
         }
     }
 }
@@ -618,7 +757,7 @@ impl Default for ChangeSpendPolicy {
 }
 
 impl ChangeSpendPolicy {
-    pub(crate) fn is_satisfied_by(&self, utxo: &UTXO) -> bool {
+    pub(crate) fn is_satisfied_by(&self, utxo: &LocalUtxo) -> bool {
         match self {
             ChangeSpendPolicy::ChangeAllowed => true,
             ChangeSpendPolicy::OnlyChange => utxo.keychain == KeychainKind::Internal,
@@ -629,12 +768,12 @@ impl ChangeSpendPolicy {
 
 #[cfg(test)]
 mod test {
-    const ORDERING_TEST_TX: &'static str = "0200000003c26f3eb7932f7acddc5ddd26602b77e7516079b03090a16e2c2f54\
-                                            85d1fd600f0100000000ffffffffc26f3eb7932f7acddc5ddd26602b77e75160\
-                                            79b03090a16e2c2f5485d1fd600f0000000000ffffffff571fb3e02278217852\
-                                            dd5d299947e2b7354a639adc32ec1fa7b82cfb5dec530e0500000000ffffffff\
-                                            03e80300000000000002aaeee80300000000000001aa200300000000000001ff\
-                                            00000000";
+    const ORDERING_TEST_TX: &str = "0200000003c26f3eb7932f7acddc5ddd26602b77e7516079b03090a16e2c2f54\
+                                    85d1fd600f0100000000ffffffffc26f3eb7932f7acddc5ddd26602b77e75160\
+                                    79b03090a16e2c2f5485d1fd600f0000000000ffffffff571fb3e02278217852\
+                                    dd5d299947e2b7354a639adc32ec1fa7b82cfb5dec530e0500000000ffffffff\
+                                    03e80300000000000002aaeee80300000000000001aa200300000000000001ff\
+                                    00000000";
     macro_rules! ordering_test_tx {
         () => {
             deserialize::<bitcoin::Transaction>(&Vec::<u8>::from_hex(ORDERING_TEST_TX).unwrap())
@@ -678,9 +817,9 @@ mod test {
         use std::str::FromStr;
 
         let original_tx = ordering_test_tx!();
-        let mut tx = original_tx.clone();
+        let mut tx = original_tx;
 
-        TxOrdering::BIP69Lexicographic.sort_tx(&mut tx);
+        TxOrdering::Bip69Lexicographic.sort_tx(&mut tx);
 
         assert_eq!(
             tx.input[0].previous_output,
@@ -709,23 +848,25 @@ mod test {
         assert_eq!(tx.output[2].script_pubkey, From::from(vec![0xAA, 0xEE]));
     }
 
-    fn get_test_utxos() -> Vec<UTXO> {
+    fn get_test_utxos() -> Vec<LocalUtxo> {
         vec![
-            UTXO {
+            LocalUtxo {
                 outpoint: OutPoint {
                     txid: Default::default(),
                     vout: 0,
                 },
                 txout: Default::default(),
                 keychain: KeychainKind::External,
+                is_spent: false,
             },
-            UTXO {
+            LocalUtxo {
                 outpoint: OutPoint {
                     txid: Default::default(),
                     vout: 1,
                 },
                 txout: Default::default(),
                 keychain: KeychainKind::Internal,
+                is_spent: false,
             },
         ]
     }
@@ -736,9 +877,9 @@ mod test {
         let filtered = get_test_utxos()
             .into_iter()
             .filter(|u| change_spend_policy.is_satisfied_by(u))
-            .collect::<Vec<_>>();
+            .count();
 
-        assert_eq!(filtered.len(), 2);
+        assert_eq!(filtered, 2);
     }
 
     #[test]

src/wallet/utils.rs

@@ -1,34 +1,19 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
 //
-// Copyright (c) 2020 Magical Bitcoin
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
 //
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
+use bitcoin::blockdata::script::Script;
 use bitcoin::secp256k1::{All, Secp256k1};
 
 use miniscript::{MiniscriptKey, Satisfier, ToPublicKey};
 
-// De-facto standard "dust limit" (even though it should change based on the output type)
-pub const DUST_LIMIT_SATOSHI: u64 = 546;
-
 // MSB of the nSequence. If set there's no consensus-constraint, so it must be disabled when
 // spending using CSV in order to enforce CSV rules
 pub(crate) const SEQUENCE_LOCKTIME_DISABLE_FLAG: u32 = 1 << 31;
@@ -41,18 +26,19 @@ pub(crate) const SEQUENCE_LOCKTIME_MASK: u32 = 0x0000FFFF;
 // Threshold for nLockTime to be considered a block-height-based timelock rather than time-based
 pub(crate) const BLOCKS_TIMELOCK_THRESHOLD: u32 = 500000000;
 
-/// Trait to check if a value is below the dust limit
+/// Trait to check if a value is below the dust limit.
+/// We are performing dust value calculation for a given script public key using rust-bitcoin to
+/// keep it compatible with network dust rate
 // we implement this trait to make sure we don't mess up the comparison with off-by-one like a <
-// instead of a <= etc. The constant value for the dust limit is not public on purpose, to
-// encourage the usage of this trait.
+// instead of a <= etc.
 pub trait IsDust {
     /// Check whether or not a value is below dust limit
-    fn is_dust(&self) -> bool;
+    fn is_dust(&self, script: &Script) -> bool;
 }
 
 impl IsDust for u64 {
-    fn is_dust(&self) -> bool {
-        *self <= DUST_LIMIT_SATOSHI
+    fn is_dust(&self, script: &Script) -> bool {
+        *self < script.dust_value().as_sat()
     }
 }
 
@@ -151,47 +137,32 @@ impl<Pk: MiniscriptKey + ToPublicKey> Satisfier<Pk> for Older {
 
 pub(crate) type SecpCtx = Secp256k1<All>;
 
-pub struct ChunksIterator<I: Iterator> {
-    iter: I,
-    size: usize,
-}
-
-impl<I: Iterator> ChunksIterator<I> {
-    #[allow(dead_code)]
-    pub fn new(iter: I, size: usize) -> Self {
-        ChunksIterator { iter, size }
-    }
-}
-
-impl<I: Iterator> Iterator for ChunksIterator<I> {
-    type Item = Vec<<I as std::iter::Iterator>::Item>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        let mut v = Vec::new();
-        for _ in 0..self.size {
-            let e = self.iter.next();
-
-            match e {
-                None => break,
-                Some(val) => v.push(val),
-            }
-        }
-
-        if v.is_empty() {
-            return None;
-        }
-
-        Some(v)
-    }
-}
-
 #[cfg(test)]
 mod test {
     use super::{
-        check_nlocktime, check_nsequence_rbf, BLOCKS_TIMELOCK_THRESHOLD,
+        check_nlocktime, check_nsequence_rbf, IsDust, BLOCKS_TIMELOCK_THRESHOLD,
         SEQUENCE_LOCKTIME_TYPE_FLAG,
     };
+    use crate::bitcoin::Address;
     use crate::types::FeeRate;
+    use std::str::FromStr;
+
+    #[test]
+    fn test_is_dust() {
+        let script_p2pkh = Address::from_str("1GNgwA8JfG7Kc8akJ8opdNWJUihqUztfPe")
+            .unwrap()
+            .script_pubkey();
+        assert!(script_p2pkh.is_p2pkh());
+        assert!(545.is_dust(&script_p2pkh));
+        assert!(!546.is_dust(&script_p2pkh));
+
+        let script_p2wpkh = Address::from_str("bc1qxlh2mnc0yqwas76gqq665qkggee5m98t8yskd8")
+            .unwrap()
+            .script_pubkey();
+        assert!(script_p2wpkh.is_v0_p2wpkh());
+        assert!(293.is_dust(&script_p2wpkh));
+        assert!(!294.is_dust(&script_p2wpkh));
+    }
 
     #[test]
     fn test_fee_from_btc_per_kb() {
@@ -214,31 +185,31 @@ mod test {
     #[test]
     fn test_check_nsequence_rbf_msb_set() {
         let result = check_nsequence_rbf(0x80000000, 5000);
-        assert_eq!(result, false);
+        assert!(!result);
     }
 
     #[test]
     fn test_check_nsequence_rbf_lt_csv() {
         let result = check_nsequence_rbf(4000, 5000);
-        assert_eq!(result, false);
+        assert!(!result);
     }
 
     #[test]
     fn test_check_nsequence_rbf_different_unit() {
         let result = check_nsequence_rbf(SEQUENCE_LOCKTIME_TYPE_FLAG + 5000, 5000);
-        assert_eq!(result, false);
+        assert!(!result);
     }
 
     #[test]
     fn test_check_nsequence_rbf_mask() {
         let result = check_nsequence_rbf(0x3f + 10_000, 5000);
-        assert_eq!(result, true);
+        assert!(result);
     }
 
     #[test]
     fn test_check_nsequence_rbf_same_unit_blocks() {
         let result = check_nsequence_rbf(10_000, 5000);
-        assert_eq!(result, true);
+        assert!(result);
     }
 
     #[test]
@@ -247,25 +218,25 @@ mod test {
             SEQUENCE_LOCKTIME_TYPE_FLAG + 10_000,
             SEQUENCE_LOCKTIME_TYPE_FLAG + 5000,
         );
-        assert_eq!(result, true);
+        assert!(result);
     }
 
     #[test]
     fn test_check_nlocktime_lt_cltv() {
         let result = check_nlocktime(4000, 5000);
-        assert_eq!(result, false);
+        assert!(!result);
     }
 
     #[test]
     fn test_check_nlocktime_different_unit() {
         let result = check_nlocktime(BLOCKS_TIMELOCK_THRESHOLD + 5000, 5000);
-        assert_eq!(result, false);
+        assert!(!result);
     }
 
     #[test]
     fn test_check_nlocktime_same_unit_blocks() {
         let result = check_nlocktime(10_000, 5000);
-        assert_eq!(result, true);
+        assert!(result);
     }
 
     #[test]
@@ -274,6 +245,6 @@ mod test {
             BLOCKS_TIMELOCK_THRESHOLD + 10_000,
             BLOCKS_TIMELOCK_THRESHOLD + 5000,
         );
-        assert_eq!(result, true);
+        assert!(result);
     }
 }

src/wallet/verify.rs

@@ -0,0 +1,160 @@
+// Bitcoin Dev Kit
+// Written in 2021 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Verify transactions against the consensus rules
+
+use std::collections::HashMap;
+use std::fmt;
+
+use bitcoin::consensus::serialize;
+use bitcoin::{OutPoint, Transaction, Txid};
+
+use crate::blockchain::GetTx;
+use crate::database::Database;
+use crate::error::Error;
+
+/// Verify a transaction against the consensus rules
+///
+/// This function uses [`bitcoinconsensus`] to verify transactions by fetching the required data
+/// either from the [`Database`] or using the [`Blockchain`].
+///
+/// Depending on the [capabilities](crate::blockchain::Blockchain::get_capabilities) of the
+/// [`Blockchain`] backend, the method could fail when called with old "historical" transactions or
+/// with unconfirmed transactions that have been evicted from the backend's memory.
+pub fn verify_tx<D: Database, B: GetTx>(
+    tx: &Transaction,
+    database: &D,
+    blockchain: &B,
+) -> Result<(), VerifyError> {
+    log::debug!("Verifying {}", tx.txid());
+
+    let serialized_tx = serialize(tx);
+    let mut tx_cache = HashMap::<_, Transaction>::new();
+
+    for (index, input) in tx.input.iter().enumerate() {
+        let prev_tx = if let Some(prev_tx) = tx_cache.get(&input.previous_output.txid) {
+            prev_tx.clone()
+        } else if let Some(prev_tx) = database.get_raw_tx(&input.previous_output.txid)? {
+            prev_tx
+        } else if let Some(prev_tx) = blockchain.get_tx(&input.previous_output.txid)? {
+            prev_tx
+        } else {
+            return Err(VerifyError::MissingInputTx(input.previous_output.txid));
+        };
+
+        let spent_output = prev_tx
+            .output
+            .get(input.previous_output.vout as usize)
+            .ok_or(VerifyError::InvalidInput(input.previous_output))?;
+
+        bitcoinconsensus::verify(
+            &spent_output.script_pubkey.to_bytes(),
+            spent_output.value,
+            &serialized_tx,
+            index,
+        )?;
+
+        // Since we have a local cache we might as well cache stuff from the db, as it will very
+        // likely decrease latency compared to reading from disk or performing an SQL query.
+        tx_cache.insert(prev_tx.txid(), prev_tx);
+    }
+
+    Ok(())
+}
+
+/// Error during validation of a tx agains the consensus rules
+#[derive(Debug)]
+pub enum VerifyError {
+    /// The transaction being spent is not available in the database or the blockchain client
+    MissingInputTx(Txid),
+    /// The transaction being spent doesn't have the requested output
+    InvalidInput(OutPoint),
+
+    /// Consensus error
+    Consensus(bitcoinconsensus::Error),
+
+    /// Generic error
+    ///
+    /// It has to be wrapped in a `Box` since `Error` has a variant that contains this enum
+    Global(Box<Error>),
+}
+
+impl fmt::Display for VerifyError {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(f, "{:?}", self)
+    }
+}
+
+impl std::error::Error for VerifyError {}
+
+impl From<Error> for VerifyError {
+    fn from(other: Error) -> Self {
+        VerifyError::Global(Box::new(other))
+    }
+}
+impl_error!(bitcoinconsensus::Error, Consensus, VerifyError);
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::database::{BatchOperations, MemoryDatabase};
+    use bitcoin::consensus::encode::deserialize;
+    use bitcoin::hashes::hex::FromHex;
+    use bitcoin::{Transaction, Txid};
+
+    struct DummyBlockchain;
+
+    impl GetTx for DummyBlockchain {
+        fn get_tx(&self, _txid: &Txid) -> Result<Option<Transaction>, Error> {
+            Ok(None)
+        }
+    }
+
+    #[test]
+    fn test_verify_fail_unsigned_tx() {
+        // https://blockstream.info/tx/95da344585fcf2e5f7d6cbf2c3df2dcce84f9196f7a7bb901a43275cd6eb7c3f
+        let prev_tx: Transaction = deserialize(&Vec::<u8>::from_hex("020000000101192dea5e66d444380e106f8e53acb171703f00d43fb6b3ae88ca5644bdb7e1000000006b48304502210098328d026ce138411f957966c1cf7f7597ccbb170f5d5655ee3e9f47b18f6999022017c3526fc9147830e1340e04934476a3d1521af5b4de4e98baf49ec4c072079e01210276f847f77ec8dd66d78affd3c318a0ed26d89dab33fa143333c207402fcec352feffffff023d0ac203000000001976a9144bfbaf6afb76cc5771bc6404810d1cc041a6933988aca4b956050000000017a91494d5543c74a3ee98e0cf8e8caef5dc813a0f34b48768cb0700").unwrap()).unwrap();
+        // https://blockstream.info/tx/aca326a724eda9a461c10a876534ecd5ae7b27f10f26c3862fb996f80ea2d45d
+        let signed_tx: Transaction = deserialize(&Vec::<u8>::from_hex("02000000013f7cebd65c27431a90bba7f796914fe8cc2ddfc3f2cbd6f7e5f2fc854534da95000000006b483045022100de1ac3bcdfb0332207c4a91f3832bd2c2915840165f876ab47c5f8996b971c3602201c6c053d750fadde599e6f5c4e1963df0f01fc0d97815e8157e3d59fe09ca30d012103699b464d1d8bc9e47d4fb1cdaa89a1c5783d68363c4dbc4b524ed3d857148617feffffff02836d3c01000000001976a914fc25d6d5c94003bf5b0c7b640a248e2c637fcfb088ac7ada8202000000001976a914fbed3d9b11183209a57999d54d59f67c019e756c88ac6acb0700").unwrap()).unwrap();
+
+        let mut database = MemoryDatabase::new();
+        let blockchain = DummyBlockchain;
+
+        let mut unsigned_tx = signed_tx.clone();
+        for input in &mut unsigned_tx.input {
+            input.script_sig = Default::default();
+            input.witness = Default::default();
+        }
+
+        let result = verify_tx(&signed_tx, &database, &blockchain);
+        assert!(result.is_err(), "Should fail with missing input tx");
+        assert!(
+            matches!(result, Err(VerifyError::MissingInputTx(txid)) if txid == prev_tx.txid()),
+            "Error should be a `MissingInputTx` error"
+        );
+
+        // insert the prev_tx
+        database.set_raw_tx(&prev_tx).unwrap();
+
+        let result = verify_tx(&unsigned_tx, &database, &blockchain);
+        assert!(result.is_err(), "Should fail since the TX is unsigned");
+        assert!(
+            matches!(result, Err(VerifyError::Consensus(_))),
+            "Error should be a `Consensus` error"
+        );
+
+        let result = verify_tx(&signed_tx, &database, &blockchain);
+        assert!(
+            result.is_ok(),
+            "Should work since the TX is correctly signed"
+        );
+    }
+}

testutils-macros/Cargo.toml

@@ -1,25 +0,0 @@
-[package]
-name = "bdk-testutils-macros"
-version = "0.3.0"
-authors = ["Alekos Filini <alekos.filini@gmail.com>"]
-edition = "2018"
-homepage = "https://bitcoindevkit.org"
-repository = "https://github.com/bitcoindevkit/bdk"
-documentation = "https://docs.rs/bdk-testutils-macros"
-description = "Supporting testing macros for `bdk`"
-keywords = ["bdk"]
-license = "MIT"
-
-[lib]
-proc-macro = true
-name = "testutils_macros"
-
-# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
-
-[dependencies]
-syn = { version = "1.0", features = ["parsing", "full"] }
-proc-macro2 = "1.0"
-quote = "1.0"
-
-[features]
-debug = ["syn/extra-traits"]

testutils-macros/src/lib.rs

@@ -1,565 +0,0 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
-//
-// Copyright (c) 2020 Magical Bitcoin
-//
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
-
-#[macro_use]
-extern crate quote;
-
-use proc_macro::TokenStream;
-
-use syn::spanned::Spanned;
-use syn::{parse, parse2, Ident, ReturnType};
-
-#[proc_macro_attribute]
-pub fn bdk_blockchain_tests(attr: TokenStream, item: TokenStream) -> TokenStream {
-    let root_ident = if !attr.is_empty() {
-        match parse::<syn::ExprPath>(attr) {
-            Ok(parsed) => parsed,
-            Err(e) => {
-                let error_string = e.to_string();
-                return (quote! {
-                    compile_error!("Invalid crate path: {:?}", #error_string)
-                })
-                .into();
-            }
-        }
-    } else {
-        parse2::<syn::ExprPath>(quote! { bdk }).unwrap()
-    };
-
-    match parse::<syn::ItemFn>(item) {
-        Err(_) => (quote! {
-            compile_error!("#[bdk_blockchain_tests] can only be used on `fn`s")
-        })
-        .into(),
-        Ok(parsed) => {
-            let parsed_sig_ident = parsed.sig.ident.clone();
-            let mod_name = Ident::new(
-                &format!("generated_tests_{}", parsed_sig_ident.to_string()),
-                parsed.span(),
-            );
-
-            let return_type = match parsed.sig.output {
-                ReturnType::Type(_, ref t) => t.clone(),
-                ReturnType::Default => {
-                    return (quote! {
-                        compile_error!("The tagged function must return a type that impl `Blockchain`")
-                    }).into();
-                }
-            };
-
-            let output = quote! {
-
-            #parsed
-
-            mod #mod_name {
-                use bitcoin::Network;
-
-                use miniscript::Descriptor;
-
-                use testutils::{TestClient, serial};
-
-                use #root_ident::blockchain::{Blockchain, noop_progress};
-                use #root_ident::descriptor::ExtendedDescriptor;
-                use #root_ident::database::MemoryDatabase;
-                use #root_ident::types::KeychainKind;
-                use #root_ident::{Wallet, TxBuilder, FeeRate};
-
-                use super::*;
-
-                fn get_blockchain() -> #return_type {
-                    #parsed_sig_ident()
-                }
-
-                fn get_wallet_from_descriptors(descriptors: &(String, Option<String>)) -> Wallet<#return_type, MemoryDatabase> {
-                    Wallet::new(&descriptors.0.to_string(), descriptors.1.as_ref(), Network::Regtest, MemoryDatabase::new(), get_blockchain()).unwrap()
-                }
-
-                fn init_single_sig() -> (Wallet<#return_type, MemoryDatabase>, (String, Option<String>), TestClient) {
-                    let descriptors = testutils! {
-                        @descriptors ( "wpkh(Alice)" ) ( "wpkh(Alice)" ) ( @keys ( "Alice" => (@generate_xprv "/44'/0'/0'/0/*", "/44'/0'/0'/1/*") ) )
-                    };
-
-                    let test_client = TestClient::new();
-                    let wallet = get_wallet_from_descriptors(&descriptors);
-
-                    (wallet, descriptors, test_client)
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_simple() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-
-                    let tx = testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 )
-                    };
-                    println!("{:?}", tx);
-                    let txid = test_client.receive(tx);
-
-                    wallet.sync(noop_progress(), None).unwrap();
-
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-                    assert_eq!(wallet.list_unspent().unwrap()[0].keychain, KeychainKind::External);
-
-                    let list_tx_item = &wallet.list_transactions(false).unwrap()[0];
-                    assert_eq!(list_tx_item.txid, txid);
-                    assert_eq!(list_tx_item.received, 50_000);
-                    assert_eq!(list_tx_item.sent, 0);
-                    assert_eq!(list_tx_item.height, None);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_stop_gap_20() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 5) => 50_000 )
-                    });
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 25) => 50_000 )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-
-                    assert_eq!(wallet.get_balance().unwrap(), 100_000);
-                    assert_eq!(wallet.list_transactions(false).unwrap().len(), 2);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_before_and_after_receive() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 0);
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-                    assert_eq!(wallet.list_transactions(false).unwrap().len(), 1);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_multiple_outputs_same_tx() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-
-                    let txid = test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000, (@external descriptors, 5) => 30_000 )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-
-                    assert_eq!(wallet.get_balance().unwrap(), 105_000);
-                    assert_eq!(wallet.list_transactions(false).unwrap().len(), 1);
-                    assert_eq!(wallet.list_unspent().unwrap().len(), 3);
-
-                    let list_tx_item = &wallet.list_transactions(false).unwrap()[0];
-                    assert_eq!(list_tx_item.txid, txid);
-                    assert_eq!(list_tx_item.received, 105_000);
-                    assert_eq!(list_tx_item.sent, 0);
-                    assert_eq!(list_tx_item.height, None);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_receive_multi() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 )
-                    });
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 5) => 25_000 )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-
-                    assert_eq!(wallet.get_balance().unwrap(), 75_000);
-                    assert_eq!(wallet.list_transactions(false).unwrap().len(), 2);
-                    assert_eq!(wallet.list_unspent().unwrap().len(), 2);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_address_reuse() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 25_000 )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 75_000);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_receive_rbf_replaced() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-
-                    let txid = test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 ) ( @replaceable true )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-                    assert_eq!(wallet.list_transactions(false).unwrap().len(), 1);
-                    assert_eq!(wallet.list_unspent().unwrap().len(), 1);
-
-                    let list_tx_item = &wallet.list_transactions(false).unwrap()[0];
-                    assert_eq!(list_tx_item.txid, txid);
-                    assert_eq!(list_tx_item.received, 50_000);
-                    assert_eq!(list_tx_item.sent, 0);
-                    assert_eq!(list_tx_item.height, None);
-
-                    let new_txid = test_client.bump_fee(&txid);
-
-                    wallet.sync(noop_progress(), None).unwrap();
-
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-                    assert_eq!(wallet.list_transactions(false).unwrap().len(), 1);
-                    assert_eq!(wallet.list_unspent().unwrap().len(), 1);
-
-                    let list_tx_item = &wallet.list_transactions(false).unwrap()[0];
-                    assert_eq!(list_tx_item.txid, new_txid);
-                    assert_eq!(list_tx_item.received, 50_000);
-                    assert_eq!(list_tx_item.sent, 0);
-                    assert_eq!(list_tx_item.height, None);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_reorg_block() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-
-                    let txid = test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 ) ( @confirmations 1 ) ( @replaceable true )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-                    assert_eq!(wallet.list_transactions(false).unwrap().len(), 1);
-                    assert_eq!(wallet.list_unspent().unwrap().len(), 1);
-
-                    let list_tx_item = &wallet.list_transactions(false).unwrap()[0];
-                    assert_eq!(list_tx_item.txid, txid);
-                    assert!(list_tx_item.height.is_some());
-
-                    // Invalidate 1 block
-                    test_client.invalidate(1);
-
-                    wallet.sync(noop_progress(), None).unwrap();
-
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-
-                    let list_tx_item = &wallet.list_transactions(false).unwrap()[0];
-                    assert_eq!(list_tx_item.txid, txid);
-                    assert_eq!(list_tx_item.height, None);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_after_send() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-                    println!("{}", descriptors.0);
-                    let node_addr = test_client.get_node_address(None);
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-
-                    let mut builder = wallet.build_tx();
-                    builder.add_recipient(node_addr.script_pubkey(), 25_000);
-                    let (psbt, details) = builder.finish().unwrap();
-                    let (psbt, finalized) = wallet.sign(psbt, None).unwrap();
-                    assert!(finalized, "Cannot finalize transaction");
-                    let tx = psbt.extract_tx();
-                    println!("{}", bitcoin::consensus::encode::serialize_hex(&tx));
-                    wallet.broadcast(tx).unwrap();
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), details.received);
-
-                    assert_eq!(wallet.list_transactions(false).unwrap().len(), 2);
-                    assert_eq!(wallet.list_unspent().unwrap().len(), 1);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_outgoing_from_scratch() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-                    let node_addr = test_client.get_node_address(None);
-
-                    let received_txid = test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-
-                    let mut builder = wallet.build_tx();
-                    builder.add_recipient(node_addr.script_pubkey(), 25_000);
-                    let (psbt, details) = builder.finish().unwrap();
-                    let (psbt, finalized) = wallet.sign(psbt, None).unwrap();
-                    assert!(finalized, "Cannot finalize transaction");
-                    let sent_txid = wallet.broadcast(psbt.extract_tx()).unwrap();
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), details.received);
-
-                    // empty wallet
-                    let wallet = get_wallet_from_descriptors(&descriptors);
-                    wallet.sync(noop_progress(), None).unwrap();
-
-                    let tx_map = wallet.list_transactions(false).unwrap().into_iter().map(|tx| (tx.txid, tx)).collect::<std::collections::HashMap<_, _>>();
-
-                    let received = tx_map.get(&received_txid).unwrap();
-                    assert_eq!(received.received, 50_000);
-                    assert_eq!(received.sent, 0);
-
-                    let sent = tx_map.get(&sent_txid).unwrap();
-                    assert_eq!(sent.received, details.received);
-                    assert_eq!(sent.sent, details.sent);
-                    assert_eq!(sent.fees, details.fees);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_long_change_chain() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-                    let node_addr = test_client.get_node_address(None);
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 )
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-
-                    let mut total_sent = 0;
-                    for _ in 0..5 {
-                        let mut builder = wallet.build_tx();
-                        builder.add_recipient(node_addr.script_pubkey(), 5_000);
-                        let (psbt, details) = builder.finish().unwrap();
-                        let (psbt, finalized) = wallet.sign(psbt, None).unwrap();
-                        assert!(finalized, "Cannot finalize transaction");
-                        wallet.broadcast(psbt.extract_tx()).unwrap();
-
-                        wallet.sync(noop_progress(), None).unwrap();
-
-                        total_sent += 5_000 + details.fees;
-                    }
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000 - total_sent);
-
-                    // empty wallet
-                    let wallet = get_wallet_from_descriptors(&descriptors);
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000 - total_sent);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_bump_fee() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-                    let node_addr = test_client.get_node_address(None);
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 ) (@confirmations 1)
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-
-                    let mut builder = wallet.build_tx();
-                    builder.add_recipient(node_addr.script_pubkey().clone(), 5_000).enable_rbf();
-                    let (psbt, details) = builder.finish().unwrap();
-                    let (psbt, finalized) = wallet.sign(psbt, None).unwrap();
-                    assert!(finalized, "Cannot finalize transaction");
-                    wallet.broadcast(psbt.extract_tx()).unwrap();
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000 - details.fees - 5_000);
-                    assert_eq!(wallet.get_balance().unwrap(), details.received);
-
-                    let mut builder = wallet.build_fee_bump(details.txid).unwrap();
-                    builder.fee_rate(FeeRate::from_sat_per_vb(2.1));
-                    let (new_psbt, new_details) = builder.finish().unwrap();
-                    let (new_psbt, finalized) = wallet.sign(new_psbt, None).unwrap();
-                    assert!(finalized, "Cannot finalize transaction");
-                    wallet.broadcast(new_psbt.extract_tx()).unwrap();
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000 - new_details.fees - 5_000);
-                    assert_eq!(wallet.get_balance().unwrap(), new_details.received);
-
-                    assert!(new_details.fees > details.fees);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_bump_fee_remove_change() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-                    let node_addr = test_client.get_node_address(None);
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000 ) (@confirmations 1)
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 50_000);
-
-                    let mut builder = wallet.build_tx();
-                    builder.add_recipient(node_addr.script_pubkey().clone(), 49_000).enable_rbf();
-                    let (psbt, details) = builder.finish().unwrap();
-                    let (psbt, finalized) = wallet.sign(psbt, None).unwrap();
-                    assert!(finalized, "Cannot finalize transaction");
-                    wallet.broadcast(psbt.extract_tx()).unwrap();
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 1_000 - details.fees);
-                    assert_eq!(wallet.get_balance().unwrap(), details.received);
-
-                    let mut builder = wallet.build_fee_bump(details.txid).unwrap();
-                    builder.fee_rate(FeeRate::from_sat_per_vb(5.0));
-                    let (new_psbt, new_details) = builder.finish().unwrap();
-                    let (new_psbt, finalized) = wallet.sign(new_psbt, None).unwrap();
-                    assert!(finalized, "Cannot finalize transaction");
-                    wallet.broadcast(new_psbt.extract_tx()).unwrap();
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 0);
-                    assert_eq!(new_details.received, 0);
-
-                    assert!(new_details.fees > details.fees);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_bump_fee_add_input() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-                    let node_addr = test_client.get_node_address(None);
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000 ) (@confirmations 1)
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 75_000);
-
-                    let mut builder = wallet.build_tx();
-                    builder.add_recipient(node_addr.script_pubkey().clone(), 49_000).enable_rbf();
-                    let (psbt, details) = builder.finish().unwrap();
-                    let (psbt, finalized) = wallet.sign(psbt, None).unwrap();
-                    assert!(finalized, "Cannot finalize transaction");
-                    wallet.broadcast(psbt.extract_tx()).unwrap();
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 26_000 - details.fees);
-                    assert_eq!(details.received, 1_000 - details.fees);
-
-                    let mut builder = wallet.build_fee_bump(details.txid).unwrap();
-                    builder.fee_rate(FeeRate::from_sat_per_vb(10.0));
-                    let (new_psbt, new_details) = builder.finish().unwrap();
-                    let (new_psbt, finalized) = wallet.sign(new_psbt, None).unwrap();
-                    assert!(finalized, "Cannot finalize transaction");
-                    wallet.broadcast(new_psbt.extract_tx()).unwrap();
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(new_details.sent, 75_000);
-                    assert_eq!(wallet.get_balance().unwrap(), new_details.received);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_bump_fee_add_input_no_change() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-                    let node_addr = test_client.get_node_address(None);
-
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000 ) (@confirmations 1)
-                    });
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 75_000);
-
-                    let mut builder = wallet.build_tx();
-                    builder.add_recipient(node_addr.script_pubkey().clone(), 49_000).enable_rbf();
-                    let (psbt, details) = builder.finish().unwrap();
-                    let (psbt, finalized) = wallet.sign(psbt, None).unwrap();
-                    assert!(finalized, "Cannot finalize transaction");
-                    wallet.broadcast(psbt.extract_tx()).unwrap();
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 26_000 - details.fees);
-                    assert_eq!(details.received, 1_000 - details.fees);
-
-                    let mut builder = wallet.build_fee_bump(details.txid).unwrap();
-                    builder.fee_rate(FeeRate::from_sat_per_vb(123.0));
-                    let (new_psbt, new_details) = builder.finish().unwrap();
-                    println!("{:#?}", new_details);
-
-                    let (new_psbt, finalized) = wallet.sign(new_psbt, None).unwrap();
-                    assert!(finalized, "Cannot finalize transaction");
-                    wallet.broadcast(new_psbt.extract_tx()).unwrap();
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(new_details.sent, 75_000);
-                    assert_eq!(wallet.get_balance().unwrap(), 0);
-                    assert_eq!(new_details.received, 0);
-                }
-
-                #[test]
-                #[serial]
-                fn test_sync_receive_coinbase() {
-                    let (wallet, descriptors, mut test_client) = init_single_sig();
-                    let wallet_addr = wallet.get_new_address().unwrap();
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert_eq!(wallet.get_balance().unwrap(), 0);
-
-                    test_client.generate(1, Some(wallet_addr));
-
-                    wallet.sync(noop_progress(), None).unwrap();
-                    assert!(wallet.get_balance().unwrap() > 0);
-                }
-            }
-
-                        };
-
-            output.into()
-        }
-    }
-}

testutils/.gitignore

@@ -1,2 +0,0 @@
-target/
-Cargo.lock

testutils/Cargo.toml

@@ -1,26 +0,0 @@
-[package]
-name = "bdk-testutils"
-version = "0.3.0"
-authors = ["Alekos Filini <alekos.filini@gmail.com>"]
-edition = "2018"
-homepage = "https://bitcoindevkit.org"
-repository = "https://github.com/bitcoindevkit/bdk"
-documentation = "https://docs.rs/bdk-testutils"
-description = "Supporting testing utilities for `bdk`"
-keywords = ["bdk"]
-license = "MIT"
-
-[lib]
-name = "testutils"
-
-# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
-
-[dependencies]
-log = "0.4.8"
-serde = { version = "1.0", features = ["derive"] }
-serde_json = "1.0"
-serial_test = "0.4"
-bitcoin = "0.26"
-bitcoincore-rpc = "0.13"
-miniscript = "5.1"
-electrum-client = "0.6.0"

testutils/src/lib.rs

@@ -1,575 +0,0 @@
-// Magical Bitcoin Library
-// Written in 2020 by
-//     Alekos Filini <alekos.filini@gmail.com>
-//
-// Copyright (c) 2020 Magical Bitcoin
-//
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to the following conditions:
-//
-// The above copyright notice and this permission notice shall be included in all
-// copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-// SOFTWARE.
-
-#[macro_use]
-extern crate serde_json;
-
-pub use serial_test::serial;
-
-use std::collections::HashMap;
-use std::env;
-use std::ops::Deref;
-use std::path::PathBuf;
-use std::str::FromStr;
-use std::time::Duration;
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
-
-use bitcoin::consensus::encode::{deserialize, serialize};
-use bitcoin::hashes::hex::{FromHex, ToHex};
-use bitcoin::hashes::sha256d;
-use bitcoin::secp256k1::{Secp256k1, Verification};
-use bitcoin::{Address, Amount, PublicKey, Script, Transaction, Txid};
-
-use miniscript::descriptor::DescriptorPublicKey;
-use miniscript::{Descriptor, MiniscriptKey, TranslatePk};
-
-pub use bitcoincore_rpc::bitcoincore_rpc_json::AddressType;
-pub use bitcoincore_rpc::{Auth, Client as RpcClient, RpcApi};
-
-pub use electrum_client::{Client as ElectrumClient, ElectrumApi};
-
-// TODO: we currently only support env vars, we could also parse a toml file
-fn get_auth() -> Auth {
-    match env::var("MAGICAL_RPC_AUTH").as_ref().map(String::as_ref) {
-        Ok("USER_PASS") => Auth::UserPass(
-            env::var("MAGICAL_RPC_USER").unwrap(),
-            env::var("MAGICAL_RPC_PASS").unwrap(),
-        ),
-        _ => Auth::CookieFile(PathBuf::from(
-            env::var("MAGICAL_RPC_COOKIEFILE")
-                .unwrap_or("/home/user/.bitcoin/regtest/.cookie".to_string()),
-        )),
-    }
-}
-
-pub fn get_electrum_url() -> String {
-    env::var("MAGICAL_ELECTRUM_URL").unwrap_or("tcp://127.0.0.1:50001".to_string())
-}
-
-pub struct TestClient {
-    client: RpcClient,
-    electrum: ElectrumClient,
-}
-
-#[derive(Clone, Debug)]
-pub struct TestIncomingOutput {
-    pub value: u64,
-    pub to_address: String,
-}
-
-impl TestIncomingOutput {
-    pub fn new(value: u64, to_address: Address) -> Self {
-        Self {
-            value,
-            to_address: to_address.to_string(),
-        }
-    }
-}
-
-#[derive(Clone, Debug)]
-pub struct TestIncomingTx {
-    pub output: Vec<TestIncomingOutput>,
-    pub min_confirmations: Option<u64>,
-    pub locktime: Option<i64>,
-    pub replaceable: Option<bool>,
-}
-
-impl TestIncomingTx {
-    pub fn new(
-        output: Vec<TestIncomingOutput>,
-        min_confirmations: Option<u64>,
-        locktime: Option<i64>,
-        replaceable: Option<bool>,
-    ) -> Self {
-        Self {
-            output,
-            min_confirmations,
-            locktime,
-            replaceable,
-        }
-    }
-
-    pub fn add_output(&mut self, output: TestIncomingOutput) {
-        self.output.push(output);
-    }
-}
-
-#[doc(hidden)]
-pub trait TranslateDescriptor {
-    // derive and translate a `Descriptor<DescriptorPublicKey>` into a `Descriptor<PublicKey>`
-    fn derive_translated<C: Verification>(
-        &self,
-        secp: &Secp256k1<C>,
-        index: u32,
-    ) -> Descriptor<PublicKey>;
-}
-
-impl TranslateDescriptor for Descriptor<DescriptorPublicKey> {
-    fn derive_translated<C: Verification>(
-        &self,
-        secp: &Secp256k1<C>,
-        index: u32,
-    ) -> Descriptor<PublicKey> {
-        let translate = |key: &DescriptorPublicKey| -> PublicKey {
-            match key {
-                DescriptorPublicKey::XPub(xpub) => {
-                    xpub.xkey
-                        .derive_pub(secp, &xpub.derivation_path)
-                        .expect("hardened derivation steps")
-                        .public_key
-                }
-                DescriptorPublicKey::SinglePub(key) => key.key,
-            }
-        };
-
-        self.derive(index)
-            .translate_pk_infallible(|pk| translate(pk), |pkh| translate(pkh).to_pubkeyhash())
-    }
-}
-
-#[macro_export]
-macro_rules! testutils {
-    ( @external $descriptors:expr, $child:expr ) => ({
-        use bitcoin::secp256k1::Secp256k1;
-        use miniscript::descriptor::{Descriptor, DescriptorPublicKey, DescriptorTrait};
-
-        use $crate::TranslateDescriptor;
-
-        let secp = Secp256k1::new();
-
-        let parsed = Descriptor::<DescriptorPublicKey>::parse_descriptor(&secp, &$descriptors.0).expect("Failed to parse descriptor in `testutils!(@external)`").0;
-        parsed.derive_translated(&secp, $child).address(bitcoin::Network::Regtest).expect("No address form")
-    });
-    ( @internal $descriptors:expr, $child:expr ) => ({
-        use bitcoin::secp256k1::Secp256k1;
-        use miniscript::descriptor::{Descriptor, DescriptorPublicKey, DescriptorTrait};
-
-        use $crate::TranslateDescriptor;
-
-        let secp = Secp256k1::new();
-
-        let parsed = Descriptor::<DescriptorPublicKey>::parse_descriptor(&secp, &$descriptors.1.expect("Missing internal descriptor")).expect("Failed to parse descriptor in `testutils!(@internal)`").0;
-        parsed.derive_translated(&secp, $child).address(bitcoin::Network::Regtest).expect("No address form")
-    });
-    ( @e $descriptors:expr, $child:expr ) => ({ testutils!(@external $descriptors, $child) });
-    ( @i $descriptors:expr, $child:expr ) => ({ testutils!(@internal $descriptors, $child) });
-
-    ( @tx ( $( ( $( $addr:tt )* ) => $amount:expr ),+ ) $( ( @locktime $locktime:expr ) )* $( ( @confirmations $confirmations:expr ) )* $( ( @replaceable $replaceable:expr ) )* ) => ({
-        let mut outs = Vec::new();
-        $( outs.push(testutils::TestIncomingOutput::new($amount, testutils!( $($addr)* ))); )+
-
-        let mut locktime = None::<i64>;
-        $( locktime = Some($locktime); )*
-
-        let mut min_confirmations = None::<u64>;
-        $( min_confirmations = Some($confirmations); )*
-
-        let mut replaceable = None::<bool>;
-        $( replaceable = Some($replaceable); )*
-
-        testutils::TestIncomingTx::new(outs, min_confirmations, locktime, replaceable)
-    });
-
-    ( @literal $key:expr ) => ({
-        let key = $key.to_string();
-        (key, None::<String>, None::<String>)
-    });
-    ( @generate_xprv $( $external_path:expr )* $( ,$internal_path:expr )* ) => ({
-        use rand::Rng;
-
-        let mut seed = [0u8; 32];
-        rand::thread_rng().fill(&mut seed[..]);
-
-        let key = bitcoin::util::bip32::ExtendedPrivKey::new_master(
-            bitcoin::Network::Testnet,
-            &seed,
-        );
-
-        let mut external_path = None::<String>;
-        $( external_path = Some($external_path.to_string()); )*
-
-        let mut internal_path = None::<String>;
-        $( internal_path = Some($internal_path.to_string()); )*
-
-        (key.unwrap().to_string(), external_path, internal_path)
-    });
-    ( @generate_wif ) => ({
-        use rand::Rng;
-
-        let mut key = [0u8; bitcoin::secp256k1::constants::SECRET_KEY_SIZE];
-        rand::thread_rng().fill(&mut key[..]);
-
-        (bitcoin::PrivateKey {
-            compressed: true,
-            network: bitcoin::Network::Testnet,
-            key: bitcoin::secp256k1::SecretKey::from_slice(&key).unwrap(),
-        }.to_string(), None::<String>, None::<String>)
-    });
-
-    ( @keys ( $( $alias:expr => ( $( $key_type:tt )* ) ),+ ) ) => ({
-        let mut map = std::collections::HashMap::new();
-        $(
-            let alias: &str = $alias;
-            map.insert(alias, testutils!( $($key_type)* ));
-        )+
-
-        map
-    });
-
-    ( @descriptors ( $external_descriptor:expr ) $( ( $internal_descriptor:expr ) )* $( ( @keys $( $keys:tt )* ) )* ) => ({
-        use std::str::FromStr;
-        use std::collections::HashMap;
-        use std::convert::TryInto;
-
-        use miniscript::descriptor::{Descriptor, DescriptorPublicKey};
-        use miniscript::TranslatePk;
-
-        let mut keys: HashMap<&'static str, (String, Option<String>, Option<String>)> = HashMap::new();
-        $(
-            keys = testutils!{ @keys $( $keys )* };
-        )*
-
-        let external: Descriptor<String> = FromStr::from_str($external_descriptor).unwrap();
-        let external: Descriptor<String> = external.translate_pk_infallible::<_, _>(|k| {
-            if let Some((key, ext_path, _)) = keys.get(&k.as_str()) {
-                format!("{}{}", key, ext_path.as_ref().unwrap_or(&"".into()))
-            } else {
-                k.clone()
-            }
-        }, |kh| {
-            if let Some((key, ext_path, _)) = keys.get(&kh.as_str()) {
-                format!("{}{}", key, ext_path.as_ref().unwrap_or(&"".into()))
-            } else {
-                kh.clone()
-            }
-
-        });
-        let external = external.to_string();
-
-        let mut internal = None::<String>;
-        $(
-            let string_internal: Descriptor<String> = FromStr::from_str($internal_descriptor).unwrap();
-
-            let string_internal: Descriptor<String> = string_internal.translate_pk_infallible::<_, _>(|k| {
-                if let Some((key, _, int_path)) = keys.get(&k.as_str()) {
-                    format!("{}{}", key, int_path.as_ref().unwrap_or(&"".into()))
-                } else {
-                    k.clone()
-                }
-            }, |kh| {
-                if let Some((key, _, int_path)) = keys.get(&kh.as_str()) {
-                    format!("{}{}", key, int_path.as_ref().unwrap_or(&"".into()))
-                } else {
-                    kh.clone()
-                }
-            });
-            internal = Some(string_internal.to_string());
-        )*
-
-        (external, internal)
-    })
-}
-
-fn exponential_backoff_poll<T, F>(mut poll: F) -> T
-where
-    F: FnMut() -> Option<T>,
-{
-    let mut delay = Duration::from_millis(64);
-    loop {
-        match poll() {
-            Some(data) => break data,
-            None if delay.as_millis() < 512 => delay = delay.mul_f32(2.0),
-            None => {}
-        }
-
-        std::thread::sleep(delay);
-    }
-}
-
-impl TestClient {
-    pub fn new() -> Self {
-        let url = env::var("MAGICAL_RPC_URL").unwrap_or("127.0.0.1:18443".to_string());
-        let client = RpcClient::new(format!("http://{}", url), get_auth()).unwrap();
-        let electrum = ElectrumClient::new(&get_electrum_url()).unwrap();
-
-        TestClient { client, electrum }
-    }
-
-    fn wait_for_tx(&mut self, txid: Txid, monitor_script: &Script) {
-        // wait for electrs to index the tx
-        exponential_backoff_poll(|| {
-            trace!("wait_for_tx {}", txid);
-
-            self.electrum
-                .script_get_history(monitor_script)
-                .unwrap()
-                .iter()
-                .position(|entry| entry.tx_hash == txid)
-        });
-    }
-
-    fn wait_for_block(&mut self, min_height: usize) {
-        self.electrum.block_headers_subscribe().unwrap();
-
-        loop {
-            let header = exponential_backoff_poll(|| {
-                self.electrum.ping().unwrap();
-                self.electrum.block_headers_pop().unwrap()
-            });
-            if header.height >= min_height {
-                break;
-            }
-        }
-    }
-
-    pub fn receive(&mut self, meta_tx: TestIncomingTx) -> Txid {
-        assert!(
-            meta_tx.output.len() > 0,
-            "can't create a transaction with no outputs"
-        );
-
-        let mut map = HashMap::new();
-
-        let mut required_balance = 0;
-        for out in &meta_tx.output {
-            required_balance += out.value;
-            map.insert(out.to_address.clone(), Amount::from_sat(out.value));
-        }
-
-        if self.get_balance(None, None).unwrap() < Amount::from_sat(required_balance) {
-            panic!("Insufficient funds in bitcoind. Plase generate a few blocks with: `bitcoin-cli generatetoaddress 10 {}`", self.get_new_address(None, None).unwrap());
-        }
-
-        // FIXME: core can't create a tx with two outputs to the same address
-        let tx = self
-            .create_raw_transaction_hex(&[], &map, meta_tx.locktime, meta_tx.replaceable)
-            .unwrap();
-        let tx = self.fund_raw_transaction(tx, None, None).unwrap();
-        let mut tx: Transaction = deserialize(&tx.hex).unwrap();
-
-        if let Some(true) = meta_tx.replaceable {
-            // for some reason core doesn't set this field right
-            for input in &mut tx.input {
-                input.sequence = 0xFFFFFFFD;
-            }
-        }
-
-        let tx = self
-            .sign_raw_transaction_with_wallet(&serialize(&tx), None, None)
-            .unwrap();
-
-        // broadcast through electrum so that it caches the tx immediately
-        let txid = self
-            .electrum
-            .transaction_broadcast(&deserialize(&tx.hex).unwrap())
-            .unwrap();
-
-        if let Some(num) = meta_tx.min_confirmations {
-            self.generate(num, None);
-        }
-
-        let monitor_script = Address::from_str(&meta_tx.output[0].to_address)
-            .unwrap()
-            .script_pubkey();
-        self.wait_for_tx(txid, &monitor_script);
-
-        debug!("Sent tx: {}", txid);
-
-        txid
-    }
-
-    pub fn bump_fee(&mut self, txid: &Txid) -> Txid {
-        let tx = self.get_raw_transaction_info(txid, None).unwrap();
-        assert!(
-            tx.confirmations.is_none(),
-            "Can't bump tx {} because it's already confirmed",
-            txid
-        );
-
-        let bumped: serde_json::Value = self.call("bumpfee", &[txid.to_string().into()]).unwrap();
-        let new_txid = Txid::from_str(&bumped["txid"].as_str().unwrap().to_string()).unwrap();
-
-        let monitor_script =
-            tx.vout[0].script_pub_key.addresses.as_ref().unwrap()[0].script_pubkey();
-        self.wait_for_tx(new_txid, &monitor_script);
-
-        debug!("Bumped {}, new txid {}", txid, new_txid);
-
-        new_txid
-    }
-
-    pub fn generate_manually(&mut self, txs: Vec<Transaction>) -> String {
-        use bitcoin::blockdata::block::{Block, BlockHeader};
-        use bitcoin::blockdata::script::Builder;
-        use bitcoin::blockdata::transaction::{OutPoint, TxIn, TxOut};
-        use bitcoin::hash_types::{BlockHash, TxMerkleNode};
-
-        let block_template: serde_json::Value = self
-            .call("getblocktemplate", &[json!({"rules": ["segwit"]})])
-            .unwrap();
-        trace!("getblocktemplate: {:#?}", block_template);
-
-        let header = BlockHeader {
-            version: block_template["version"].as_i64().unwrap() as i32,
-            prev_blockhash: BlockHash::from_hex(
-                block_template["previousblockhash"].as_str().unwrap(),
-            )
-            .unwrap(),
-            merkle_root: TxMerkleNode::default(),
-            time: block_template["curtime"].as_u64().unwrap() as u32,
-            bits: u32::from_str_radix(block_template["bits"].as_str().unwrap(), 16).unwrap(),
-            nonce: 0,
-        };
-        debug!("header: {:#?}", header);
-
-        let height = block_template["height"].as_u64().unwrap() as i64;
-        let witness_reserved_value: Vec<u8> = sha256d::Hash::default().as_ref().into();
-        // burn block subsidy and fees, not a big deal
-        let mut coinbase_tx = Transaction {
-            version: 1,
-            lock_time: 0,
-            input: vec![TxIn {
-                previous_output: OutPoint::null(),
-                script_sig: Builder::new().push_int(height).into_script(),
-                sequence: 0xFFFFFFFF,
-                witness: vec![witness_reserved_value],
-            }],
-            output: vec![],
-        };
-
-        let mut txdata = vec![coinbase_tx.clone()];
-        txdata.extend_from_slice(&txs);
-
-        let mut block = Block { header, txdata };
-
-        let witness_root = block.witness_root();
-        let witness_commitment =
-            Block::compute_witness_commitment(&witness_root, &coinbase_tx.input[0].witness[0]);
-
-        // now update and replace the coinbase tx
-        let mut coinbase_witness_commitment_script = vec![0x6a, 0x24, 0xaa, 0x21, 0xa9, 0xed];
-        coinbase_witness_commitment_script.extend_from_slice(&witness_commitment);
-
-        coinbase_tx.output.push(TxOut {
-            value: 0,
-            script_pubkey: coinbase_witness_commitment_script.into(),
-        });
-        block.txdata[0] = coinbase_tx;
-
-        // set merkle root
-        let merkle_root = block.merkle_root();
-        block.header.merkle_root = merkle_root;
-
-        assert!(block.check_merkle_root());
-        assert!(block.check_witness_commitment());
-
-        // now do PoW :)
-        let target = block.header.target();
-        while block.header.validate_pow(&target).is_err() {
-            block.header.nonce = block.header.nonce.checked_add(1).unwrap(); // panic if we run out of nonces
-        }
-
-        let block_hex: String = serialize(&block).to_hex();
-        debug!("generated block hex: {}", block_hex);
-
-        self.electrum.block_headers_subscribe().unwrap();
-
-        let submit_result: serde_json::Value =
-            self.call("submitblock", &[block_hex.into()]).unwrap();
-        debug!("submitblock: {:?}", submit_result);
-        assert!(
-            submit_result.is_null(),
-            "submitblock error: {:?}",
-            submit_result.as_str()
-        );
-
-        self.wait_for_block(height as usize);
-
-        block.header.block_hash().to_hex()
-    }
-
-    pub fn generate(&mut self, num_blocks: u64, address: Option<Address>) {
-        let address = address.unwrap_or_else(|| self.get_new_address(None, None).unwrap());
-        let hashes = self.generate_to_address(num_blocks, &address).unwrap();
-        let best_hash = hashes.last().unwrap();
-        let height = self.get_block_info(best_hash).unwrap().height;
-
-        self.wait_for_block(height);
-
-        debug!("Generated blocks to new height {}", height);
-    }
-
-    pub fn invalidate(&mut self, num_blocks: u64) {
-        self.electrum.block_headers_subscribe().unwrap();
-
-        let best_hash = self.get_best_block_hash().unwrap();
-        let initial_height = self.get_block_info(&best_hash).unwrap().height;
-
-        let mut to_invalidate = best_hash;
-        for i in 1..=num_blocks {
-            trace!(
-                "Invalidating block {}/{} ({})",
-                i,
-                num_blocks,
-                to_invalidate
-            );
-
-            self.invalidate_block(&to_invalidate).unwrap();
-            to_invalidate = self.get_best_block_hash().unwrap();
-        }
-
-        self.wait_for_block(initial_height - num_blocks as usize);
-
-        debug!(
-            "Invalidated {} blocks to new height of {}",
-            num_blocks,
-            initial_height - num_blocks as usize
-        );
-    }
-
-    pub fn reorg(&mut self, num_blocks: u64) {
-        self.invalidate(num_blocks);
-        self.generate(num_blocks, None);
-    }
-
-    pub fn get_node_address(&self, address_type: Option<AddressType>) -> Address {
-        Address::from_str(
-            &self
-                .get_new_address(None, address_type)
-                .unwrap()
-                .to_string(),
-        )
-        .unwrap()
-    }
-}
-
-impl Deref for TestClient {
-    type Target = RpcClient;
-
-    fn deref(&self) -> &Self::Target {
-        &self.client
-    }
-}