Bump version to 0.22.0-rc.1

3451d1c12e Fix docs.rs features (Alekos Filini)

Pull request description:

  ### Description

  Fix docs.rs features

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### New Features:

  * [ ] I've added tests for the new feature
  * [ ] I've added docs for the new feature
  * [ ] I've updated `CHANGELOG.md`

ACKs for top commit:
  danielabrozzoni:
    utACK 3451d1c12e
  notmandatory:
    ACK 3451d1c12e

Tree-SHA512: 96dc4f816b21cf20fc2828dcfd56865f3f9add8e4aa643205879c810d09e6f2e73d6643061bf3ca98145b37e726907dedd31a145f92d6646c172a43ae2285aa8

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,26 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: 'bug'
+assignees: ''
+
+---
+
+**Describe the bug**  
+<!-- A clear and concise description of what the bug is. -->
+
+**To Reproduce**  
+<!-- Steps or code to reproduce the behavior. -->
+
+**Expected behavior**  
+<!-- A clear and concise description of what you expected to happen. -->
+
+**Build environment**  
+ - BDK tag/commit: <!-- e.g. v0.13.0, 3a07614 --> 
+ - OS+version: <!-- e.g. ubuntu 20.04.01, macOS 12.0.1, windows -->  
+ - Rust/Cargo version: <!-- e.g. 1.56.0 --> 
+ - Rust/Cargo target: <!-- e.g. x86_64-apple-darwin, x86_64-unknown-linux-gnu, etc. -->  
+
+**Additional context**  
+<!-- Add any other context about the problem here. --> 

.github/ISSUE_TEMPLATE/minor_release.md

@@ -0,0 +1,89 @@
+---
+name: Minor Release
+about: Create a new minor release [for release managers only]
+title: 'Release MAJOR.MINOR+1.0'
+labels: 'release'
+assignees: ''
+
+---
+
+## Create a new minor release
+
+### Summary
+
+<--release summary to be used in announcements-->
+
+### Commit
+
+<--latest commit ID to include in this release-->
+
+### Changelog
+
+<--add notices from PRs merged since the prior release, see ["keep a changelog"]-->
+
+### Checklist 
+
+Release numbering must follow [Semantic Versioning]. These steps assume the current `master` 
+branch **development** version is *MAJOR.MINOR.0*.
+
+#### On the day of the feature freeze
+
+Change the `master` branch to the next MINOR+1 version:
+
+- [ ] Switch to the `master` branch.
+- [ ] Create a new PR branch called `bump_dev_MAJOR_MINOR+1`, eg. `bump_dev_0_22`.
+- [ ] Bump the `bump_dev_MAJOR_MINOR+1` branch to the next development MINOR+1 version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR+1.0`.
+  - The commit message should be "Bump version to MAJOR.MINOR+1.0".
+- [ ] Create PR and merge the `bump_dev_MAJOR_MINOR+1` branch to `master`.
+  - Title PR "Bump version to MAJOR.MINOR+1.0".
+  
+Create a new release branch and release candidate tag:
+
+- [ ] Double check that your local `master` is up-to-date with the upstream repo.
+- [ ] Create a new branch called `release/MAJOR.MINOR+1` from `master`.
+- [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR+1` branch.
+  - The tag name should be `vMAJOR.MINOR+1.0-RC.1`
+  - Use message "Release MAJOR.MINOR+1.0 RC.1".  
+  - Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
+- [ ] Push the `release/MAJOR.MINOR` branch and new tag to the `bitcoindevkit/bdk` repo.
+  - Use `git push --tags` option to push the new `vMAJOR.MINOR+1.0-RC.1` tag.
+  
+If any issues need to be fixed before the *MAJOR.MINOR+1.0* version is released:
+
+- [ ] Merge fix PRs to the `master` branch.
+- [ ] Git cherry-pick fix commits to the `release/MAJOR.MINOR+1` branch.
+- [ ] Verify fixes in `release/MAJOR.MINOR+1` branch.
+- [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR+1` branch.
+  - The tag name should be `vMAJOR.MINOR+1.0-RC.x+1`, where x is the current release candidate number.
+  - Use tag message "Release MAJOR.MINOR+1.0 RC.x+1".  
+  - Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
+- [ ] Push the new tag to the `bitcoindevkit/bdk` repo.
+  - Use `git push --tags` option to push the new `vMAJOR.MINOR+1.0-RC.x+1` tag.
+
+#### On the day of the release
+
+Tag and publish new release:
+
+- [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR+1` branch.
+  - The tag name should be `vMAJOR.MINOR+1.0`
+  - The first line of the tag message should be "Release MAJOR.MINOR+1.0".
+  - In the body of the tag message put a copy of the **Summary** and **Changelog** for the release.
+  - Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
+- [ ] Wait for the CI to finish one last time.
+- [ ] Push the new tag to the `bitcoindevkit/bdk` repo.
+- [ ] Publish **all** the updated crates to crates.io.
+- [ ] Create the release on GitHub.
+  - Go to "tags", click on the dots on the right and select "Create Release".
+  - Set the title to `Release MAJOR.MINOR+1.0`.
+  - In the release notes body put the **Summary** and **Changelog**.
+  - Use the "+ Auto-generate release notes" button to add details from included PRs.
+  - Until we reach a `1.0.0` release check the "Pre-release" box.
+- [ ] Make sure the new release shows up on [crates.io] and that the docs are built correctly on [docs.rs].
+- [ ] Announce the release, using the **Summary**, on Discord, Twitter and Mastodon.
+- [ ] Celebrate 🎉
+
+[Semantic Versioning]: https://semver.org/
+[crates.io]: https://crates.io/crates/bdk
+[docs.rs]: https://docs.rs/bdk/latest/bdk
+["keep a changelog"]: https://keepachangelog.com/en/1.0.0/

.github/ISSUE_TEMPLATE/patch_release.md

@@ -0,0 +1,67 @@
+---
+name: Patch Release
+about: Create a new patch release [for release managers only]
+title: 'Release MAJOR.MINOR.PATCH+1'
+labels: 'release'
+assignees: ''
+
+---
+
+## Create a new patch release
+
+### Summary
+
+<--release summary to be used in announcements-->
+
+### Commit
+
+<--latest commit ID to include in this release-->
+
+### Changelog
+
+<--add notices from PRs merged since the prior release, see ["keep a changelog"]-->
+
+### Checklist  
+
+Release numbering must follow [Semantic Versioning]. These steps assume the current `master` 
+branch **development** version is *MAJOR.MINOR.PATCH*.
+
+### On the day of the patch release
+
+Change the `master` branch to the new PATCH+1 version:
+
+- [ ] Switch to the `master` branch.
+- [ ] Create a new PR branch called `bump_dev_MAJOR_MINOR_PATCH+1`, eg. `bump_dev_0_22_1`.
+- [ ] Bump the `bump_dev_MAJOR_MINOR` branch to the next development PATCH+1 version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR.PATCH+1`.
+  - The commit message should be "Bump version to MAJOR.MINOR.PATCH+1".
+- [ ] Create PR and merge the `bump_dev_MAJOR_MINOR_PATCH+1` branch to `master`.
+  - Title PR "Bump version to MAJOR.MINOR.PATCH+1".
+
+Cherry-pick, tag and publish new PATCH+1 release:
+
+- [ ] Merge fix PRs to the `master` branch.
+- [ ] Git cherry-pick fix commits to the `release/MAJOR.MINOR` branch to be patched.
+- [ ] Verify fixes in `release/MAJOR.MINOR` branch.
+- [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR` branch.
+  - The tag name should be `vMAJOR.MINOR.PATCH+1`
+  - The first line of the tag message should be "Release MAJOR.MINOR.PATCH+1".
+  - In the body of the tag message put a copy of the **Summary** and **Changelog** for the release.
+  - Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
+- [ ] Wait for the CI to finish one last time.
+- [ ] Push the new tag to the `bitcoindevkit/bdk` repo.
+- [ ] Publish **all** the updated crates to crates.io.
+- [ ] Create the release on GitHub.
+  - Go to "tags", click on the dots on the right and select "Create Release".
+  - Set the title to `Release MAJOR.MINOR.PATCH+1`.
+  - In the release notes body put the **Summary** and **Changelog**.
+  - Use the "+ Auto-generate release notes" button to add details from included PRs.
+  - Until we reach a `1.0.0` release check the "Pre-release" box.
+- [ ] Make sure the new release shows up on [crates.io] and that the docs are built correctly on [docs.rs].
+- [ ] Announce the release, using the **Summary**, on Discord, Twitter and Mastodon.
+- [ ] Celebrate 🎉
+
+[Semantic Versioning]: https://semver.org/
+[crates.io]: https://crates.io/crates/bdk
+[docs.rs]: https://docs.rs/bdk/latest/bdk
+["keep a changelog"]: https://keepachangelog.com/en/1.0.0/

.github/ISSUE_TEMPLATE/summer_project.md

@@ -0,0 +1,77 @@
+---
+name: Summer of Bitcoin Project
+about: Template to suggest a new https://www.summerofbitcoin.org/ project.
+title: ''
+labels: 'summer-of-bitcoin'
+assignees: ''
+
+---
+
+<!--
+## Overview  
+
+Project ideas are scoped for a university-level student with a basic background in CS and bitcoin 
+fundamentals - achievable over 12-weeks. Below are just a few types of ideas: 
+
+ - Low-hanging fruit: Relatively short projects with clear goals; requires basic technical knowledge 
+   and minimal familiarity with the codebase.
+ - Core development: These projects derive from the ongoing work from the core of your development 
+   team. The list of features and bugs is never-ending, and help is always welcome.
+ - Risky/Exploratory: These projects push the scope boundaries of your development effort. They 
+   might require expertise in an area not covered by your current development team. They might take 
+   advantage of a new technology. There is a reasonable chance that the project might be less 
+   successful, but the potential rewards make it worth the attempt.
+ - Infrastructure/Automation: These projects are the code that your organization uses to get its 
+   development work done; for example, projects that improve the automation of releases, regression 
+   tests and automated builds. This is a category where a Summer of Bitcoin student can be really 
+   helpful, doing work that the development team has been putting off while they focus on core 
+   development.
+ - Quality Assurance/Testing: Projects that work on and test your project's software development 
+   process. Additionally, projects that involve a thorough test and review of individual PRs.
+ - Fun/Peripheral: These projects might not be related to the current core development focus, but 
+   create new innovations and new perspectives for your project.
+-->
+
+**Description**
+<!-- Description: 3-7 sentences describing the project background and tasks to be done. -->  
+
+**Expected Outcomes**  
+<!-- Short bullet list describing what is to be accomplished -->   
+
+**Resources**  
+<!-- 2-3 reading materials for candidate to learn about the repo, project, scope etc -->  
+<!-- Recommended reading such as a developer/contributor guide -->  
+<!-- [Another example a paper citation](https://arxiv.org/pdf/1802.08091.pdf) -->  
+<!-- [Another example an existing issue](https://github.com/opencv/opencv/issues/11013) -->  
+<!-- [An existing related module](https://github.com/opencv/opencv_contrib/tree/master/modules/optflow) -->  
+
+**Skills Required**  
+<!-- 3-4 technical skills that the candidate should know -->  
+<!-- hands on experience with git -->  
+<!-- mastery plus experience coding in C++ -->  
+<!-- basic knowledge in matrix and tensor computations, college course work in cryptography -->  
+<!-- strong mathematical background -->
+<!-- Bonus - has experience with React Native. Best if you have also worked with OSSFuzz -->
+
+**Mentor(s)**  
+<!-- names of mentor(s) for this project go here -->
+
+**Difficulty**  
+<!-- Easy, Medium, Hard -->
+
+**Competency Test (optional)**  
+<!-- 2-3 technical tasks related to the project idea or repository you’d like a candidate to 
+    perform in order to demonstrate competency, good first bugs, warm-up exercises -->
+<!-- ex. Read the instructions here to get Bitcoin core running on your machine -->
+<!-- ex. pick an issue labeled as “newcomer” in the repository, and send a merge request to the 
+    repository. You can also suggest some other improvement that we did not think of yet, or 
+    something that you find interesting or useful -->
+<!-- ex. fixes for coding style are usually easy to do, and are good issues for first time 
+    contributions for those learning how to interact with the project. After you are done with the 
+    coding style issue, try making a different contribution. -->
+<!-- ex. setup a full Debian packaging development environment and learn the basics of Debian 
+    packaging. Then identify and package the missing dependencies to package Specter Desktop -->
+<!-- ex. write a pull parser for CSV files. You'll be judged by the decisions to store the parser 
+    state and how flexible it is to wrap this parser in other scenarios. -->
+<!-- ex. Stretch Goal: Implement some basic metaprogram/app to prove you're very familiar with BDK. 
+    Be prepared to make adjustments as we judge your solution. -->

.github/pull_request_template.md

@@ -9,6 +9,11 @@
 <!-- In this section you can include notes directed to the reviewers, like explaining why some parts
 of the PR were done in a specific way -->
 
+### Changelog notice
+
+<!-- Notice the release manager should include in the release tag message changelog -->
+<!-- See https://keepachangelog.com/en/1.0.0/ for examples -->
+
 ### Checklists
 
 #### All Submissions:
@@ -21,7 +26,6 @@ of the PR were done in a specific way -->
 
 * [ ] I've added tests for the new feature
 * [ ] I've added docs for the new feature
-* [ ] I've updated `CHANGELOG.md`
 
 #### Bugfixes:
 

.github/workflows/cont_integration.yml

@@ -10,8 +10,9 @@ jobs:
     strategy:
       matrix:
         rust:
-          - 1.56.0 # STABLE
-          - 1.46.0 # MSRV
+          - version: 1.60.0 # STABLE
+            clippy: true
+          - version: 1.56.1 # MSRV
         features:
           - default
           - minimal
@@ -27,11 +28,12 @@ jobs:
           - async-interface
           - use-esplora-reqwest
           - sqlite
+          - sqlite-bundled
     steps:
       - name: checkout
         uses: actions/checkout@v2
       - name: Generate cache key
-        run: echo "${{ matrix.rust }} ${{ matrix.features }}" | tee .cache_key
+        run: echo "${{ matrix.rust.version }} ${{ matrix.features }}" | tee .cache_key
       - name: cache
         uses: actions/cache@v2
         with:
@@ -41,16 +43,18 @@ jobs:
             target
           key: ${{ runner.os }}-cargo-${{ hashFiles('.cache_key') }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
       - name: Set default toolchain
-        run: rustup default ${{ matrix.rust }}
+        run: rustup default ${{ matrix.rust.version }}
       - name: Set profile
         run: rustup set profile minimal
       - name: Add clippy
+        if: ${{ matrix.rust.clippy }}
         run: rustup component add clippy
       - name: Update toolchain
         run: rustup update
       - name: Build
         run: cargo build --features ${{ matrix.features }} --no-default-features
       - name: Clippy
+        if: ${{ matrix.rust.clippy }}
         run: cargo clippy --all-targets --features ${{ matrix.features }} --no-default-features -- -D warnings
       - name: Test
         run: cargo test --features ${{ matrix.features }} --no-default-features
@@ -86,13 +90,20 @@ jobs:
       matrix:
         blockchain:
           - name: electrum
-            features: test-electrum
+            testprefix: blockchain::electrum::test
+            features: test-electrum,verify
           - name: rpc
+            testprefix: blockchain::rpc::test
             features: test-rpc
+          - name: rpc-legacy
+            testprefix: blockchain::rpc::test
+            features: test-rpc-legacy
           - name: esplora
-            features: test-esplora,use-esplora-reqwest
+            testprefix: esplora
+            features: test-esplora,use-esplora-reqwest,verify
           - name: esplora
-            features: test-esplora,use-esplora-ureq    
+            testprefix: esplora
+            features: test-esplora,use-esplora-ureq,verify
     steps:
       - name: Checkout
         uses: actions/checkout@v2
@@ -110,8 +121,8 @@ jobs:
           toolchain: stable
           override: true
       - name: Test
-        run: cargo test --no-default-features --features ${{ matrix.blockchain.features }} ${{ matrix.blockchain.name }}::bdk_blockchain_tests
-  
+        run: cargo test --no-default-features --features ${{ matrix.blockchain.features }} ${{ matrix.blockchain.testprefix }}::bdk_blockchain_tests
+
   check-wasm:
     name: Check WASM
     runs-on: ubuntu-20.04
@@ -135,7 +146,7 @@ jobs:
       - run: sudo apt-get update || exit 1
       - run: sudo apt-get install -y libclang-common-10-dev clang-10 libc6-dev-i386 || exit 1
       - name: Set default toolchain
-        run: rustup default 1.56.0 # STABLE
+        run: rustup default 1.56.1 # STABLE
       - name: Set profile
         run: rustup set profile minimal
       - name: Add target wasm32
@@ -161,3 +172,32 @@ jobs:
         run: rustup update
       - name: Check fmt
         run: cargo fmt --all -- --config format_code_in_doc_comments=true --check
+
+  test_harware_wallet:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        rust:
+          - version: 1.60.0 # STABLE
+          - version: 1.56.1 # MSRV
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v3
+    - name: Build simulator image
+      run: docker build -t hwi/ledger_emulator ./ci -f ci/Dockerfile.ledger
+    - name: Run simulator image
+      run: docker run --name simulator --network=host hwi/ledger_emulator &
+    - name: Install Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.9'
+    - name: Install python dependencies
+      run: pip install hwi==2.1.1 protobuf==3.20.1
+    - name: Set default toolchain
+      run: rustup default ${{ matrix.rust.version }}
+    - name: Set profile
+      run: rustup set profile minimal
+    - name: Update toolchain
+      run: rustup update
+    - name: Test
+      run: cargo test --features test-hardware-signer

.github/workflows/nightly_docs.yml

@@ -18,7 +18,7 @@ jobs:
             target
           key: nightly-docs-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
       - name: Set default toolchain
-        run: rustup default nightly
+        run: rustup default nightly-2022-01-25
       - name: Set profile
         run: rustup set profile minimal
       - name: Update toolchain
@@ -44,18 +44,18 @@ jobs:
           repository: bitcoindevkit/bitcoindevkit.org
           ref: master
       - name: Create directories
-        run: mkdir -p ./static/docs-rs/bdk/nightly
+        run: mkdir -p ./docs/.vuepress/public/docs-rs/bdk/nightly
       - name: Remove old latest
-        run: rm -rf ./static/docs-rs/bdk/nightly/latest
+        run: rm -rf ./docs/.vuepress/public/docs-rs/bdk/nightly/latest
       - name: Download built docs
         uses: actions/download-artifact@v1
         with:
           name: built-docs
-          path: ./static/docs-rs/bdk/nightly/latest
+          path: ./docs/.vuepress/public/docs-rs/bdk/nightly/latest
       - name: Configure git
         run: git config user.email "github-actions@github.com" && git config user.name "github-actions"
       - name: Commit
         continue-on-error: true         # If there's nothing to commit this step fails, but it's fine
-        run: git add ./static && git commit -m "Publish autogenerated nightly docs"
+        run: git add ./docs/.vuepress/public/docs-rs && git commit -m "Publish autogenerated nightly docs"
       - name: Push
         run: git push origin master

CHANGELOG.md

@@ -1,10 +1,99 @@
 # Changelog
-All notable changes to this project will be documented in this file.
+All notable changes to this project prior to release **0.22.0** are documented in this file. Future
+changelog information can be found in each release's git tag and can be viewed with `git tag -ln100 "v*"`.
+Changelog info is also documented on the [GitHub releases](https://github.com/bitcoindevkit/bdk/releases)
+page. See [DEVELOPMENT_CYCLE.md](DEVELOPMENT_CYCLE.md) for more details.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [v0.21.0] - [v0.20.0]
+
+- Add `descriptor::checksum::get_checksum_bytes` method.
+- Add `Excess` enum to handle remaining amount after coin selection.
+- Move change creation from `Wallet::create_tx` to `CoinSelectionAlgorithm::coin_select`.
+- Change the interface of `SqliteDatabase::new` to accept any type that implement AsRef<Path>
+- Add the ability to specify which leaves to sign in a taproot transaction through `TapLeavesOptions` in `SignOptions`
+- Add the ability to specify whether a taproot transaction should be signed using the internal key or not, using `sign_with_tap_internal_key` in `SignOptions`
+- Consolidate params `fee_amount` and `amount_needed` in `target_amount` in `CoinSelectionAlgorithm::coin_select` signature.
+- Change the meaning of the `fee_amount` field inside `CoinSelectionResult`: from now on the `fee_amount` will represent only the fees asociated with the utxos in the `selected` field of `CoinSelectionResult`.
+- New `RpcBlockchain` implementation with various fixes.
+- Return balance in separate categories, namely `confirmed`, `trusted_pending`, `untrusted_pending` & `immature`.
+
+## [v0.20.0] - [v0.19.0]
+
+- New MSRV set to `1.56.1`
+- Fee sniping discouraging through nLockTime - if the user specifies a `current_height`, we use that as a nlocktime, otherwise we use the last sync height (or 0 if we never synced)
+- Fix hang when `ElectrumBlockchainConfig::stop_gap` is zero.
+- Set coin type in BIP44, BIP49, and BIP84 templates
+- Get block hash given a block height - A `get_block_hash` method is now defined on the `GetBlockHash` trait and implemented on every blockchain backend. This method expects a block height and returns the corresponding block hash.
+- Add `remove_partial_sigs` and `try_finalize` to `SignOptions`
+- Deprecate `AddressValidator`
+- Fix Electrum wallet sync potentially causing address index decrement - compare proposed index and current index before applying batch operations during sync.
+
+## [v0.19.0] - [v0.18.0]
+
+- added `OldestFirstCoinSelection` impl to `CoinSelectionAlgorithm`
+- New MSRV set to `1.56`
+- Unpinned tokio to `1`
+- Add traits to reuse `Blockchain`s across multiple wallets (`BlockchainFactory` and `StatelessBlockchain`).
+- Upgrade to rust-bitcoin `0.28`
+- If using the `sqlite-db` feature all cached wallet data is deleted due to a possible UTXO inconsistency, a wallet.sync will recreate it
+- Update `PkOrF` in the policy module to become an enum
+- Add experimental support for Taproot, including:
+  - Support for `tr()` descriptors with complex tapscript trees
+  - Creation of Taproot PSBTs (BIP-371)
+  - Signing Taproot PSBTs (key spend and script spend)
+  - Support for `tr()` descriptors in the `descriptor!()` macro
+- Add support for Bitcoin Core 23.0 when using the `rpc` blockchain
+
+## [v0.18.0] - [v0.17.0]
+
+- Add `sqlite-bundled` feature for deployments that need a bundled version of sqlite, i.e. for mobile platforms.
+- Added `Wallet::get_signers()`, `Wallet::descriptor_checksum()` and `Wallet::get_address_validators()`, exposed the `AsDerived` trait.
+- Deprecate `database::Database::flush()`, the function is only needed for the sled database on mobile, instead for mobile use the sqlite database.
+- Add `keychain: KeychainKind` to `wallet::AddressInfo`.
+- Improve key generation traits
+- Rename `WalletExport` to `FullyNodedExport`, deprecate the former.
+- Bump `miniscript` dependency version to `^6.1`.
+
+## [v0.17.0] - [v0.16.1]
+
+- Removed default verification from `wallet::sync`. sync-time verification is added in `script_sync` and is activated by `verify` feature flag.
+- `verify` flag removed from `TransactionDetails`.
+- Add `get_internal_address` to allow you to get internal addresses just as you get external addresses.
+- added `ensure_addresses_cached` to `Wallet` to let offline wallets load and cache addresses in their database
+- Add `is_spent` field to `LocalUtxo`; when we notice that a utxo has been spent we set `is_spent` field to true instead of deleting it from the db.
+
+### Sync API change
+
+To decouple the `Wallet` from the `Blockchain` we've made major changes:
+
+- Removed `Blockchain` from Wallet.
+- Removed `Wallet::broadcast` (just use `Blockchain::broadcast`)
+- Deprecated `Wallet::new_offline` (all wallets are offline now)
+- Changed `Wallet::sync` to take a `Blockchain`.
+- Stop making a request for the block height when calling `Wallet:new`.
+- Added `SyncOptions` to capture extra (future) arguments to `Wallet::sync`.
+- Removed `max_addresses` sync parameter which determined how many addresses to cache before syncing since this can just be done with `ensure_addresses_cached`.
+- remove `flush` method from the `Database` trait.
+
+## [v0.16.1] - [v0.16.0]
+
+- Pin tokio dependency version to ~1.14 to prevent errors due to their new MSRV 1.49.0
+
+## [v0.16.0] - [v0.15.0]
+
+- Disable `reqwest` default features.
+- Added `reqwest-default-tls` feature: Use this to restore the TLS defaults of reqwest if you don't want to add a dependency to it in your own manifest.
+- Use dust_value from rust-bitcoin
+- Fixed generating WIF in the correct network format.
+
+## [v0.15.0] - [v0.14.0]
+
+- Overhauled sync logic for electrum and esplora.
+- Unify ureq and reqwest esplora backends to have the same configuration parameters. This means reqwest now has a timeout parameter and ureq has a concurrency parameter.
+- Fixed esplora fee estimation.
 
 ## [v0.14.0] - [v0.13.0]
 
@@ -381,7 +470,6 @@ final transaction is created by calling `finish` on the builder.
 - Use `MemoryDatabase` in the compiler example
 - Make the REPL return JSON
 
-[unreleased]: https://github.com/bitcoindevkit/bdk/compare/v0.11.0...HEAD
 [0.1.0-beta.1]: https://github.com/bitcoindevkit/bdk/compare/96c87ea5...0.1.0-beta.1
 [v0.2.0]: https://github.com/bitcoindevkit/bdk/compare/0.1.0-beta.1...v0.2.0
 [v0.3.0]: https://github.com/bitcoindevkit/bdk/compare/v0.2.0...v0.3.0
@@ -397,3 +485,11 @@ final transaction is created by calling `finish` on the builder.
 [v0.12.0]: https://github.com/bitcoindevkit/bdk/compare/v0.11.0...v0.12.0
 [v0.13.0]: https://github.com/bitcoindevkit/bdk/compare/v0.12.0...v0.13.0
 [v0.14.0]: https://github.com/bitcoindevkit/bdk/compare/v0.13.0...v0.14.0
+[v0.15.0]: https://github.com/bitcoindevkit/bdk/compare/v0.14.0...v0.15.0
+[v0.16.0]: https://github.com/bitcoindevkit/bdk/compare/v0.15.0...v0.16.0
+[v0.16.1]: https://github.com/bitcoindevkit/bdk/compare/v0.16.0...v0.16.1
+[v0.17.0]: https://github.com/bitcoindevkit/bdk/compare/v0.16.1...v0.17.0
+[v0.18.0]: https://github.com/bitcoindevkit/bdk/compare/v0.17.0...v0.18.0
+[v0.19.0]: https://github.com/bitcoindevkit/bdk/compare/v0.18.0...v0.19.0
+[v0.20.0]: https://github.com/bitcoindevkit/bdk/compare/v0.19.0...v0.20.0
+[v0.21.0]: https://github.com/bitcoindevkit/bdk/compare/v0.20.0...v0.21.0

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "bdk"
-version = "0.14.0"
+version = "0.22.0-rc.1"
 edition = "2018"
 authors = ["Alekos Filini <alekos.filini@gmail.com>", "Riccardo Casatta <riccardo@casatta.it>"]
 homepage = "https://bitcoindevkit.org"
@@ -14,18 +14,18 @@ license = "MIT OR Apache-2.0"
 [dependencies]
 bdk-macros = "^0.6"
 log = "^0.4"
-miniscript = { version = "^6.0", features = ["use-serde"] }
-bitcoin = { version = "^0.27", features = ["use-serde", "base64"] }
+miniscript = { version = "7.0", features = ["use-serde"] }
+bitcoin = { version = "0.28.1", features = ["use-serde", "base64", "rand"] }
 serde = { version = "^1.0", features = ["derive"] }
 serde_json = { version = "^1.0" }
 rand = "^0.7"
 
 # Optional dependencies
 sled = { version = "0.34", optional = true }
-electrum-client = { version = "0.8", optional = true }
-rusqlite = { version = "0.25.3", optional = true }
-ahash = { version = "=0.7.4", optional = true }
-reqwest = { version = "0.11", optional = true, features = ["json"] }
+electrum-client = { version = "0.10", optional = true }
+rusqlite = { version = "0.27.0", optional = true }
+ahash = { version = "0.7.6", optional = true }
+reqwest = { version = "0.11", optional = true, default-features = false, features = ["json"] }
 ureq = { version = "~2.2.0", features = ["json"], optional = true }
 futures = { version = "0.3", optional = true }
 async-trait = { version = "0.1", optional = true }
@@ -33,12 +33,13 @@ rocksdb = { version = "0.14", default-features = false, features = ["snappy"], o
 cc = { version = ">=1.0.64", optional = true }
 socks = { version = "0.3", optional = true }
 lazy_static = { version = "1.4", optional = true }
+hwi = { version = "0.2.2", optional = true }
 
 bip39 = { version = "1.0.1", optional = true }
 bitcoinconsensus = { version = "0.19.0-3", optional = true }
 
-# Needed by bdk_blockchain_tests macro
-bitcoincore-rpc = { version = "0.14", optional = true }
+# Needed by bdk_blockchain_tests macro and the `rpc` feature
+bitcoincore-rpc = { version = "0.15", optional = true }
 
 # Platform-specific dependencies
 [target.'cfg(not(target_arch = "wasm32"))'.dependencies]
@@ -55,11 +56,13 @@ compiler = ["miniscript/compiler"]
 verify = ["bitcoinconsensus"]
 default = ["key-value-db", "electrum"]
 sqlite = ["rusqlite", "ahash"]
+sqlite-bundled = ["sqlite", "rusqlite/bundled"]
 compact_filters = ["rocksdb", "socks", "lazy_static", "cc"]
 key-value-db = ["sled"]
 all-keys = ["keys-bip39"]
 keys-bip39 = ["bip39"]
 rpc = ["bitcoincore-rpc"]
+hardware-signer = ["hwi"]
 
 # We currently provide mulitple implementations of `Blockchain`, all are
 # blocking except for the `EsploraBlockchain` which can be either async or
@@ -82,19 +85,23 @@ use-esplora-ureq = ["esplora", "ureq", "ureq/socks"]
 # Typical configurations will not need to use `esplora` feature directly.
 esplora = []
 
+# Use below feature with `use-esplora-reqwest` to enable reqwest default TLS support
+reqwest-default-tls = ["reqwest/default-tls"]
 
 # Debug/Test features
 test-blockchains = ["bitcoincore-rpc", "electrum-client"]
-test-electrum = ["electrum", "electrsd/electrs_0_8_10", "test-blockchains"]
-test-rpc = ["rpc", "electrsd/electrs_0_8_10", "test-blockchains"]
-test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]
+test-electrum = ["electrum", "electrsd/electrs_0_8_10", "electrsd/bitcoind_22_0", "test-blockchains"]
+test-rpc = ["rpc", "electrsd/electrs_0_8_10", "electrsd/bitcoind_22_0", "test-blockchains"]
+test-rpc-legacy = ["rpc", "electrsd/electrs_0_8_10", "electrsd/bitcoind_0_20_0", "test-blockchains"]
+test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "electrsd/bitcoind_22_0", "test-blockchains"]
 test-md-docs = ["electrum"]
+test-hardware-signer = ["hardware-signer"]
 
 [dev-dependencies]
 lazy_static = "1.4"
 env_logger = "0.7"
 clap = "2.33"
-electrsd = { version= "0.12", features = ["trigger", "bitcoind_22_0"] }
+electrsd = "0.19.1"
 
 [[example]]
 name = "address_validator"
@@ -107,9 +114,14 @@ name = "miniscriptc"
 path = "examples/compiler.rs"
 required-features = ["compiler"]
 
+[[example]]
+name = "rpcwallet"
+path = "examples/rpcwallet.rs"
+required-features = ["keys-bip39", "key-value-db", "rpc", "electrsd/bitcoind_22_0"]
+
 [workspace]
 members = ["macros"]
 [package.metadata.docs.rs]
-features = ["compiler", "electrum", "esplora", "ureq", "compact_filters", "rpc", "key-value-db", "sqlite", "all-keys", "verify"]
+features = ["compiler", "electrum", "esplora", "use-esplora-ureq", "compact_filters", "rpc", "key-value-db", "sqlite", "all-keys", "verify", "hardware-signer"]
 # defines the configuration attribute `docsrs`
 rustdoc-args = ["--cfg", "docsrs"]

DEVELOPMENT_CYCLE.md

@@ -1,46 +1,16 @@
 # Development Cycle
 
-This project follows a regular releasing schedule similar to the one [used by the Rust language](https://doc.rust-lang.org/book/appendix-07-nightly-rust.html). In short, this means that a new release is made at a regular cadence, with all the feature/bugfixes that made it to `master` in time. This ensures that we don't keep delaying releases waiting for "just one more little thing".
+This project follows a regular releasing schedule similar to the one [used by the Rust language]. In short, this means that a new release is made at a regular cadence, with all the feature/bugfixes that made it to `master` in time. This ensures that we don't keep delaying releases waiting for "just one more little thing".
+
+This project uses [Semantic Versioning], but is currently at MAJOR version zero (0.y.z) meaning it is still in initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable. Until we reach version `1.0.0` we will do our best to document any breaking API changes in the changelog info attached to each release tag.
 
 We decided to maintain a faster release cycle while the library is still in "beta", i.e. before release `1.0.0`: since we are constantly adding new features and, even more importantly, fixing issues, we want developers to have access to those updates as fast as possible. For this reason we will make a release **every 4 weeks**.
 
-Once the project will have reached a more mature state (>= `1.0.0`), we will very likely switch to longer release cycles of **6 weeks**.
+Once the project reaches a more mature state (>= `1.0.0`), we will very likely switch to longer release cycles of **6 weeks**.
 
 The "feature freeze" will happen **one week before the release date**. This means a new branch will be created originating from the `master` tip at that time, and in that branch we will stop adding new features and only focus on ensuring the ones we've added are working properly.
 
-```
-master:           - - - - * - - - * - - - - - - * - - - * ...
-                          |      /              |       |
-release/0.x.0:            * - - #               |       |
-                                                |      /
-release/0.y.0:                                  * - - #
-```
+To create a new release a release manager will create a new issue using the `Release` template and follow the template instructions.
 
-As soon as the release is tagged and published, the `release` branch will be merged back into `master` to update the version in the `Cargo.toml` to apply the new `Cargo.toml` version and all the other fixes made during the feature freeze window.
-
-## Making the Release
-
-What follows are notes and procedures that maintainers can refer to when making releases. All the commits and tags must be signed and, ideally, also [timestamped](https://github.com/opentimestamps/opentimestamps-client/blob/master/doc/git-integration.md).
-
-Pre-`v1.0.0` our "major" releases only affect the "minor" semver value. Accordingly, our "minor" releases will only affect the "patch" value.
-
-1. Create a new branch called `release/x.y.z` from `master`. Double check that your local `master` is up-to-date with the upstream repo before doing so.
-2. Make a commit on the release branch to bump the version to `x.y.z-rc.1`. The message should be "Bump version to x.y.z-rc.1".
-3. Push the new branch to `bitcoindevkit/bdk` on GitHub.
-4. During the one week of feature freeze run additional tests on the release branch.
-5. If a bug is found:
-    - If it's a minor issue you can just fix it in the release branch, since it will be merged back to `master` eventually
-    - For bigger issues you can fix them on `master` and then *cherry-pick* the commit to the release branch
-6. Update the changelog with the new release version.
-7. Update `src/lib.rs` with the new version (line ~43)
-8. On release day, make a commit on the release branch to bump the version to `x.y.z`. The message should be "Bump version to x.y.z".
-9. Add a tag to this commit. The tag name should be `vx.y.z` (for example `v0.5.0`), and the message "Release x.y.z". Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
-10. Push the new commits to the upstream release branch, wait for the CI to finish one last time.
-11. Publish **all** the updated crates to crates.io.
-12. Make a new commit to bump the version value to `x.y.(z+1)-dev`. The message should be "Bump version to x.y.(z+1)-dev".
-13. Merge the release branch back into `master`.
-14. If the `master` branch contains any unreleased changes to the `bdk-macros` crate, change the `bdk` Cargo.toml `[dependencies]` to point to the local path (ie. `bdk-macros = { path = "./macros"}`)
-15. Create the release on GitHub: go to "tags", click on the dots on the right and select "Create Release". Then set the title to `vx.y.z` and write down some brief release notes.
-16. Make sure the new release shows up on crates.io and that the docs are built correctly on docs.rs.
-17. Announce the release on Twitter, Discord and Telegram.
-18. Celebrate :tada:
+[used by the Rust language]: https://doc.rust-lang.org/book/appendix-07-nightly-rust.html
+[Semantic Versioning]: https://semver.org/

README.md

@@ -1,7 +1,7 @@
 <div align="center">
   <h1>BDK</h1>
 
-  <img src="./static/bdk.svg" width="220" />
+  <img src="./static/bdk.png" width="220" />
 
   <p>
     <strong>A modern, lightweight, descriptor-based wallet library written in Rust!</strong>
@@ -13,7 +13,7 @@
     <a href="https://github.com/bitcoindevkit/bdk/actions?query=workflow%3ACI"><img alt="CI Status" src="https://github.com/bitcoindevkit/bdk/workflows/CI/badge.svg"></a>
     <a href="https://codecov.io/gh/bitcoindevkit/bdk"><img src="https://codecov.io/gh/bitcoindevkit/bdk/branch/master/graph/badge.svg"/></a>
     <a href="https://docs.rs/bdk"><img alt="API Docs" src="https://img.shields.io/badge/docs.rs-bdk-green"/></a>
-    <a href="https://blog.rust-lang.org/2020/08/27/Rust-1.46.0.html"><img alt="Rustc Version 1.46+" src="https://img.shields.io/badge/rustc-1.46%2B-lightgrey.svg"/></a>
+    <a href="https://blog.rust-lang.org/2021/11/01/Rust-1.56.1.html"><img alt="Rustc Version 1.56.1+" src="https://img.shields.io/badge/rustc-1.56.1%2B-lightgrey.svg"/></a>
     <a href="https://discord.gg/d7NkDKm"><img alt="Chat on Discord" src="https://img.shields.io/discord/753336465005608961?logo=discord"></a>
   </p>
 
@@ -41,21 +41,21 @@ The `bdk` library aims to be the core building block for Bitcoin wallets of any
 ```rust,no_run
 use bdk::Wallet;
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::{noop_progress, ElectrumBlockchain};
-
+use bdk::blockchain::ElectrumBlockchain;
+use bdk::SyncOptions;
 use bdk::electrum_client::Client;
+use bdk::bitcoin::Network;
 
 fn main() -> Result<(), bdk::Error> {
-    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let blockchain = ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
-        bitcoin::Network::Testnet,
+        Network::Testnet,
         MemoryDatabase::default(),
-        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(noop_progress(), None)?;
+    wallet.sync(&blockchain, SyncOptions::default())?;
 
     println!("Descriptor balance: {} SAT", wallet.get_balance()?);
 
@@ -70,7 +70,7 @@ use bdk::{Wallet, database::MemoryDatabase};
 use bdk::wallet::AddressIndex::New;
 
 fn main() -> Result<(), bdk::Error> {
-    let wallet = Wallet::new_offline(
+    let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
@@ -88,26 +88,26 @@ fn main() -> Result<(), bdk::Error> {
 ### Create a transaction
 
 ```rust,no_run
-use bdk::{FeeRate, Wallet};
+use bdk::{FeeRate, Wallet, SyncOptions};
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::{noop_progress, ElectrumBlockchain};
+use bdk::blockchain::ElectrumBlockchain;
 
 use bdk::electrum_client::Client;
 use bdk::wallet::AddressIndex::New;
 
+use bitcoin::base64;
 use bitcoin::consensus::serialize;
 
 fn main() -> Result<(), bdk::Error> {
-    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let blockchain = ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
-        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(noop_progress(), None)?;
+    wallet.sync(&blockchain, SyncOptions::default())?;
 
     let send_to = wallet.get_address(New)?;
     let (psbt, details) = {
@@ -132,10 +132,11 @@ fn main() -> Result<(), bdk::Error> {
 ```rust,no_run
 use bdk::{Wallet, SignOptions, database::MemoryDatabase};
 
+use bitcoin::base64;
 use bitcoin::consensus::deserialize;
 
 fn main() -> Result<(), bdk::Error> {
-    let wallet = Wallet::new_offline(
+    let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
         bitcoin::Network::Testnet,
@@ -155,7 +156,7 @@ fn main() -> Result<(), bdk::Error> {
 
 ### Unit testing
 
-```
+```bash
 cargo test
 ```
 
@@ -163,11 +164,11 @@ cargo test
 
 Integration testing require testing features, for example:
 
-```
+```bash
 cargo test --features test-electrum
 ```
 
-The other options are `test-esplora` or `test-rpc`.
+The other options are `test-esplora`, `test-rpc` or `test-rpc-legacy` which runs against an older version of Bitcoin Core.
 Note that `electrs` and `bitcoind` binaries are automatically downloaded (on mac and linux), to specify you already have installed binaries you must use `--no-default-features` and provide `BITCOIND_EXE` and `ELECTRS_EXE` as environment variables.
 
 ## License

ci/Dockerfile.ledger

@@ -0,0 +1,9 @@
+# Taken from bitcoindevkit/rust-hwi
+FROM ghcr.io/ledgerhq/speculos
+
+RUN apt-get update
+RUN apt-get install wget -y
+RUN wget "https://github.com/LedgerHQ/speculos/blob/master/apps/nanos%23btc%232.1%231c8db8da.elf?raw=true" -O /speculos/btc.elf
+ADD automation.json /speculos/automation.json
+
+ENTRYPOINT ["python", "./speculos.py", "--automation", "file:automation.json", "--display", "headless", "--vnc-port", "41000", "btc.elf"]

ci/automation.json

@@ -0,0 +1,30 @@
+{
+    "version": 1,
+    "rules": [
+        {
+            "regexp": "Address \\(\\d/\\d\\)|Message hash \\(\\d/\\d\\)|Confirm|Fees|Review|Amount",
+            "actions": [
+                [ "button", 2, true ],
+                [ "button", 2, false ]
+            ]
+        },
+        {
+            "text": "Sign",
+            "conditions": [
+                [ "seen", false ]
+            ],
+            "actions": [
+                [ "button", 2, true ],
+                [ "button", 2, false ],
+                [ "setbool", "seen", true ]
+            ]
+        },
+        {
+            "regexp": "Approve|Sign|Accept",
+            "actions": [
+                [ "button", 3, true ],
+                [ "button", 3, false ]
+            ]
+        }
+    ]
+}

examples/address_validator.rs

@@ -14,6 +14,7 @@ use std::sync::Arc;
 use bdk::bitcoin;
 use bdk::database::MemoryDatabase;
 use bdk::descriptor::HdKeyPaths;
+#[allow(deprecated)]
 use bdk::wallet::address_validator::{AddressValidator, AddressValidatorError};
 use bdk::KeychainKind;
 use bdk::Wallet;
@@ -25,6 +26,7 @@ use bitcoin::{Network, Script};
 
 #[derive(Debug)]
 struct DummyValidator;
+#[allow(deprecated)]
 impl AddressValidator for DummyValidator {
     fn validate(
         &self,
@@ -48,9 +50,9 @@ impl AddressValidator for DummyValidator {
 
 fn main() -> Result<(), bdk::Error> {
     let descriptor = "sh(and_v(v:pk(tpubDDpWvmUrPZrhSPmUzCMBHffvC3HyMAPnWDSAQNBTnj1iZeJa7BZQEttFiP4DS4GCcXQHezdXhn86Hj6LHX5EDstXPWrMaSneRWM8yUf6NFd/*),after(630000)))";
-    let mut wallet =
-        Wallet::new_offline(descriptor, None, Network::Regtest, MemoryDatabase::new())?;
+    let mut wallet = Wallet::new(descriptor, None, Network::Regtest, MemoryDatabase::new())?;
 
+    #[allow(deprecated)]
     wallet.add_address_validator(Arc::new(DummyValidator));
 
     wallet.get_address(New)?;

examples/compact_filters_balance.rs

@@ -10,7 +10,6 @@
 // licenses.
 
 use bdk::blockchain::compact_filters::*;
-use bdk::blockchain::noop_progress;
 use bdk::database::MemoryDatabase;
 use bdk::*;
 use bitcoin::*;
@@ -35,9 +34,8 @@ fn main() -> Result<(), CompactFiltersError> {
     let descriptor = "wpkh(tpubD6NzVbkrYhZ4X2yy78HWrr1M9NT8dKeWfzNiQqDdMqqa9UmmGztGGz6TaLFGsLfdft5iu32gxq1T4eMNxExNNWzVCpf9Y6JZi5TnqoC9wJq/*)";
 
     let database = MemoryDatabase::default();
-    let wallet =
-        Arc::new(Wallet::new(descriptor, None, Network::Testnet, database, blockchain).unwrap());
-    wallet.sync(noop_progress(), None).unwrap();
+    let wallet = Arc::new(Wallet::new(descriptor, None, Network::Testnet, database).unwrap());
+    wallet.sync(&blockchain, SyncOptions::default()).unwrap();
     info!("balance: {}", wallet.get_balance()?);
     Ok(())
 }

examples/compiler.rs

@@ -85,11 +85,11 @@ fn main() -> Result<(), Box<dyn Error>> {
 
     let network = matches
         .value_of("network")
-        .map(|n| Network::from_str(n))
+        .map(Network::from_str)
         .transpose()
         .unwrap()
         .unwrap_or(Network::Testnet);
-    let wallet = Wallet::new_offline(&format!("{}", descriptor), None, network, database)?;
+    let wallet = Wallet::new(&format!("{}", descriptor), None, network, database)?;
 
     info!("... First address: {}", wallet.get_address(New)?);
 

examples/rpcwallet.rs

@@ -0,0 +1,229 @@
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use bdk::bitcoin::secp256k1::Secp256k1;
+use bdk::bitcoin::Amount;
+use bdk::bitcoin::Network;
+use bdk::bitcoincore_rpc::RpcApi;
+
+use bdk::blockchain::rpc::{Auth, RpcBlockchain, RpcConfig};
+use bdk::blockchain::ConfigurableBlockchain;
+
+use bdk::keys::bip39::{Language, Mnemonic, WordCount};
+use bdk::keys::{DerivableKey, GeneratableKey, GeneratedKey};
+
+use bdk::miniscript::miniscript::Segwitv0;
+
+use bdk::sled;
+use bdk::template::Bip84;
+use bdk::wallet::{signer::SignOptions, wallet_name_from_descriptor, AddressIndex, SyncOptions};
+use bdk::KeychainKind;
+use bdk::Wallet;
+
+use bdk::blockchain::Blockchain;
+
+use electrsd;
+
+use std::error::Error;
+use std::path::PathBuf;
+use std::str::FromStr;
+
+/// This example demonstrates a typical way to create a wallet and work with bdk.
+///
+/// This example bdk wallet is connected to a bitcoin core rpc regtest node,
+/// and will attempt to receive, create and broadcast transactions.
+///
+/// To start a bitcoind regtest node programmatically, this example uses
+/// `electrsd` library, which is also a bdk dev-dependency.
+///
+/// But you can start your own bitcoind backend, and the rest of the example should work fine.
+
+fn main() -> Result<(), Box<dyn Error>> {
+    // -- Setting up background bitcoind process
+
+    println!(">> Setting up bitcoind");
+
+    // Start the bitcoind process
+    let bitcoind_conf = electrsd::bitcoind::Conf::default();
+
+    // electrsd will automatically download the bitcoin core binaries
+    let bitcoind_exe =
+        electrsd::bitcoind::downloaded_exe_path().expect("We should always have downloaded path");
+
+    // Launch bitcoind and gather authentication access
+    let bitcoind = electrsd::bitcoind::BitcoinD::with_conf(bitcoind_exe, &bitcoind_conf).unwrap();
+    let bitcoind_auth = Auth::Cookie {
+        file: bitcoind.params.cookie_file.clone(),
+    };
+
+    // Get a new core address
+    let core_address = bitcoind.client.get_new_address(None, None)?;
+
+    // Generate 101 blocks and use the above address as coinbase
+    bitcoind.client.generate_to_address(101, &core_address)?;
+
+    println!(">> bitcoind setup complete");
+    println!(
+        "Available coins in Core wallet : {}",
+        bitcoind.client.get_balance(None, None)?
+    );
+
+    // -- Setting up the Wallet
+
+    println!("\n>> Setting up BDK wallet");
+
+    // Get a random private key
+    let xprv = generate_random_ext_privkey()?;
+
+    // Use the derived descriptors from the privatekey to
+    // create unique wallet name.
+    // This is a special utility function exposed via `bdk::wallet_name_from_descriptor()`
+    let wallet_name = wallet_name_from_descriptor(
+        Bip84(xprv.clone(), KeychainKind::External),
+        Some(Bip84(xprv.clone(), KeychainKind::Internal)),
+        Network::Regtest,
+        &Secp256k1::new(),
+    )?;
+
+    // Create a database (using default sled type) to store wallet data
+    let mut datadir = PathBuf::from_str("/tmp/")?;
+    datadir.push(".bdk-example");
+    let database = sled::open(datadir)?;
+    let database = database.open_tree(wallet_name.clone())?;
+
+    // Create a RPC configuration of the running bitcoind backend we created in last step
+    // Note: If you are using custom regtest node, use the appropriate url and auth
+    let rpc_config = RpcConfig {
+        url: bitcoind.params.rpc_socket.to_string(),
+        auth: bitcoind_auth,
+        network: Network::Regtest,
+        wallet_name,
+        sync_params: None,
+    };
+
+    // Use the above configuration to create a RPC blockchain backend
+    let blockchain = RpcBlockchain::from_config(&rpc_config)?;
+
+    // Combine Database + Descriptor to create the final wallet
+    let wallet = Wallet::new(
+        Bip84(xprv.clone(), KeychainKind::External),
+        Some(Bip84(xprv.clone(), KeychainKind::Internal)),
+        Network::Regtest,
+        database,
+    )?;
+
+    // The `wallet` and the `blockchain` are independent structs.
+    // The wallet will be used to do all wallet level actions
+    // The blockchain can be used to do all blockchain level actions.
+    // For certain actions (like sync) the wallet will ask for a blockchain.
+
+    // Sync the wallet
+    // The first sync is important as this will instantiate the
+    // wallet files.
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> BDK wallet setup complete.");
+    println!(
+        "Available initial coins in BDK wallet : {} sats",
+        wallet.get_balance()?
+    );
+
+    // -- Wallet transaction demonstration
+
+    println!("\n>> Sending coins: Core --> BDK, 10 BTC");
+    // Get a new address to receive coins
+    let bdk_new_addr = wallet.get_address(AddressIndex::New)?.address;
+
+    // Send 10 BTC from core wallet to bdk wallet
+    bitcoind.client.send_to_address(
+        &bdk_new_addr,
+        Amount::from_btc(10.0)?,
+        None,
+        None,
+        None,
+        None,
+        None,
+        None,
+    )?;
+
+    // Confirm transaction by generating 1 block
+    bitcoind.client.generate_to_address(1, &core_address)?;
+
+    // Sync the BDK wallet
+    // This time the sync will fetch the new transaction and update it in
+    // wallet database
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> Received coins in BDK wallet");
+    println!(
+        "Available balance in BDK wallet: {} sats",
+        wallet.get_balance()?
+    );
+
+    println!("\n>> Sending coins: BDK --> Core, 5 BTC");
+    // Attempt to send back 5.0 BTC to core address by creating a transaction
+    //
+    // Transactions are created using a `TxBuilder`.
+    // This helps us to systematically build a transaction with all
+    // required customization.
+    // A full list of APIs offered by `TxBuilder` can be found at
+    // https://docs.rs/bdk/latest/bdk/wallet/tx_builder/struct.TxBuilder.html
+    let mut tx_builder = wallet.build_tx();
+
+    // For a regular transaction, just set the recipient and amount
+    tx_builder.set_recipients(vec![(core_address.script_pubkey(), 500000000)]);
+
+    // Finalize the transaction and extract the PSBT
+    let (mut psbt, _) = tx_builder.finish()?;
+
+    // Set signing option
+    let signopt = SignOptions {
+        assume_height: None,
+        ..Default::default()
+    };
+
+    // Sign the psbt
+    wallet.sign(&mut psbt, signopt)?;
+
+    // Extract the signed transaction
+    let tx = psbt.extract_tx();
+
+    // Broadcast the transaction
+    blockchain.broadcast(&tx)?;
+
+    // Confirm transaction by generating some blocks
+    bitcoind.client.generate_to_address(1, &core_address)?;
+
+    // Sync the BDK wallet
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> Coins sent to Core wallet");
+    println!(
+        "Remaining BDK wallet balance: {} sats",
+        wallet.get_balance()?
+    );
+    println!("\nCongrats!! you made your first test transaction with bdk and bitcoin core.");
+
+    Ok(())
+}
+
+// Helper function demonstrating privatekey extraction using bip39 mnemonic
+// The mnemonic can be shown to user to safekeeping and the same wallet
+// private descriptors can be recreated from it.
+fn generate_random_ext_privkey() -> Result<impl DerivableKey<Segwitv0> + Clone, Box<dyn Error>> {
+    // a Bip39 passphrase can be set optionally
+    let password = Some("random password".to_string());
+
+    // Generate a random mnemonic, and use that to create a "DerivableKey"
+    let mnemonic: GeneratedKey<_, _> = Mnemonic::generate((WordCount::Words12, Language::English))
+        .map_err(|e| e.expect("Unknown Error"))?;
+
+    // `Ok(mnemonic)` would also work if there's no passphrase and it would
+    // yield the same result as this construct with `password` = `None`.
+    Ok((mnemonic, password))
+}

src/blockchain/any.rs

@@ -16,61 +16,17 @@
 //!
 //! ## Example
 //!
-//! In this example both `wallet_electrum` and `wallet_esplora` have the same type of
-//! `Wallet<AnyBlockchain, MemoryDatabase>`. This means that they could both, for instance, be
-//! assigned to a struct member.
-//!
-//! ```no_run
-//! # use bitcoin::Network;
-//! # use bdk::blockchain::*;
-//! # use bdk::database::MemoryDatabase;
-//! # use bdk::Wallet;
-//! # #[cfg(feature = "electrum")]
-//! # {
-//! let electrum_blockchain = ElectrumBlockchain::from(electrum_client::Client::new("...")?);
-//! let wallet_electrum: Wallet<AnyBlockchain, _> = Wallet::new(
-//!     "...",
-//!     None,
-//!     Network::Testnet,
-//!     MemoryDatabase::default(),
-//!     electrum_blockchain.into(),
-//! )?;
-//! # }
-//!
-//! # #[cfg(all(feature = "esplora", feature = "ureq"))]
-//! # {
-//! let esplora_blockchain = EsploraBlockchain::new("...", 20);
-//! let wallet_esplora: Wallet<AnyBlockchain, _> = Wallet::new(
-//!     "...",
-//!     None,
-//!     Network::Testnet,
-//!     MemoryDatabase::default(),
-//!     esplora_blockchain.into(),
-//! )?;
-//! # }
-//!
-//! # Ok::<(), bdk::Error>(())
-//! ```
-//!
-//! When paired with the use of [`ConfigurableBlockchain`], it allows creating wallets with any
+//! When paired with the use of [`ConfigurableBlockchain`], it allows creating any
 //! blockchain type supported using a single line of code:
 //!
 //! ```no_run
 //! # use bitcoin::Network;
 //! # use bdk::blockchain::*;
-//! # use bdk::database::MemoryDatabase;
-//! # use bdk::Wallet;
 //! # #[cfg(all(feature = "esplora", feature = "ureq"))]
 //! # {
 //! let config = serde_json::from_str("...")?;
 //! let blockchain = AnyBlockchain::from_config(&config)?;
-//! let wallet = Wallet::new(
-//!     "...",
-//!     None,
-//!     Network::Testnet,
-//!     MemoryDatabase::default(),
-//!     blockchain,
-//! )?;
+//! let height = blockchain.get_height();
 //! # }
 //! # Ok::<(), bdk::Error>(())
 //! ```
@@ -78,6 +34,14 @@
 use super::*;
 
 macro_rules! impl_from {
+    ( boxed $from:ty, $to:ty, $variant:ident, $( $cfg:tt )* ) => {
+        $( $cfg )*
+        impl From<$from> for $to {
+            fn from(inner: $from) -> Self {
+                <$to>::$variant(Box::new(inner))
+            }
+        }
+    };
     ( $from:ty, $to:ty, $variant:ident, $( $cfg:tt )* ) => {
         $( $cfg )*
         impl From<$from> for $to {
@@ -112,19 +76,19 @@ pub enum AnyBlockchain {
     #[cfg(feature = "electrum")]
     #[cfg_attr(docsrs, doc(cfg(feature = "electrum")))]
     /// Electrum client
-    Electrum(electrum::ElectrumBlockchain),
+    Electrum(Box<electrum::ElectrumBlockchain>),
     #[cfg(feature = "esplora")]
     #[cfg_attr(docsrs, doc(cfg(feature = "esplora")))]
     /// Esplora client
-    Esplora(esplora::EsploraBlockchain),
+    Esplora(Box<esplora::EsploraBlockchain>),
     #[cfg(feature = "compact_filters")]
     #[cfg_attr(docsrs, doc(cfg(feature = "compact_filters")))]
     /// Compact filters client
-    CompactFilters(compact_filters::CompactFiltersBlockchain),
+    CompactFilters(Box<compact_filters::CompactFiltersBlockchain>),
     #[cfg(feature = "rpc")]
     #[cfg_attr(docsrs, doc(cfg(feature = "rpc")))]
     /// RPC client
-    Rpc(rpc::RpcBlockchain),
+    Rpc(Box<rpc::RpcBlockchain>),
 }
 
 #[maybe_async]
@@ -133,40 +97,69 @@ impl Blockchain for AnyBlockchain {
         maybe_await!(impl_inner_method!(self, get_capabilities))
     }
 
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(impl_inner_method!(self, setup, database, progress_update))
-    }
-    fn sync<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(impl_inner_method!(self, sync, database, progress_update))
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        maybe_await!(impl_inner_method!(self, get_tx, txid))
-    }
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         maybe_await!(impl_inner_method!(self, broadcast, tx))
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        maybe_await!(impl_inner_method!(self, get_height))
-    }
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         maybe_await!(impl_inner_method!(self, estimate_fee, target))
     }
 }
 
-impl_from!(electrum::ElectrumBlockchain, AnyBlockchain, Electrum, #[cfg(feature = "electrum")]);
-impl_from!(esplora::EsploraBlockchain, AnyBlockchain, Esplora, #[cfg(feature = "esplora")]);
-impl_from!(compact_filters::CompactFiltersBlockchain, AnyBlockchain, CompactFilters, #[cfg(feature = "compact_filters")]);
-impl_from!(rpc::RpcBlockchain, AnyBlockchain, Rpc, #[cfg(feature = "rpc")]);
+#[maybe_async]
+impl GetHeight for AnyBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        maybe_await!(impl_inner_method!(self, get_height))
+    }
+}
+
+#[maybe_async]
+impl GetTx for AnyBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        maybe_await!(impl_inner_method!(self, get_tx, txid))
+    }
+}
+
+#[maybe_async]
+impl GetBlockHash for AnyBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        maybe_await!(impl_inner_method!(self, get_block_hash, height))
+    }
+}
+
+#[maybe_async]
+impl WalletSync for AnyBlockchain {
+    fn wallet_sync<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(
+            self,
+            wallet_sync,
+            database,
+            progress_update
+        ))
+    }
+
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(
+            self,
+            wallet_setup,
+            database,
+            progress_update
+        ))
+    }
+}
+
+impl_from!(boxed electrum::ElectrumBlockchain, AnyBlockchain, Electrum, #[cfg(feature = "electrum")]);
+impl_from!(boxed esplora::EsploraBlockchain, AnyBlockchain, Esplora, #[cfg(feature = "esplora")]);
+impl_from!(boxed compact_filters::CompactFiltersBlockchain, AnyBlockchain, CompactFilters, #[cfg(feature = "compact_filters")]);
+impl_from!(boxed rpc::RpcBlockchain, AnyBlockchain, Rpc, #[cfg(feature = "rpc")]);
 
 /// Type that can contain any of the blockchain configurations defined by the library
 ///
@@ -229,19 +222,19 @@ impl ConfigurableBlockchain for AnyBlockchain {
         Ok(match config {
             #[cfg(feature = "electrum")]
             AnyBlockchainConfig::Electrum(inner) => {
-                AnyBlockchain::Electrum(electrum::ElectrumBlockchain::from_config(inner)?)
+                AnyBlockchain::Electrum(Box::new(electrum::ElectrumBlockchain::from_config(inner)?))
             }
             #[cfg(feature = "esplora")]
             AnyBlockchainConfig::Esplora(inner) => {
-                AnyBlockchain::Esplora(esplora::EsploraBlockchain::from_config(inner)?)
+                AnyBlockchain::Esplora(Box::new(esplora::EsploraBlockchain::from_config(inner)?))
             }
             #[cfg(feature = "compact_filters")]
-            AnyBlockchainConfig::CompactFilters(inner) => AnyBlockchain::CompactFilters(
+            AnyBlockchainConfig::CompactFilters(inner) => AnyBlockchain::CompactFilters(Box::new(
                 compact_filters::CompactFiltersBlockchain::from_config(inner)?,
-            ),
+            )),
             #[cfg(feature = "rpc")]
             AnyBlockchainConfig::Rpc(inner) => {
-                AnyBlockchain::Rpc(rpc::RpcBlockchain::from_config(inner)?)
+                AnyBlockchain::Rpc(Box::new(rpc::RpcBlockchain::from_config(inner)?))
             }
         })
     }

src/blockchain/compact_filters/mod.rs

@@ -67,7 +67,7 @@ mod peer;
 mod store;
 mod sync;
 
-use super::{Blockchain, Capability, ConfigurableBlockchain, Progress};
+use crate::blockchain::*;
 use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
 use crate::error::Error;
 use crate::types::{KeychainKind, LocalUtxo, TransactionDetails};
@@ -163,11 +163,19 @@ impl CompactFiltersBlockchain {
             if let Some(previous_output) = database.get_previous_output(&input.previous_output)? {
                 inputs_sum += previous_output.value;
 
-                if database.is_mine(&previous_output.script_pubkey)? {
+                // this output is ours, we have a path to derive it
+                if let Some((keychain, _)) =
+                    database.get_path_from_script_pubkey(&previous_output.script_pubkey)?
+                {
                     outgoing += previous_output.value;
 
-                    debug!("{} input #{} is mine, removing from utxo", tx.txid(), i);
-                    updates.del_utxo(&input.previous_output)?;
+                    debug!("{} input #{} is mine, setting utxo as spent", tx.txid(), i);
+                    updates.set_utxo(&LocalUtxo {
+                        outpoint: input.previous_output,
+                        txout: previous_output.clone(),
+                        keychain,
+                        is_spent: true,
+                    })?;
                 }
             }
         }
@@ -185,6 +193,7 @@ impl CompactFiltersBlockchain {
                     outpoint: OutPoint::new(tx.txid(), i as u32),
                     txout: output.clone(),
                     keychain,
+                    is_spent: false,
                 })?;
                 incoming += output.value;
 
@@ -207,7 +216,6 @@ impl CompactFiltersBlockchain {
                 received: incoming,
                 sent: outgoing,
                 confirmation_time: BlockTime::new(height, timestamp),
-                verified: height.is_some(),
                 fee: Some(inputs_sum.saturating_sub(outputs_sum)),
             };
 
@@ -226,11 +234,48 @@ impl Blockchain for CompactFiltersBlockchain {
         vec![Capability::FullHistory].into_iter().collect()
     }
 
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        self.peers[0].broadcast_tx(tx.clone())?;
+
+        Ok(())
+    }
+
+    fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
+        // TODO
+        Ok(FeeRate::default())
+    }
+}
+
+impl GetHeight for CompactFiltersBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.headers.get_height()? as u32)
+    }
+}
+
+impl GetTx for CompactFiltersBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.peers[0]
+            .get_mempool()
+            .get_tx(&Inventory::Transaction(*txid)))
+    }
+}
+
+impl GetBlockHash for CompactFiltersBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        self.headers
+            .get_block_hash(height as usize)?
+            .ok_or(Error::CompactFilters(
+                CompactFiltersError::BlockHashNotFound,
+            ))
+    }
+}
+
+impl WalletSync for CompactFiltersBlockchain {
     #[allow(clippy::mutex_atomic)] // Mutex is easier to understand than a CAS loop.
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
+    fn wallet_setup<D: BatchDatabase>(
         &self,
         database: &mut D,
-        progress_update: P,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error> {
         let first_peer = &self.peers[0];
 
@@ -431,27 +476,6 @@ impl Blockchain for CompactFiltersBlockchain {
 
         Ok(())
     }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.peers[0]
-            .get_mempool()
-            .get_tx(&Inventory::Transaction(*txid)))
-    }
-
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        self.peers[0].broadcast_tx(tx.clone())?;
-
-        Ok(())
-    }
-
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(self.headers.get_height()? as u32)
-    }
-
-    fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
-        // TODO
-        Ok(FeeRate::default())
-    }
 }
 
 /// Data to connect to a Bitcoin P2P peer
@@ -472,7 +496,7 @@ pub struct CompactFiltersBlockchainConfig {
     pub peers: Vec<BitcoinPeerConfig>,
     /// Network used
     pub network: Network,
-    /// Storage dir to save partially downloaded headers and full blocks
+    /// Storage dir to save partially downloaded headers and full blocks. Should be a separate directory per descriptor. Consider using [crate::wallet::wallet_name_from_descriptor] for this.
     pub storage_dir: String,
     /// Optionally skip initial `skip_blocks` blocks (default: 0)
     pub skip_blocks: Option<usize>,
@@ -522,6 +546,8 @@ pub enum CompactFiltersError {
     InvalidFilter,
     /// The peer is missing a block in the valid chain
     MissingBlock,
+    /// Block hash at specified height not found
+    BlockHashNotFound,
     /// The data stored in the block filters storage are corrupted
     DataCorruption,
 

src/blockchain/compact_filters/peer.rs

@@ -10,6 +10,7 @@
 // licenses.
 
 use std::collections::HashMap;
+use std::io::BufReader;
 use std::net::{TcpStream, ToSocketAddrs};
 use std::sync::{Arc, Condvar, Mutex, RwLock};
 use std::thread;
@@ -19,14 +20,13 @@ use socks::{Socks5Stream, ToTargetAddr};
 
 use rand::{thread_rng, Rng};
 
-use bitcoin::consensus::Encodable;
+use bitcoin::consensus::{Decodable, Encodable};
 use bitcoin::hash_types::BlockHash;
 use bitcoin::network::constants::ServiceFlags;
 use bitcoin::network::message::{NetworkMessage, RawNetworkMessage};
 use bitcoin::network::message_blockdata::*;
 use bitcoin::network::message_filter::*;
 use bitcoin::network::message_network::VersionMessage;
-use bitcoin::network::stream_reader::StreamReader;
 use bitcoin::network::Address;
 use bitcoin::{Block, Network, Transaction, Txid, Wtxid};
 
@@ -94,8 +94,7 @@ impl Mempool {
             TxIdentifier::Wtxid(wtxid) => self.0.read().unwrap().wtxids.get(&wtxid).cloned(),
         };
 
-        txid.map(|txid| self.0.read().unwrap().txs.get(&txid).cloned())
-            .flatten()
+        txid.and_then(|txid| self.0.read().unwrap().txs.get(&txid).cloned())
     }
 
     /// Return whether or not the mempool contains a transaction with a given txid
@@ -111,6 +110,7 @@ impl Mempool {
 
 /// A Bitcoin peer
 #[derive(Debug)]
+#[allow(dead_code)]
 pub struct Peer {
     writer: Arc<Mutex<TcpStream>>,
     responses: Arc<RwLock<ResponsesMap>>,
@@ -327,9 +327,10 @@ impl Peer {
             };
         }
 
-        let mut reader = StreamReader::new(connection, None);
+        let mut reader = BufReader::new(connection);
         loop {
-            let raw_message: RawNetworkMessage = check_disconnect!(reader.read_next());
+            let raw_message: RawNetworkMessage =
+                check_disconnect!(Decodable::consensus_decode(&mut reader));
 
             let in_message = if raw_message.magic != network.magic() {
                 continue;

src/blockchain/compact_filters/store.rs

@@ -398,7 +398,7 @@ impl ChainStore<Full> {
             );
         }
 
-        // Delete full blocks overriden by snapshot
+        // Delete full blocks overridden by snapshot
         let from_key = StoreEntry::Block(Some(snaphost.min_height)).get_key();
         let to_key = StoreEntry::Block(Some(usize::MAX)).get_key();
         batch.delete_range(&from_key, &to_key);

src/blockchain/electrum.rs

@@ -24,20 +24,21 @@
 //! # Ok::<(), bdk::Error>(())
 //! ```
 
-use std::collections::HashSet;
+use std::collections::{HashMap, HashSet};
+use std::ops::Deref;
 
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
-use bitcoin::{BlockHeader, Script, Transaction, Txid};
+use bitcoin::{Transaction, Txid};
 
 use electrum_client::{Client, ConfigBuilder, ElectrumApi, Socks5Config};
 
-use self::utils::{ElectrumLikeSync, ElsGetHistoryRes};
+use super::script_sync::Request;
 use super::*;
-use crate::database::BatchDatabase;
+use crate::database::{BatchDatabase, Database};
 use crate::error::Error;
-use crate::FeeRate;
+use crate::{BlockTime, FeeRate};
 
 /// Wrapper over an Electrum Client that implements the required blockchain traits
 ///
@@ -68,32 +69,10 @@ impl Blockchain for ElectrumBlockchain {
         .collect()
     }
 
-    fn setup<D: BatchDatabase, P: Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        self.client
-            .electrum_like_setup(self.stop_gap, database, progress_update)
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.client.transaction_get(txid).map(Option::Some)?)
-    }
-
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         Ok(self.client.transaction_broadcast(tx).map(|_| ())?)
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        // TODO: unsubscribe when added to the client, or is there a better call to use here?
-
-        Ok(self
-            .client
-            .block_headers_subscribe()
-            .map(|data| data.height as u32)?)
-    }
-
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         Ok(FeeRate::from_btc_per_kvb(
             self.client.estimate_fee(target)? as f32
@@ -101,43 +80,218 @@ impl Blockchain for ElectrumBlockchain {
     }
 }
 
-impl ElectrumLikeSync for Client {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script> + Clone>(
+impl Deref for ElectrumBlockchain {
+    type Target = Client;
+
+    fn deref(&self) -> &Self::Target {
+        &self.client
+    }
+}
+
+impl StatelessBlockchain for ElectrumBlockchain {}
+
+impl GetHeight for ElectrumBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        // TODO: unsubscribe when added to the client, or is there a better call to use here?
+
+        Ok(self
+            .client
+            .block_headers_subscribe()
+            .map(|data| data.height as u32)?)
+    }
+}
+
+impl GetTx for ElectrumBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.client.transaction_get(txid).map(Option::Some)?)
+    }
+}
+
+impl GetBlockHash for ElectrumBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        let block_header = self.client.block_header(height as usize)?;
+        Ok(block_header.block_hash())
+    }
+}
+
+impl WalletSync for ElectrumBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
         &self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error> {
-        self.batch_script_get_history(scripts)
-            .map(|v| {
-                v.into_iter()
-                    .map(|v| {
-                        v.into_iter()
-                            .map(
-                                |electrum_client::GetHistoryRes {
-                                     height, tx_hash, ..
-                                 }| ElsGetHistoryRes {
-                                    height,
-                                    tx_hash,
-                                },
-                            )
-                            .collect()
-                    })
-                    .collect()
-            })
-            .map_err(Error::Electrum)
+        database: &mut D,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut block_times = HashMap::<u32, u32>::new();
+        let mut txid_to_height = HashMap::<Txid, u32>::new();
+        let mut tx_cache = TxCache::new(database, &self.client);
+
+        // Set chunk_size to the smallest value capable of finding a gap greater than stop_gap.
+        let chunk_size = self.stop_gap + 1;
+
+        // The electrum server has been inconsistent somehow in its responses during sync. For
+        // example, we do a batch request of transactions and the response contains less
+        // tranascations than in the request. This should never happen but we don't want to panic.
+        let electrum_goof = || Error::Generic("electrum server misbehaving".to_string());
+
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let scripts = script_req.request().take(chunk_size);
+                    let txids_per_script: Vec<Vec<_>> = self
+                        .client
+                        .batch_script_get_history(scripts)
+                        .map_err(Error::Electrum)?
+                        .into_iter()
+                        .map(|txs| {
+                            txs.into_iter()
+                                .map(|tx| {
+                                    let tx_height = match tx.height {
+                                        none if none <= 0 => None,
+                                        height => {
+                                            txid_to_height.insert(tx.tx_hash, height as u32);
+                                            Some(height as u32)
+                                        }
+                                    };
+                                    (tx.tx_hash, tx_height)
+                                })
+                                .collect()
+                        })
+                        .collect();
+
+                    script_req.satisfy(txids_per_script)?
+                }
+
+                Request::Conftime(conftime_req) => {
+                    // collect up to chunk_size heights to fetch from electrum
+                    let needs_block_height = conftime_req
+                        .request()
+                        .filter_map(|txid| txid_to_height.get(txid).cloned())
+                        .filter(|height| block_times.get(height).is_none())
+                        .take(chunk_size)
+                        .collect::<HashSet<u32>>();
+
+                    let new_block_headers = self
+                        .client
+                        .batch_block_header(needs_block_height.iter().cloned())?;
+
+                    for (height, header) in needs_block_height.into_iter().zip(new_block_headers) {
+                        block_times.insert(height, header.time);
+                    }
+
+                    let conftimes = conftime_req
+                        .request()
+                        .take(chunk_size)
+                        .map(|txid| {
+                            let confirmation_time = txid_to_height
+                                .get(txid)
+                                .map(|height| {
+                                    let timestamp =
+                                        *block_times.get(height).ok_or_else(electrum_goof)?;
+                                    Result::<_, Error>::Ok(BlockTime {
+                                        height: *height,
+                                        timestamp: timestamp.into(),
+                                    })
+                                })
+                                .transpose()?;
+                            Ok(confirmation_time)
+                        })
+                        .collect::<Result<_, Error>>()?;
+
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let needs_full = tx_req.request().take(chunk_size);
+                    tx_cache.save_txs(needs_full.clone())?;
+                    let full_transactions = needs_full
+                        .map(|txid| tx_cache.get(*txid).ok_or_else(electrum_goof))
+                        .collect::<Result<Vec<_>, _>>()?;
+                    let input_txs = full_transactions.iter().flat_map(|tx| {
+                        tx.input
+                            .iter()
+                            .filter(|input| !input.previous_output.is_null())
+                            .map(|input| &input.previous_output.txid)
+                    });
+                    tx_cache.save_txs(input_txs)?;
+
+                    let full_details = full_transactions
+                        .into_iter()
+                        .map(|tx| {
+                            let mut input_index = 0usize;
+                            let prev_outputs = tx
+                                .input
+                                .iter()
+                                .map(|input| {
+                                    if input.previous_output.is_null() {
+                                        return Ok(None);
+                                    }
+                                    let prev_tx = tx_cache
+                                        .get(input.previous_output.txid)
+                                        .ok_or_else(electrum_goof)?;
+                                    let txout = prev_tx
+                                        .output
+                                        .get(input.previous_output.vout as usize)
+                                        .ok_or_else(electrum_goof)?;
+                                    input_index += 1;
+                                    Ok(Some(txout.clone()))
+                                })
+                                .collect::<Result<Vec<_>, Error>>()?;
+                            Ok((prev_outputs, tx))
+                        })
+                        .collect::<Result<Vec<_>, Error>>()?;
+
+                    tx_req.satisfy(full_details)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+        Ok(())
+    }
+}
+
+struct TxCache<'a, 'b, D> {
+    db: &'a D,
+    client: &'b Client,
+    cache: HashMap<Txid, Transaction>,
+}
+
+impl<'a, 'b, D: Database> TxCache<'a, 'b, D> {
+    fn new(db: &'a D, client: &'b Client) -> Self {
+        TxCache {
+            db,
+            client,
+            cache: HashMap::default(),
+        }
+    }
+    fn save_txs<'c>(&mut self, txids: impl Iterator<Item = &'c Txid>) -> Result<(), Error> {
+        let mut need_fetch = vec![];
+        for txid in txids {
+            if self.cache.get(txid).is_some() {
+                continue;
+            } else if let Some(transaction) = self.db.get_raw_tx(txid)? {
+                self.cache.insert(*txid, transaction);
+            } else {
+                need_fetch.push(txid);
+            }
+        }
+
+        if !need_fetch.is_empty() {
+            let txs = self
+                .client
+                .batch_transaction_get(need_fetch.clone())
+                .map_err(Error::Electrum)?;
+            for (tx, _txid) in txs.into_iter().zip(need_fetch) {
+                debug_assert_eq!(*_txid, tx.txid());
+                self.cache.insert(tx.txid(), tx);
+            }
+        }
+
+        Ok(())
     }
 
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid> + Clone>(
-        &self,
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error> {
-        self.batch_transaction_get(txids).map_err(Error::Electrum)
-    }
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32> + Clone>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error> {
-        self.batch_block_header(heights).map_err(Error::Electrum)
+    fn get(&self, txid: Txid) -> Option<Transaction> {
+        self.cache.get(&txid).map(Clone::clone)
     }
 }
 
@@ -178,8 +332,93 @@ impl ConfigurableBlockchain for ElectrumBlockchain {
 
 #[cfg(test)]
 #[cfg(feature = "test-electrum")]
-crate::bdk_blockchain_tests! {
-    fn test_instance(test_client: &TestClient) -> ElectrumBlockchain {
-        ElectrumBlockchain::from(Client::new(&test_client.electrsd.electrum_url).unwrap())
+mod test {
+    use std::sync::Arc;
+
+    use super::*;
+    use crate::database::MemoryDatabase;
+    use crate::testutils::blockchain_tests::TestClient;
+    use crate::testutils::configurable_blockchain_tests::ConfigurableBlockchainTester;
+    use crate::wallet::{AddressIndex, Wallet};
+
+    crate::bdk_blockchain_tests! {
+        fn test_instance(test_client: &TestClient) -> ElectrumBlockchain {
+            ElectrumBlockchain::from(Client::new(&test_client.electrsd.electrum_url).unwrap())
+        }
+    }
+
+    fn get_factory() -> (TestClient, Arc<ElectrumBlockchain>) {
+        let test_client = TestClient::default();
+
+        let factory = Arc::new(ElectrumBlockchain::from(
+            Client::new(&test_client.electrsd.electrum_url).unwrap(),
+        ));
+
+        (test_client, factory)
+    }
+
+    #[test]
+    fn test_electrum_blockchain_factory() {
+        let (_test_client, factory) = get_factory();
+
+        let a = factory.build("aaaaaa", None).unwrap();
+        let b = factory.build("bbbbbb", None).unwrap();
+
+        assert_eq!(
+            a.client.block_headers_subscribe().unwrap().height,
+            b.client.block_headers_subscribe().unwrap().height
+        );
+    }
+
+    #[test]
+    fn test_electrum_blockchain_factory_sync_wallet() {
+        let (mut test_client, factory) = get_factory();
+
+        let db = MemoryDatabase::new();
+        let wallet = Wallet::new(
+            "wpkh(L5EZftvrYaSudiozVRzTqLcHLNDoVn7H5HSfM9BAN6tMJX8oTWz6)",
+            None,
+            bitcoin::Network::Regtest,
+            db,
+        )
+        .unwrap();
+
+        let address = wallet.get_address(AddressIndex::New).unwrap();
+
+        let tx = testutils! {
+            @tx ( (@addr address.address) => 50_000 )
+        };
+        test_client.receive(tx);
+
+        factory
+            .sync_wallet(&wallet, None, Default::default())
+            .unwrap();
+
+        assert_eq!(wallet.get_balance().unwrap().untrusted_pending, 50_000);
+    }
+
+    #[test]
+    fn test_electrum_with_variable_configs() {
+        struct ElectrumTester;
+
+        impl ConfigurableBlockchainTester<ElectrumBlockchain> for ElectrumTester {
+            const BLOCKCHAIN_NAME: &'static str = "Electrum";
+
+            fn config_with_stop_gap(
+                &self,
+                test_client: &mut TestClient,
+                stop_gap: usize,
+            ) -> Option<ElectrumBlockchainConfig> {
+                Some(ElectrumBlockchainConfig {
+                    url: test_client.electrsd.electrum_url.clone(),
+                    socks5: None,
+                    retry: 0,
+                    timeout: None,
+                    stop_gap: stop_gap,
+                })
+            }
+        }
+
+        ElectrumTester.run();
     }
 }

src/blockchain/esplora/api.rs

@@ -0,0 +1,117 @@
+//! structs from the esplora API
+//!
+//! see: <https://github.com/Blockstream/esplora/blob/master/API.md>
+use crate::BlockTime;
+use bitcoin::{OutPoint, Script, Transaction, TxIn, TxOut, Txid, Witness};
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct PrevOut {
+    pub value: u64,
+    pub scriptpubkey: Script,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct Vin {
+    pub txid: Txid,
+    pub vout: u32,
+    // None if coinbase
+    pub prevout: Option<PrevOut>,
+    pub scriptsig: Script,
+    #[serde(deserialize_with = "deserialize_witness", default)]
+    pub witness: Vec<Vec<u8>>,
+    pub sequence: u32,
+    pub is_coinbase: bool,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct Vout {
+    pub value: u64,
+    pub scriptpubkey: Script,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct TxStatus {
+    pub confirmed: bool,
+    pub block_height: Option<u32>,
+    pub block_time: Option<u64>,
+}
+
+#[derive(serde::Deserialize, Clone, Debug)]
+pub struct Tx {
+    pub txid: Txid,
+    pub version: i32,
+    pub locktime: u32,
+    pub vin: Vec<Vin>,
+    pub vout: Vec<Vout>,
+    pub status: TxStatus,
+    pub fee: u64,
+}
+
+impl Tx {
+    pub fn to_tx(&self) -> Transaction {
+        Transaction {
+            version: self.version,
+            lock_time: self.locktime,
+            input: self
+                .vin
+                .iter()
+                .cloned()
+                .map(|vin| TxIn {
+                    previous_output: OutPoint {
+                        txid: vin.txid,
+                        vout: vin.vout,
+                    },
+                    script_sig: vin.scriptsig,
+                    sequence: vin.sequence,
+                    witness: Witness::from_vec(vin.witness),
+                })
+                .collect(),
+            output: self
+                .vout
+                .iter()
+                .cloned()
+                .map(|vout| TxOut {
+                    value: vout.value,
+                    script_pubkey: vout.scriptpubkey,
+                })
+                .collect(),
+        }
+    }
+
+    pub fn confirmation_time(&self) -> Option<BlockTime> {
+        match self.status {
+            TxStatus {
+                confirmed: true,
+                block_height: Some(height),
+                block_time: Some(timestamp),
+            } => Some(BlockTime { timestamp, height }),
+            _ => None,
+        }
+    }
+
+    pub fn previous_outputs(&self) -> Vec<Option<TxOut>> {
+        self.vin
+            .iter()
+            .cloned()
+            .map(|vin| {
+                vin.prevout.map(|po| TxOut {
+                    script_pubkey: po.scriptpubkey,
+                    value: po.value,
+                })
+            })
+            .collect()
+    }
+}
+
+fn deserialize_witness<'de, D>(d: D) -> Result<Vec<Vec<u8>>, D::Error>
+where
+    D: serde::de::Deserializer<'de>,
+{
+    use crate::serde::Deserialize;
+    use bitcoin::hashes::hex::FromHex;
+    let list = Vec::<String>::deserialize(d)?;
+    list.into_iter()
+        .map(|hex_str| Vec::<u8>::from_hex(&hex_str))
+        .collect::<Result<Vec<Vec<u8>>, _>>()
+        .map_err(serde::de::Error::custom)
+}

src/blockchain/esplora/mod.rs

@@ -21,8 +21,6 @@ use std::collections::HashMap;
 use std::fmt;
 use std::io;
 
-use serde::Deserialize;
-
 use bitcoin::consensus;
 use bitcoin::{BlockHash, Txid};
 
@@ -41,33 +39,24 @@ mod ureq;
 #[cfg(feature = "ureq")]
 pub use self::ureq::*;
 
+mod api;
+
 fn into_fee_rate(target: usize, estimates: HashMap<String, f64>) -> Result<FeeRate, Error> {
-    let fee_val = estimates
-        .into_iter()
-        .map(|(k, v)| Ok::<_, std::num::ParseIntError>((k.parse::<usize>()?, v)))
-        .collect::<Result<Vec<_>, _>>()
-        .map_err(|e| Error::Generic(e.to_string()))?
-        .into_iter()
-        .take_while(|(k, _)| k <= &target)
-        .map(|(_, v)| v)
-        .last()
-        .unwrap_or(1.0);
-
+    let fee_val = {
+        let mut pairs = estimates
+            .into_iter()
+            .filter_map(|(k, v)| Some((k.parse::<usize>().ok()?, v)))
+            .collect::<Vec<_>>();
+        pairs.sort_unstable_by_key(|(k, _)| std::cmp::Reverse(*k));
+        pairs
+            .into_iter()
+            .find(|(k, _)| k <= &target)
+            .map(|(_, v)| v)
+            .unwrap_or(1.0)
+    };
     Ok(FeeRate::from_sat_per_vb(fee_val as f32))
 }
 
-/// Data type used when fetching transaction history from Esplora.
-#[derive(Deserialize)]
-pub struct EsploraGetHistory {
-    txid: Txid,
-    status: EsploraGetHistoryStatus,
-}
-
-#[derive(Deserialize)]
-struct EsploraGetHistoryStatus {
-    block_height: Option<usize>,
-}
-
 /// Errors that can happen during a sync with [`EsploraBlockchain`]
 #[derive(Debug)]
 pub enum EsploraError {
@@ -107,10 +96,50 @@ impl fmt::Display for EsploraError {
     }
 }
 
+/// Configuration for an [`EsploraBlockchain`]
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
+pub struct EsploraBlockchainConfig {
+    /// Base URL of the esplora service
+    ///
+    /// eg. `https://blockstream.info/api/`
+    pub base_url: String,
+    /// Optional URL of the proxy to use to make requests to the Esplora server
+    ///
+    /// The string should be formatted as: `<protocol>://<user>:<password>@host:<port>`.
+    ///
+    /// Note that the format of this value and the supported protocols change slightly between the
+    /// sync version of esplora (using `ureq`) and the async version (using `reqwest`). For more
+    /// details check with the documentation of the two crates. Both of them are compiled with
+    /// the `socks` feature enabled.
+    ///
+    /// The proxy is ignored when targeting `wasm32`.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub proxy: Option<String>,
+    /// Number of parallel requests sent to the esplora service (default: 4)
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub concurrency: Option<u8>,
+    /// Stop searching addresses for transactions after finding an unused gap of this length.
+    pub stop_gap: usize,
+    /// Socket timeout.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub timeout: Option<u64>,
+}
+
+impl EsploraBlockchainConfig {
+    /// create a config with default values given the base url and stop gap
+    pub fn new(base_url: String, stop_gap: usize) -> Self {
+        Self {
+            base_url,
+            proxy: None,
+            timeout: None,
+            stop_gap,
+            concurrency: None,
+        }
+    }
+}
+
 impl std::error::Error for EsploraError {}
 
-#[cfg(feature = "ureq")]
-impl_error!(::ureq::Error, Ureq, EsploraError);
 #[cfg(feature = "ureq")]
 impl_error!(::ureq::Transport, UreqTransport, EsploraError);
 #[cfg(feature = "reqwest")]
@@ -127,3 +156,91 @@ crate::bdk_blockchain_tests! {
         EsploraBlockchain::new(&format!("http://{}",test_client.electrsd.esplora_url.as_ref().unwrap()), 20)
     }
 }
+
+const DEFAULT_CONCURRENT_REQUESTS: u8 = 4;
+
+#[cfg(test)]
+mod test {
+    use super::*;
+
+    #[test]
+    fn feerate_parsing() {
+        let esplora_fees = serde_json::from_str::<HashMap<String, f64>>(
+            r#"{
+  "25": 1.015,
+  "5": 2.3280000000000003,
+  "12": 2.0109999999999997,
+  "15": 1.018,
+  "17": 1.018,
+  "11": 2.0109999999999997,
+  "3": 3.01,
+  "2": 4.9830000000000005,
+  "6": 2.2359999999999998,
+  "21": 1.018,
+  "13": 1.081,
+  "7": 2.2359999999999998,
+  "8": 2.2359999999999998,
+  "16": 1.018,
+  "20": 1.018,
+  "22": 1.017,
+  "23": 1.017,
+  "504": 1,
+  "9": 2.2359999999999998,
+  "14": 1.018,
+  "10": 2.0109999999999997,
+  "24": 1.017,
+  "1008": 1,
+  "1": 4.9830000000000005,
+  "4": 2.3280000000000003,
+  "19": 1.018,
+  "144": 1,
+  "18": 1.018
+}
+"#,
+        )
+        .unwrap();
+        assert_eq!(
+            into_fee_rate(6, esplora_fees.clone()).unwrap(),
+            FeeRate::from_sat_per_vb(2.236)
+        );
+        assert_eq!(
+            into_fee_rate(26, esplora_fees).unwrap(),
+            FeeRate::from_sat_per_vb(1.015),
+            "should inherit from value for 25"
+        );
+    }
+
+    #[test]
+    #[cfg(feature = "test-esplora")]
+    fn test_esplora_with_variable_configs() {
+        use crate::testutils::{
+            blockchain_tests::TestClient,
+            configurable_blockchain_tests::ConfigurableBlockchainTester,
+        };
+
+        struct EsploraTester;
+
+        impl ConfigurableBlockchainTester<EsploraBlockchain> for EsploraTester {
+            const BLOCKCHAIN_NAME: &'static str = "Esplora";
+
+            fn config_with_stop_gap(
+                &self,
+                test_client: &mut TestClient,
+                stop_gap: usize,
+            ) -> Option<EsploraBlockchainConfig> {
+                Some(EsploraBlockchainConfig {
+                    base_url: format!(
+                        "http://{}",
+                        test_client.electrsd.esplora_url.as_ref().unwrap()
+                    ),
+                    proxy: None,
+                    concurrency: None,
+                    stop_gap: stop_gap,
+                    timeout: None,
+                })
+            }
+        }
+
+        EsploraTester.run();
+    }
+}

src/blockchain/esplora/reqwest.rs

@@ -12,6 +12,7 @@
 //! Esplora by way of `reqwest` HTTP client.
 
 use std::collections::{HashMap, HashSet};
+use std::ops::Deref;
 
 use bitcoin::consensus::{deserialize, serialize};
 use bitcoin::hashes::hex::{FromHex, ToHex};
@@ -21,22 +22,19 @@ use bitcoin::{BlockHeader, Script, Transaction, Txid};
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
-use futures::stream::{self, FuturesOrdered, StreamExt, TryStreamExt};
-
 use ::reqwest::{Client, StatusCode};
+use futures::stream::{FuturesOrdered, TryStreamExt};
 
-use crate::blockchain::esplora::{EsploraError, EsploraGetHistory};
-use crate::blockchain::utils::{ElectrumLikeSync, ElsGetHistoryRes};
+use super::api::Tx;
+use crate::blockchain::esplora::EsploraError;
 use crate::blockchain::*;
 use crate::database::BatchDatabase;
 use crate::error::Error;
-use crate::wallet::utils::ChunksIterator;
 use crate::FeeRate;
 
-const DEFAULT_CONCURRENT_REQUESTS: u8 = 4;
-
+/// Structure encapsulates Esplora client
 #[derive(Debug)]
-struct UrlClient {
+pub struct UrlClient {
     url: String,
     // We use the async client instead of the blocking one because it automatically uses `fetch`
     // when the target platform is wasm32.
@@ -70,7 +68,7 @@ impl EsploraBlockchain {
             url_client: UrlClient {
                 url: base_url.to_string(),
                 client: Client::new(),
-                concurrency: DEFAULT_CONCURRENT_REQUESTS,
+                concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
             },
             stop_gap,
         }
@@ -95,39 +93,141 @@ impl Blockchain for EsploraBlockchain {
         .collect()
     }
 
-    fn setup<D: BatchDatabase, P: Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(self
-            .url_client
-            .electrum_like_setup(self.stop_gap, database, progress_update))
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(await_or_block!(self.url_client._get_tx(txid))?)
-    }
-
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         Ok(await_or_block!(self.url_client._broadcast(tx))?)
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(await_or_block!(self.url_client._get_height())?)
-    }
-
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         let estimates = await_or_block!(self.url_client._get_fee_estimates())?;
         super::into_fee_rate(target, estimates)
     }
 }
 
-impl UrlClient {
-    fn script_to_scripthash(script: &Script) -> String {
-        sha256::Hash::hash(script.as_bytes()).into_inner().to_hex()
-    }
+impl Deref for EsploraBlockchain {
+    type Target = UrlClient;
 
+    fn deref(&self) -> &Self::Target {
+        &self.url_client
+    }
+}
+
+impl StatelessBlockchain for EsploraBlockchain {}
+
+#[maybe_async]
+impl GetHeight for EsploraBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(await_or_block!(self.url_client._get_height())?)
+    }
+}
+
+#[maybe_async]
+impl GetTx for EsploraBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(await_or_block!(self.url_client._get_tx(txid))?)
+    }
+}
+
+#[maybe_async]
+impl GetBlockHash for EsploraBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        let block_header = await_or_block!(self.url_client._get_header(height as u32))?;
+        Ok(block_header.block_hash())
+    }
+}
+
+#[maybe_async]
+impl WalletSync for EsploraBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        use crate::blockchain::script_sync::Request;
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut tx_index: HashMap<Txid, Tx> = HashMap::new();
+
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let futures: FuturesOrdered<_> = script_req
+                        .request()
+                        .take(self.url_client.concurrency as usize)
+                        .map(|script| async move {
+                            let mut related_txs: Vec<Tx> =
+                                self.url_client._scripthash_txs(script, None).await?;
+
+                            let n_confirmed =
+                                related_txs.iter().filter(|tx| tx.status.confirmed).count();
+                            // esplora pages on 25 confirmed transactions. If there's 25 or more we
+                            // keep requesting to see if there's more.
+                            if n_confirmed >= 25 {
+                                loop {
+                                    let new_related_txs: Vec<Tx> = self
+                                        .url_client
+                                        ._scripthash_txs(
+                                            script,
+                                            Some(related_txs.last().unwrap().txid),
+                                        )
+                                        .await?;
+                                    let n = new_related_txs.len();
+                                    related_txs.extend(new_related_txs);
+                                    // we've reached the end
+                                    if n < 25 {
+                                        break;
+                                    }
+                                }
+                            }
+                            Result::<_, Error>::Ok(related_txs)
+                        })
+                        .collect();
+                    let txs_per_script: Vec<Vec<Tx>> = await_or_block!(futures.try_collect())?;
+                    let mut satisfaction = vec![];
+
+                    for txs in txs_per_script {
+                        satisfaction.push(
+                            txs.iter()
+                                .map(|tx| (tx.txid, tx.status.block_height))
+                                .collect(),
+                        );
+                        for tx in txs {
+                            tx_index.insert(tx.txid, tx);
+                        }
+                    }
+
+                    script_req.satisfy(satisfaction)?
+                }
+                Request::Conftime(conftime_req) => {
+                    let conftimes = conftime_req
+                        .request()
+                        .map(|txid| {
+                            tx_index
+                                .get(txid)
+                                .expect("must be in index")
+                                .confirmation_time()
+                        })
+                        .collect();
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let full_txs = tx_req
+                        .request()
+                        .map(|txid| {
+                            let tx = tx_index.get(txid).expect("must be in index");
+                            Ok((tx.previous_outputs(), tx.to_tx()))
+                        })
+                        .collect::<Result<_, Error>>()?;
+                    tx_req.satisfy(full_txs)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+        Ok(())
+    }
+}
+
+impl UrlClient {
     async fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
         let resp = self
             .client
@@ -196,71 +296,27 @@ impl UrlClient {
         Ok(req.error_for_status()?.text().await?.parse()?)
     }
 
-    async fn _script_get_history(
+    async fn _scripthash_txs(
         &self,
         script: &Script,
-    ) -> Result<Vec<ElsGetHistoryRes>, EsploraError> {
-        let mut result = Vec::new();
-        let scripthash = Self::script_to_scripthash(script);
-
-        // Add the unconfirmed transactions first
-        result.extend(
-            self.client
-                .get(&format!(
-                    "{}/scripthash/{}/txs/mempool",
-                    self.url, scripthash
-                ))
-                .send()
-                .await?
-                .error_for_status()?
-                .json::<Vec<EsploraGetHistory>>()
-                .await?
-                .into_iter()
-                .map(|x| ElsGetHistoryRes {
-                    tx_hash: x.txid,
-                    height: x.status.block_height.unwrap_or(0) as i32,
-                }),
-        );
-
-        debug!(
-            "Found {} mempool txs for {} - {:?}",
-            result.len(),
-            scripthash,
-            script
-        );
-
-        // Then go through all the pages of confirmed transactions
-        let mut last_txid = String::new();
-        loop {
-            let response = self
-                .client
-                .get(&format!(
-                    "{}/scripthash/{}/txs/chain/{}",
-                    self.url, scripthash, last_txid
-                ))
-                .send()
-                .await?
-                .error_for_status()?
-                .json::<Vec<EsploraGetHistory>>()
-                .await?;
-            let len = response.len();
-            if let Some(elem) = response.last() {
-                last_txid = elem.txid.to_hex();
-            }
-
-            debug!("... adding {} confirmed transactions", len);
-
-            result.extend(response.into_iter().map(|x| ElsGetHistoryRes {
-                tx_hash: x.txid,
-                height: x.status.block_height.unwrap_or(0) as i32,
-            }));
-
-            if len < 25 {
-                break;
-            }
-        }
-
-        Ok(result)
+        last_seen: Option<Txid>,
+    ) -> Result<Vec<Tx>, EsploraError> {
+        let script_hash = sha256::Hash::hash(script.as_bytes()).into_inner().to_hex();
+        let url = match last_seen {
+            Some(last_seen) => format!(
+                "{}/scripthash/{}/txs/chain/{}",
+                self.url, script_hash, last_seen
+            ),
+            None => format!("{}/scripthash/{}/txs", self.url, script_hash),
+        };
+        Ok(self
+            .client
+            .get(url)
+            .send()
+            .await?
+            .error_for_status()?
+            .json::<Vec<Tx>>()
+            .await?)
     }
 
     async fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
@@ -275,83 +331,8 @@ impl UrlClient {
     }
 }
 
-#[maybe_async]
-impl ElectrumLikeSync for UrlClient {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script>>(
-        &self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error> {
-        let mut results = vec![];
-        for chunk in ChunksIterator::new(scripts.into_iter(), self.concurrency as usize) {
-            let mut futs = FuturesOrdered::new();
-            for script in chunk {
-                futs.push(self._script_get_history(script));
-            }
-            let partial_results: Vec<Vec<ElsGetHistoryRes>> = await_or_block!(futs.try_collect())?;
-            results.extend(partial_results);
-        }
-        Ok(await_or_block!(stream::iter(results).collect()))
-    }
-
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid>>(
-        &self,
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error> {
-        let mut results = vec![];
-        for chunk in ChunksIterator::new(txids.into_iter(), self.concurrency as usize) {
-            let mut futs = FuturesOrdered::new();
-            for txid in chunk {
-                futs.push(self._get_tx_no_opt(txid));
-            }
-            let partial_results: Vec<Transaction> = await_or_block!(futs.try_collect())?;
-            results.extend(partial_results);
-        }
-        Ok(await_or_block!(stream::iter(results).collect()))
-    }
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32>>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error> {
-        let mut results = vec![];
-        for chunk in ChunksIterator::new(heights.into_iter(), self.concurrency as usize) {
-            let mut futs = FuturesOrdered::new();
-            for height in chunk {
-                futs.push(self._get_header(height));
-            }
-            let partial_results: Vec<BlockHeader> = await_or_block!(futs.try_collect())?;
-            results.extend(partial_results);
-        }
-        Ok(await_or_block!(stream::iter(results).collect()))
-    }
-}
-
-/// Configuration for an [`EsploraBlockchain`]
-#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
-pub struct EsploraBlockchainConfig {
-    /// Base URL of the esplora service
-    ///
-    /// eg. `https://blockstream.info/api/`
-    pub base_url: String,
-    /// Optional URL of the proxy to use to make requests to the Esplora server
-    ///
-    /// The string should be formatted as: `<protocol>://<user>:<password>@host:<port>`.
-    ///
-    /// Note that the format of this value and the supported protocols change slightly between the
-    /// sync version of esplora (using `ureq`) and the async version (using `reqwest`). For more
-    /// details check with the documentation of the two crates. Both of them are compiled with
-    /// the `socks` feature enabled.
-    ///
-    /// The proxy is ignored when targeting `wasm32`.
-    pub proxy: Option<String>,
-    /// Number of parallel requests sent to the esplora service (default: 4)
-    pub concurrency: Option<u8>,
-    /// Stop searching addresses for transactions after finding an unused gap of this length.
-    pub stop_gap: usize,
-}
-
 impl ConfigurableBlockchain for EsploraBlockchain {
-    type Config = EsploraBlockchainConfig;
+    type Config = super::EsploraBlockchainConfig;
 
     fn from_config(config: &Self::Config) -> Result<Self, Error> {
         let map_e = |e: reqwest::Error| Error::Esplora(Box::new(e.into()));
@@ -360,13 +341,19 @@ impl ConfigurableBlockchain for EsploraBlockchain {
         if let Some(concurrency) = config.concurrency {
             blockchain.url_client.concurrency = concurrency;
         }
+        let mut builder = Client::builder();
         #[cfg(not(target_arch = "wasm32"))]
         if let Some(proxy) = &config.proxy {
-            blockchain.url_client.client = Client::builder()
-                .proxy(reqwest::Proxy::all(proxy).map_err(map_e)?)
-                .build()
-                .map_err(map_e)?;
+            builder = builder.proxy(reqwest::Proxy::all(proxy).map_err(map_e)?);
         }
+
+        #[cfg(not(target_arch = "wasm32"))]
+        if let Some(timeout) = config.timeout {
+            builder = builder.timeout(core::time::Duration::from_secs(timeout));
+        }
+
+        blockchain.url_client.client = builder.build().map_err(map_e)?;
+
         Ok(blockchain)
     }
 }

src/blockchain/esplora/ureq.rs

@@ -14,6 +14,7 @@
 use std::collections::{HashMap, HashSet};
 use std::io;
 use std::io::Read;
+use std::ops::Deref;
 use std::time::Duration;
 
 #[allow(unused_imports)]
@@ -26,15 +27,16 @@ use bitcoin::hashes::hex::{FromHex, ToHex};
 use bitcoin::hashes::{sha256, Hash};
 use bitcoin::{BlockHeader, Script, Transaction, Txid};
 
-use crate::blockchain::esplora::{EsploraError, EsploraGetHistory};
-use crate::blockchain::utils::{ElectrumLikeSync, ElsGetHistoryRes};
+use super::api::Tx;
+use crate::blockchain::esplora::EsploraError;
 use crate::blockchain::*;
 use crate::database::BatchDatabase;
 use crate::error::Error;
 use crate::FeeRate;
 
-#[derive(Debug)]
-struct UrlClient {
+/// Structure encapsulates ureq Esplora client
+#[derive(Debug, Clone)]
+pub struct UrlClient {
     url: String,
     agent: Agent,
 }
@@ -47,15 +49,7 @@ struct UrlClient {
 pub struct EsploraBlockchain {
     url_client: UrlClient,
     stop_gap: usize,
-}
-
-impl std::convert::From<UrlClient> for EsploraBlockchain {
-    fn from(url_client: UrlClient) -> Self {
-        EsploraBlockchain {
-            url_client,
-            stop_gap: 20,
-        }
-    }
+    concurrency: u8,
 }
 
 impl EsploraBlockchain {
@@ -66,6 +60,7 @@ impl EsploraBlockchain {
                 url: base_url.to_string(),
                 agent: Agent::new(),
             },
+            concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
             stop_gap,
         }
     }
@@ -75,6 +70,12 @@ impl EsploraBlockchain {
         self.url_client.agent = agent;
         self
     }
+
+    /// Set the number of parallel requests the client can make.
+    pub fn with_concurrency(mut self, concurrency: u8) -> Self {
+        self.concurrency = concurrency;
+        self
+    }
 }
 
 impl Blockchain for EsploraBlockchain {
@@ -88,39 +89,144 @@ impl Blockchain for EsploraBlockchain {
         .collect()
     }
 
-    fn setup<D: BatchDatabase, P: Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        self.url_client
-            .electrum_like_setup(self.stop_gap, database, progress_update)
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.url_client._get_tx(txid)?)
-    }
-
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        let _txid = self.url_client._broadcast(tx)?;
+        self.url_client._broadcast(tx)?;
         Ok(())
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(self.url_client._get_height()?)
-    }
-
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         let estimates = self.url_client._get_fee_estimates()?;
         super::into_fee_rate(target, estimates)
     }
 }
 
-impl UrlClient {
-    fn script_to_scripthash(script: &Script) -> String {
-        sha256::Hash::hash(script.as_bytes()).into_inner().to_hex()
-    }
+impl Deref for EsploraBlockchain {
+    type Target = UrlClient;
 
+    fn deref(&self) -> &Self::Target {
+        &self.url_client
+    }
+}
+
+impl StatelessBlockchain for EsploraBlockchain {}
+
+impl GetHeight for EsploraBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.url_client._get_height()?)
+    }
+}
+
+impl GetTx for EsploraBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.url_client._get_tx(txid)?)
+    }
+}
+
+impl GetBlockHash for EsploraBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        let block_header = self.url_client._get_header(height as u32)?;
+        Ok(block_header.block_hash())
+    }
+}
+
+impl WalletSync for EsploraBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        use crate::blockchain::script_sync::Request;
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut tx_index: HashMap<Txid, Tx> = HashMap::new();
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let scripts = script_req
+                        .request()
+                        .take(self.concurrency as usize)
+                        .cloned();
+
+                    let mut handles = vec![];
+                    for script in scripts {
+                        let client = self.url_client.clone();
+                        // make each request in its own thread.
+                        handles.push(std::thread::spawn(move || {
+                            let mut related_txs: Vec<Tx> = client._scripthash_txs(&script, None)?;
+
+                            let n_confirmed =
+                                related_txs.iter().filter(|tx| tx.status.confirmed).count();
+                            // esplora pages on 25 confirmed transactions. If there's 25 or more we
+                            // keep requesting to see if there's more.
+                            if n_confirmed >= 25 {
+                                loop {
+                                    let new_related_txs: Vec<Tx> = client._scripthash_txs(
+                                        &script,
+                                        Some(related_txs.last().unwrap().txid),
+                                    )?;
+                                    let n = new_related_txs.len();
+                                    related_txs.extend(new_related_txs);
+                                    // we've reached the end
+                                    if n < 25 {
+                                        break;
+                                    }
+                                }
+                            }
+                            Result::<_, Error>::Ok(related_txs)
+                        }));
+                    }
+
+                    let txs_per_script: Vec<Vec<Tx>> = handles
+                        .into_iter()
+                        .map(|handle| handle.join().unwrap())
+                        .collect::<Result<_, _>>()?;
+                    let mut satisfaction = vec![];
+
+                    for txs in txs_per_script {
+                        satisfaction.push(
+                            txs.iter()
+                                .map(|tx| (tx.txid, tx.status.block_height))
+                                .collect(),
+                        );
+                        for tx in txs {
+                            tx_index.insert(tx.txid, tx);
+                        }
+                    }
+
+                    script_req.satisfy(satisfaction)?
+                }
+                Request::Conftime(conftime_req) => {
+                    let conftimes = conftime_req
+                        .request()
+                        .map(|txid| {
+                            tx_index
+                                .get(txid)
+                                .expect("must be in index")
+                                .confirmation_time()
+                        })
+                        .collect();
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let full_txs = tx_req
+                        .request()
+                        .map(|txid| {
+                            let tx = tx_index.get(txid).expect("must be in index");
+                            Ok((tx.previous_outputs(), tx.to_tx()))
+                        })
+                        .collect::<Result<_, Error>>()?;
+                    tx_req.satisfy(full_txs)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+
+        Ok(())
+    }
+}
+
+impl UrlClient {
     fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
         let resp = self
             .agent
@@ -200,81 +306,6 @@ impl UrlClient {
         }
     }
 
-    fn _script_get_history(&self, script: &Script) -> Result<Vec<ElsGetHistoryRes>, EsploraError> {
-        let mut result = Vec::new();
-        let scripthash = Self::script_to_scripthash(script);
-
-        // Add the unconfirmed transactions first
-
-        let resp = self
-            .agent
-            .get(&format!(
-                "{}/scripthash/{}/txs/mempool",
-                self.url, scripthash
-            ))
-            .call();
-
-        let v = match resp {
-            Ok(resp) => {
-                let v: Vec<EsploraGetHistory> = resp.into_json()?;
-                Ok(v)
-            }
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }?;
-
-        result.extend(v.into_iter().map(|x| ElsGetHistoryRes {
-            tx_hash: x.txid,
-            height: x.status.block_height.unwrap_or(0) as i32,
-        }));
-
-        debug!(
-            "Found {} mempool txs for {} - {:?}",
-            result.len(),
-            scripthash,
-            script
-        );
-
-        // Then go through all the pages of confirmed transactions
-        let mut last_txid = String::new();
-        loop {
-            let resp = self
-                .agent
-                .get(&format!(
-                    "{}/scripthash/{}/txs/chain/{}",
-                    self.url, scripthash, last_txid
-                ))
-                .call();
-
-            let v = match resp {
-                Ok(resp) => {
-                    let v: Vec<EsploraGetHistory> = resp.into_json()?;
-                    Ok(v)
-                }
-                Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-                Err(e) => Err(EsploraError::Ureq(e)),
-            }?;
-
-            let len = v.len();
-            if let Some(elem) = v.last() {
-                last_txid = elem.txid.to_hex();
-            }
-
-            debug!("... adding {} confirmed transactions", len);
-
-            result.extend(v.into_iter().map(|x| ElsGetHistoryRes {
-                tx_hash: x.txid,
-                height: x.status.block_height.unwrap_or(0) as i32,
-            }));
-
-            if len < 25 {
-                break;
-            }
-        }
-
-        Ok(result)
-    }
-
     fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
         let resp = self
             .agent
@@ -292,6 +323,22 @@ impl UrlClient {
 
         Ok(map)
     }
+
+    fn _scripthash_txs(
+        &self,
+        script: &Script,
+        last_seen: Option<Txid>,
+    ) -> Result<Vec<Tx>, EsploraError> {
+        let script_hash = sha256::Hash::hash(script.as_bytes()).into_inner().to_hex();
+        let url = match last_seen {
+            Some(last_seen) => format!(
+                "{}/scripthash/{}/txs/chain/{}",
+                self.url, script_hash, last_seen
+            ),
+            None => format!("{}/scripthash/{}/txs", self.url, script_hash),
+        };
+        Ok(self.agent.get(&url).call()?.into_json()?)
+    }
 }
 
 fn is_status_not_found(status: u16) -> bool {
@@ -315,84 +362,37 @@ fn into_bytes(resp: Response) -> Result<Vec<u8>, io::Error> {
     Ok(buf)
 }
 
-impl ElectrumLikeSync for UrlClient {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script>>(
-        &self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error> {
-        let mut results = vec![];
-        for script in scripts.into_iter() {
-            let v = self._script_get_history(script)?;
-            results.push(v);
-        }
-        Ok(results)
-    }
-
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid>>(
-        &self,
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error> {
-        let mut results = vec![];
-        for txid in txids.into_iter() {
-            let tx = self._get_tx_no_opt(txid)?;
-            results.push(tx);
-        }
-        Ok(results)
-    }
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32>>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error> {
-        let mut results = vec![];
-        for height in heights.into_iter() {
-            let header = self._get_header(height)?;
-            results.push(header);
-        }
-        Ok(results)
-    }
-}
-
-/// Configuration for an [`EsploraBlockchain`]
-#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
-pub struct EsploraBlockchainConfig {
-    /// Base URL of the esplora service eg. `https://blockstream.info/api/`
-    pub base_url: String,
-    /// Optional URL of the proxy to use to make requests to the Esplora server
-    ///
-    /// The string should be formatted as: `<protocol>://<user>:<password>@host:<port>`.
-    ///
-    /// Note that the format of this value and the supported protocols change slightly between the
-    /// sync version of esplora (using `ureq`) and the async version (using `reqwest`). For more
-    /// details check with the documentation of the two crates. Both of them are compiled with
-    /// the `socks` feature enabled.
-    ///
-    /// The proxy is ignored when targeting `wasm32`.
-    pub proxy: Option<String>,
-    /// Socket read timeout.
-    pub timeout_read: u64,
-    /// Socket write timeout.
-    pub timeout_write: u64,
-    /// Stop searching addresses for transactions after finding an unused gap of this length.
-    pub stop_gap: usize,
-}
-
 impl ConfigurableBlockchain for EsploraBlockchain {
-    type Config = EsploraBlockchainConfig;
+    type Config = super::EsploraBlockchainConfig;
 
     fn from_config(config: &Self::Config) -> Result<Self, Error> {
-        let mut agent_builder = ureq::AgentBuilder::new()
-            .timeout_read(Duration::from_secs(config.timeout_read))
-            .timeout_write(Duration::from_secs(config.timeout_write));
+        let mut agent_builder = ureq::AgentBuilder::new();
+
+        if let Some(timeout) = config.timeout {
+            agent_builder = agent_builder.timeout(Duration::from_secs(timeout));
+        }
 
         if let Some(proxy) = &config.proxy {
             agent_builder = agent_builder
                 .proxy(Proxy::new(proxy).map_err(|e| Error::Esplora(Box::new(e.into())))?);
         }
 
-        Ok(
-            EsploraBlockchain::new(config.base_url.as_str(), config.stop_gap)
-                .with_agent(agent_builder.build()),
-        )
+        let mut blockchain = EsploraBlockchain::new(config.base_url.as_str(), config.stop_gap)
+            .with_agent(agent_builder.build());
+
+        if let Some(concurrency) = config.concurrency {
+            blockchain = blockchain.with_concurrency(concurrency);
+        }
+
+        Ok(blockchain)
+    }
+}
+
+impl From<ureq::Error> for EsploraError {
+    fn from(e: ureq::Error) -> Self {
+        match e {
+            ureq::Error::Status(code, _) => EsploraError::HttpResponse(code),
+            e => EsploraError::Ureq(e),
+        }
     }
 }

src/blockchain/mod.rs

@@ -21,14 +21,12 @@ use std::ops::Deref;
 use std::sync::mpsc::{channel, Receiver, Sender};
 use std::sync::Arc;
 
-use bitcoin::{Transaction, Txid};
+use bitcoin::{BlockHash, Transaction, Txid};
 
 use crate::database::BatchDatabase;
 use crate::error::Error;
-use crate::FeeRate;
-
-#[cfg(any(feature = "electrum", feature = "esplora"))]
-pub(crate) mod utils;
+use crate::wallet::{wallet_name_from_descriptor, Wallet};
+use crate::{FeeRate, KeychainKind};
 
 #[cfg(any(
     feature = "electrum",
@@ -37,6 +35,8 @@ pub(crate) mod utils;
     feature = "rpc"
 ))]
 pub mod any;
+mod script_sync;
+
 #[cfg(any(
     feature = "electrum",
     feature = "esplora",
@@ -87,28 +87,57 @@ pub enum Capability {
 
 /// Trait that defines the actions that must be supported by a blockchain backend
 #[maybe_async]
-pub trait Blockchain {
+pub trait Blockchain: WalletSync + GetHeight + GetTx + GetBlockHash {
     /// Return the set of [`Capability`] supported by this backend
     fn get_capabilities(&self) -> HashSet<Capability>;
+    /// Broadcast a transaction
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error>;
+    /// Estimate the fee rate required to confirm a transaction in a given `target` of blocks
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error>;
+}
 
+/// Trait for getting the current height of the blockchain.
+#[maybe_async]
+pub trait GetHeight {
+    /// Return the current height
+    fn get_height(&self) -> Result<u32, Error>;
+}
+
+#[maybe_async]
+/// Trait for getting a transaction by txid
+pub trait GetTx {
+    /// Fetch a transaction given its txid
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
+}
+
+#[maybe_async]
+/// Trait for getting block hash by block height
+pub trait GetBlockHash {
+    /// fetch block hash given its height
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error>;
+}
+
+/// Trait for blockchains that can sync by updating the database directly.
+#[maybe_async]
+pub trait WalletSync {
     /// Setup the backend and populate the internal database for the first time
     ///
-    /// This method is the equivalent of [`Blockchain::sync`], but it's guaranteed to only be
+    /// This method is the equivalent of [`Self::wallet_sync`], but it's guaranteed to only be
     /// called once, at the first [`Wallet::sync`](crate::wallet::Wallet::sync).
     ///
     /// The rationale behind the distinction between `sync` and `setup` is that some custom backends
     /// might need to perform specific actions only the first time they are synced.
     ///
     /// For types that do not have that distinction, only this method can be implemented, since
-    /// [`Blockchain::sync`] defaults to calling this internally if not overridden.
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
+    /// [`WalletSync::wallet_sync`] defaults to calling this internally if not overridden.
+    /// Populate the internal database with transactions and UTXOs
+    fn wallet_setup<D: BatchDatabase>(
         &self,
         database: &mut D,
-        progress_update: P,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error>;
-    /// Populate the internal database with transactions and UTXOs
-    ///
-    /// If not overridden, it defaults to calling [`Blockchain::setup`] internally.
+
+    /// If not overridden, it defaults to calling [`Self::wallet_setup`] internally.
     ///
     /// This method should implement the logic required to iterate over the list of the wallet's
     /// script_pubkeys using [`Database::iter_script_pubkeys`] and look for relevant transactions
@@ -125,23 +154,13 @@ pub trait Blockchain {
     /// [`BatchOperations::set_tx`]: crate::database::BatchOperations::set_tx
     /// [`BatchOperations::set_utxo`]: crate::database::BatchOperations::set_utxo
     /// [`BatchOperations::del_utxo`]: crate::database::BatchOperations::del_utxo
-    fn sync<D: BatchDatabase, P: 'static + Progress>(
+    fn wallet_sync<D: BatchDatabase>(
         &self,
         database: &mut D,
-        progress_update: P,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error> {
-        maybe_await!(self.setup(database, progress_update))
+        maybe_await!(self.wallet_setup(database, progress_update))
     }
-
-    /// Fetch a transaction from the blockchain given its txid
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
-    /// Broadcast a transaction
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error>;
-
-    /// Return the current height
-    fn get_height(&self) -> Result<u32, Error>;
-    /// Estimate the fee rate required to confirm a transaction in a given `target` of blocks
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error>;
 }
 
 /// Trait for [`Blockchain`] types that can be created given a configuration
@@ -153,12 +172,112 @@ pub trait ConfigurableBlockchain: Blockchain + Sized {
     fn from_config(config: &Self::Config) -> Result<Self, Error>;
 }
 
+/// Trait for blockchains that don't contain any state
+///
+/// Statless blockchains can be used to sync multiple wallets with different descriptors.
+///
+/// [`BlockchainFactory`] is automatically implemented for `Arc<T>` where `T` is a stateless
+/// blockchain.
+pub trait StatelessBlockchain: Blockchain {}
+
+/// Trait for a factory of blockchains that share the underlying connection or configuration
+#[cfg_attr(
+    not(feature = "async-interface"),
+    doc = r##"
+## Example
+
+This example shows how to sync multiple walles and return the sum of their balances
+
+```no_run
+# use bdk::Error;
+# use bdk::blockchain::*;
+# use bdk::database::*;
+# use bdk::wallet::*;
+# use bdk::*;
+fn sum_of_balances<B: BlockchainFactory>(blockchain_factory: B, wallets: &[Wallet<MemoryDatabase>]) -> Result<Balance, Error> {
+    Ok(wallets
+        .iter()
+        .map(|w| -> Result<_, Error> {
+            blockchain_factory.sync_wallet(&w, None, SyncOptions::default())?;
+            w.get_balance()
+        })
+        .collect::<Result<Vec<_>, _>>()?
+        .into_iter()
+        .sum())
+}
+```
+"##
+)]
+pub trait BlockchainFactory {
+    /// The type returned when building a blockchain from this factory
+    type Inner: Blockchain;
+
+    /// Build a new blockchain for the given descriptor wallet_name
+    ///
+    /// If `override_skip_blocks` is `None`, the returned blockchain will inherit the number of blocks
+    /// from the factory. Since it's not possible to override the value to `None`, set it to
+    /// `Some(0)` to rescan from the genesis.
+    fn build(
+        &self,
+        wallet_name: &str,
+        override_skip_blocks: Option<u32>,
+    ) -> Result<Self::Inner, Error>;
+
+    /// Build a new blockchain for a given wallet
+    ///
+    /// Internally uses [`wallet_name_from_descriptor`] to derive the name, and then calls
+    /// [`BlockchainFactory::build`] to create the blockchain instance.
+    fn build_for_wallet<D: BatchDatabase>(
+        &self,
+        wallet: &Wallet<D>,
+        override_skip_blocks: Option<u32>,
+    ) -> Result<Self::Inner, Error> {
+        let wallet_name = wallet_name_from_descriptor(
+            wallet.public_descriptor(KeychainKind::External)?.unwrap(),
+            wallet.public_descriptor(KeychainKind::Internal)?,
+            wallet.network(),
+            wallet.secp_ctx(),
+        )?;
+        self.build(&wallet_name, override_skip_blocks)
+    }
+
+    /// Use [`BlockchainFactory::build_for_wallet`] to get a blockchain, then sync the wallet
+    ///
+    /// This can be used when a new blockchain would only be used to sync a wallet and then
+    /// immediately dropped. Keep in mind that specific blockchain factories may perform slow
+    /// operations to build a blockchain for a given wallet, so if a wallet needs to be synced
+    /// often it's recommended to use [`BlockchainFactory::build_for_wallet`] to reuse the same
+    /// blockchain multiple times.
+    #[cfg(not(any(target_arch = "wasm32", feature = "async-interface")))]
+    #[cfg_attr(
+        docsrs,
+        doc(cfg(not(any(target_arch = "wasm32", feature = "async-interface"))))
+    )]
+    fn sync_wallet<D: BatchDatabase>(
+        &self,
+        wallet: &Wallet<D>,
+        override_skip_blocks: Option<u32>,
+        sync_options: crate::wallet::SyncOptions,
+    ) -> Result<(), Error> {
+        let blockchain = self.build_for_wallet(wallet, override_skip_blocks)?;
+        wallet.sync(&blockchain, sync_options)
+    }
+}
+
+impl<T: StatelessBlockchain> BlockchainFactory for Arc<T> {
+    type Inner = Self;
+
+    fn build(&self, _wallet_name: &str, _override_skip_blocks: Option<u32>) -> Result<Self, Error> {
+        Ok(Arc::clone(self))
+    }
+}
+
 /// Data sent with a progress update over a [`channel`]
 pub type ProgressData = (f32, Option<String>);
 
-/// Trait for types that can receive and process progress updates during [`Blockchain::sync`] and
-/// [`Blockchain::setup`]
-pub trait Progress: Send {
+/// Trait for types that can receive and process progress updates during [`WalletSync::wallet_sync`] and
+/// [`WalletSync::wallet_setup`]
+pub trait Progress: Send + 'static + core::fmt::Debug {
     /// Send a new progress update
     ///
     /// The `progress` value should be in the range 0.0 - 100.0, and the `message` value is an
@@ -183,7 +302,7 @@ impl Progress for Sender<ProgressData> {
 }
 
 /// Type that implements [`Progress`] and drops every update received
-#[derive(Clone, Copy)]
+#[derive(Clone, Copy, Default, Debug)]
 pub struct NoopProgress;
 
 /// Create a new instance of [`NoopProgress`]
@@ -198,7 +317,7 @@ impl Progress for NoopProgress {
 }
 
 /// Type that implements [`Progress`] and logs at level `INFO` every update received
-#[derive(Clone, Copy)]
+#[derive(Clone, Copy, Default, Debug)]
 pub struct LogProgress;
 
 /// Create a new instance of [`LogProgress`]
@@ -224,33 +343,51 @@ impl<T: Blockchain> Blockchain for Arc<T> {
         maybe_await!(self.deref().get_capabilities())
     }
 
-    fn setup<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(self.deref().setup(database, progress_update))
-    }
-
-    fn sync<D: BatchDatabase, P: 'static + Progress>(
-        &self,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        maybe_await!(self.deref().sync(database, progress_update))
-    }
-
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        maybe_await!(self.deref().get_tx(txid))
-    }
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         maybe_await!(self.deref().broadcast(tx))
     }
 
-    fn get_height(&self) -> Result<u32, Error> {
-        maybe_await!(self.deref().get_height())
-    }
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         maybe_await!(self.deref().estimate_fee(target))
     }
 }
+
+#[maybe_async]
+impl<T: GetTx> GetTx for Arc<T> {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        maybe_await!(self.deref().get_tx(txid))
+    }
+}
+
+#[maybe_async]
+impl<T: GetHeight> GetHeight for Arc<T> {
+    fn get_height(&self) -> Result<u32, Error> {
+        maybe_await!(self.deref().get_height())
+    }
+}
+
+#[maybe_async]
+impl<T: GetBlockHash> GetBlockHash for Arc<T> {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        maybe_await!(self.deref().get_block_hash(height))
+    }
+}
+
+#[maybe_async]
+impl<T: WalletSync> WalletSync for Arc<T> {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(self.deref().wallet_setup(database, progress_update))
+    }
+
+    fn wallet_sync<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(self.deref().wallet_sync(database, progress_update))
+    }
+}

src/blockchain/script_sync.rs

@@ -0,0 +1,467 @@
+/*!
+This models a how a sync happens where you have a server that you send your script pubkeys to and it
+returns associated transactions i.e. electrum.
+*/
+#![allow(dead_code)]
+use crate::{
+    database::{BatchDatabase, BatchOperations, DatabaseUtils},
+    error::MissingCachedScripts,
+    wallet::time::Instant,
+    BlockTime, Error, KeychainKind, LocalUtxo, TransactionDetails,
+};
+use bitcoin::{OutPoint, Script, Transaction, TxOut, Txid};
+use log::*;
+use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque};
+
+/// A request for on-chain information
+pub enum Request<'a, D: BatchDatabase> {
+    /// A request for transactions related to script pubkeys.
+    Script(ScriptReq<'a, D>),
+    /// A request for confirmation times for some transactions.
+    Conftime(ConftimeReq<'a, D>),
+    /// A request for full transaction details of some transactions.
+    Tx(TxReq<'a, D>),
+    /// Requests are finished here's a batch database update to reflect data gathered.
+    Finish(D::Batch),
+}
+
+/// starts a sync
+pub fn start<D: BatchDatabase>(db: &D, stop_gap: usize) -> Result<Request<'_, D>, Error> {
+    use rand::seq::SliceRandom;
+    let mut keychains = vec![KeychainKind::Internal, KeychainKind::External];
+    // shuffling improve privacy, the server doesn't know my first request is from my internal or external addresses
+    keychains.shuffle(&mut rand::thread_rng());
+    let keychain = keychains.pop().unwrap();
+    let scripts_needed = db
+        .iter_script_pubkeys(Some(keychain))?
+        .into_iter()
+        .collect::<VecDeque<_>>();
+    let state = State::new(db);
+
+    Ok(Request::Script(ScriptReq {
+        state,
+        initial_scripts_needed: scripts_needed.len(),
+        scripts_needed,
+        script_index: 0,
+        stop_gap,
+        keychain,
+        next_keychains: keychains,
+    }))
+}
+
+pub struct ScriptReq<'a, D: BatchDatabase> {
+    state: State<'a, D>,
+    script_index: usize,
+    initial_scripts_needed: usize, // if this is 1, we assume the descriptor is not derivable
+    scripts_needed: VecDeque<Script>,
+    stop_gap: usize,
+    keychain: KeychainKind,
+    next_keychains: Vec<KeychainKind>,
+}
+
+/// The sync starts by returning script pubkeys we are interested in.
+impl<'a, D: BatchDatabase> ScriptReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Script> + Clone {
+        self.scripts_needed.iter()
+    }
+
+    pub fn satisfy(
+        mut self,
+        // we want to know the txids assoiciated with the script and their height
+        txids: Vec<Vec<(Txid, Option<u32>)>>,
+    ) -> Result<Request<'a, D>, Error> {
+        for (txid_list, script) in txids.iter().zip(self.scripts_needed.iter()) {
+            debug!(
+                "found {} transactions for script pubkey {}",
+                txid_list.len(),
+                script
+            );
+            if !txid_list.is_empty() {
+                // the address is active
+                self.state
+                    .last_active_index
+                    .insert(self.keychain, self.script_index);
+            }
+
+            for (txid, height) in txid_list {
+                // have we seen this txid already?
+                match self.state.db.get_tx(txid, true)? {
+                    Some(mut details) => {
+                        let old_height = details.confirmation_time.as_ref().map(|x| x.height);
+                        match (old_height, height) {
+                            (None, Some(_)) => {
+                                // It looks like the tx has confirmed since we last saw it -- we
+                                // need to know the confirmation time.
+                                self.state.tx_missing_conftime.insert(*txid, details);
+                            }
+                            (Some(old_height), Some(new_height)) if old_height != *new_height => {
+                                // The height of the tx has changed !? -- It's a reorg get the new confirmation time.
+                                self.state.tx_missing_conftime.insert(*txid, details);
+                            }
+                            (Some(_), None) => {
+                                // A re-org where the tx is not in the chain anymore.
+                                details.confirmation_time = None;
+                                self.state.finished_txs.push(details);
+                            }
+                            _ => self.state.finished_txs.push(details),
+                        }
+                    }
+                    None => {
+                        // we've never seen it let's get the whole thing
+                        self.state.tx_needed.insert(*txid);
+                    }
+                };
+            }
+
+            self.script_index += 1;
+        }
+
+        self.scripts_needed.drain(..txids.len());
+
+        // last active index: 0 => No last active
+        let last = self
+            .state
+            .last_active_index
+            .get(&self.keychain)
+            .map(|&l| l + 1)
+            .unwrap_or(0);
+        // remaining scripts left to check
+        let remaining = self.scripts_needed.len();
+        // difference between current index and last active index
+        let current_gap = self.script_index - last;
+
+        // this is a hack to check whether the scripts are coming from a derivable descriptor
+        // we assume for non-derivable descriptors, the initial script count is always 1
+        let is_derivable = self.initial_scripts_needed > 1;
+
+        debug!(
+            "sync: last={}, remaining={}, diff={}, stop_gap={}",
+            last, remaining, current_gap, self.stop_gap
+        );
+
+        if is_derivable {
+            if remaining > 0 {
+                // we still have scriptPubKeys to do requests for
+                return Ok(Request::Script(self));
+            }
+
+            if last > 0 && current_gap < self.stop_gap {
+                // current gap is not large enough to stop, but we are unable to keep checking since
+                // we have exhausted cached scriptPubKeys, so return error
+                let err = MissingCachedScripts {
+                    last_count: self.script_index,
+                    missing_count: self.stop_gap - current_gap,
+                };
+                return Err(Error::MissingCachedScripts(err));
+            }
+
+            // we have exhausted cached scriptPubKeys and found no txs, continue
+        }
+
+        debug!(
+            "finished scanning for txs of keychain {:?} at index {:?}",
+            self.keychain, last
+        );
+
+        if let Some(keychain) = self.next_keychains.pop() {
+            // we still have another keychain to request txs with
+            let scripts_needed = self
+                .state
+                .db
+                .iter_script_pubkeys(Some(keychain))?
+                .into_iter()
+                .collect::<VecDeque<_>>();
+
+            self.keychain = keychain;
+            self.script_index = 0;
+            self.initial_scripts_needed = scripts_needed.len();
+            self.scripts_needed = scripts_needed;
+            return Ok(Request::Script(self));
+        }
+
+        // We have finished requesting txids, let's get the actual txs.
+        Ok(Request::Tx(TxReq { state: self.state }))
+    }
+}
+
+/// Then we get full transactions
+pub struct TxReq<'a, D> {
+    state: State<'a, D>,
+}
+
+impl<'a, D: BatchDatabase> TxReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Txid> + Clone {
+        self.state.tx_needed.iter()
+    }
+
+    pub fn satisfy(
+        mut self,
+        tx_details: Vec<(Vec<Option<TxOut>>, Transaction)>,
+    ) -> Result<Request<'a, D>, Error> {
+        let tx_details: Vec<TransactionDetails> = tx_details
+            .into_iter()
+            .zip(self.state.tx_needed.iter())
+            .map(|((vout, tx), txid)| {
+                debug!("found tx_details for {}", txid);
+                assert_eq!(tx.txid(), *txid);
+                let mut sent: u64 = 0;
+                let mut received: u64 = 0;
+                let mut inputs_sum: u64 = 0;
+                let mut outputs_sum: u64 = 0;
+
+                for (txout, (_input_index, input)) in
+                    vout.into_iter().zip(tx.input.iter().enumerate())
+                {
+                    let txout = match txout {
+                        Some(txout) => txout,
+                        None => {
+                            // skip coinbase inputs
+                            debug_assert!(
+                                input.previous_output.is_null(),
+                                "prevout should only be missing for coinbase"
+                            );
+                            continue;
+                        }
+                    };
+                    // Verify this input if requested via feature flag
+                    #[cfg(feature = "verify")]
+                    {
+                        use crate::wallet::verify::VerifyError;
+                        let serialized_tx = bitcoin::consensus::serialize(&tx);
+                        bitcoinconsensus::verify(
+                            txout.script_pubkey.to_bytes().as_ref(),
+                            txout.value,
+                            &serialized_tx,
+                            _input_index,
+                        )
+                        .map_err(VerifyError::from)?;
+                    }
+                    inputs_sum += txout.value;
+                    if self.state.db.is_mine(&txout.script_pubkey)? {
+                        sent += txout.value;
+                    }
+                }
+
+                for out in &tx.output {
+                    outputs_sum += out.value;
+                    if self.state.db.is_mine(&out.script_pubkey)? {
+                        received += out.value;
+                    }
+                }
+                // we need to saturating sub since we want coinbase txs to map to 0 fee and
+                // this subtraction will be negative for coinbase txs.
+                let fee = inputs_sum.saturating_sub(outputs_sum);
+                Result::<_, Error>::Ok(TransactionDetails {
+                    txid: *txid,
+                    transaction: Some(tx),
+                    received,
+                    sent,
+                    // we're going to fill this in later
+                    confirmation_time: None,
+                    fee: Some(fee),
+                })
+            })
+            .collect::<Result<Vec<_>, _>>()?;
+
+        for tx_detail in tx_details {
+            self.state.tx_needed.remove(&tx_detail.txid);
+            self.state
+                .tx_missing_conftime
+                .insert(tx_detail.txid, tx_detail);
+        }
+
+        if !self.state.tx_needed.is_empty() {
+            Ok(Request::Tx(self))
+        } else {
+            Ok(Request::Conftime(ConftimeReq { state: self.state }))
+        }
+    }
+}
+
+/// Final step is to get confirmation times
+pub struct ConftimeReq<'a, D> {
+    state: State<'a, D>,
+}
+
+impl<'a, D: BatchDatabase> ConftimeReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Txid> + Clone {
+        self.state.tx_missing_conftime.keys()
+    }
+
+    pub fn satisfy(
+        mut self,
+        confirmation_times: Vec<Option<BlockTime>>,
+    ) -> Result<Request<'a, D>, Error> {
+        let conftime_needed = self
+            .request()
+            .cloned()
+            .take(confirmation_times.len())
+            .collect::<Vec<_>>();
+        for (confirmation_time, txid) in confirmation_times.into_iter().zip(conftime_needed.iter())
+        {
+            debug!("confirmation time for {} was {:?}", txid, confirmation_time);
+            if let Some(mut tx_details) = self.state.tx_missing_conftime.remove(txid) {
+                tx_details.confirmation_time = confirmation_time;
+                self.state.finished_txs.push(tx_details);
+            }
+        }
+
+        if self.state.tx_missing_conftime.is_empty() {
+            Ok(Request::Finish(self.state.into_db_update()?))
+        } else {
+            Ok(Request::Conftime(self))
+        }
+    }
+}
+
+struct State<'a, D> {
+    db: &'a D,
+    last_active_index: HashMap<KeychainKind, usize>,
+    /// Transactions where we need to get the full details
+    tx_needed: BTreeSet<Txid>,
+    /// Transacitions that we know everything about
+    finished_txs: Vec<TransactionDetails>,
+    /// Transactions that discovered conftimes should be inserted into
+    tx_missing_conftime: BTreeMap<Txid, TransactionDetails>,
+    /// The start of the sync
+    start_time: Instant,
+    /// Missing number of scripts to cache per keychain
+    missing_script_counts: HashMap<KeychainKind, usize>,
+}
+
+impl<'a, D: BatchDatabase> State<'a, D> {
+    fn new(db: &'a D) -> Self {
+        State {
+            db,
+            last_active_index: HashMap::default(),
+            finished_txs: vec![],
+            tx_needed: BTreeSet::default(),
+            tx_missing_conftime: BTreeMap::default(),
+            start_time: Instant::new(),
+            missing_script_counts: HashMap::default(),
+        }
+    }
+    fn into_db_update(self) -> Result<D::Batch, Error> {
+        debug_assert!(self.tx_needed.is_empty() && self.tx_missing_conftime.is_empty());
+        let existing_txs = self.db.iter_txs(false)?;
+        let existing_txids: HashSet<Txid> = existing_txs.iter().map(|tx| tx.txid).collect();
+        let finished_txs = make_txs_consistent(&self.finished_txs);
+        let observed_txids: HashSet<Txid> = finished_txs.iter().map(|tx| tx.txid).collect();
+        let txids_to_delete = existing_txids.difference(&observed_txids);
+
+        // Ensure `last_active_index` does not decrement database's current state.
+        let index_updates = self
+            .last_active_index
+            .iter()
+            .map(|(keychain, sync_index)| {
+                let sync_index = *sync_index as u32;
+                let index_res = match self.db.get_last_index(*keychain) {
+                    Ok(Some(db_index)) => Ok(std::cmp::max(db_index, sync_index)),
+                    Ok(None) => Ok(sync_index),
+                    Err(err) => Err(err),
+                };
+                index_res.map(|index| (*keychain, index))
+            })
+            .collect::<Result<Vec<(KeychainKind, u32)>, _>>()?;
+
+        let mut batch = self.db.begin_batch();
+
+        // Delete old txs that no longer exist
+        for txid in txids_to_delete {
+            if let Some(raw_tx) = self.db.get_raw_tx(txid)? {
+                for i in 0..raw_tx.output.len() {
+                    // Also delete any utxos from the txs that no longer exist.
+                    let _ = batch.del_utxo(&OutPoint {
+                        txid: *txid,
+                        vout: i as u32,
+                    })?;
+                }
+            } else {
+                unreachable!("we should always have the raw tx");
+            }
+            batch.del_tx(txid, true)?;
+        }
+
+        let mut spent_utxos = HashSet::new();
+
+        // track all the spent utxos
+        for finished_tx in &finished_txs {
+            let tx = finished_tx
+                .transaction
+                .as_ref()
+                .expect("transaction will always be present here");
+            for input in &tx.input {
+                spent_utxos.insert(&input.previous_output);
+            }
+        }
+
+        // set every utxo we observed, unless it's already spent
+        // we don't do this in the loop above as we want to know all the spent outputs before
+        // adding the non-spent to the batch in case there are new tranasactions
+        // that spend form each other.
+        for finished_tx in &finished_txs {
+            let tx = finished_tx
+                .transaction
+                .as_ref()
+                .expect("transaction will always be present here");
+            for (i, output) in tx.output.iter().enumerate() {
+                if let Some((keychain, _)) =
+                    self.db.get_path_from_script_pubkey(&output.script_pubkey)?
+                {
+                    // add utxos we own from the new transactions we've seen.
+                    let outpoint = OutPoint {
+                        txid: finished_tx.txid,
+                        vout: i as u32,
+                    };
+
+                    batch.set_utxo(&LocalUtxo {
+                        outpoint,
+                        txout: output.clone(),
+                        keychain,
+                        // Is this UTXO in the spent_utxos set?
+                        is_spent: spent_utxos.get(&outpoint).is_some(),
+                    })?;
+                }
+            }
+
+            batch.set_tx(finished_tx)?;
+        }
+
+        // apply index updates
+        for (keychain, new_index) in index_updates {
+            debug!("updating index ({}, {})", keychain.as_byte(), new_index);
+            batch.set_last_index(keychain, new_index)?;
+        }
+
+        info!(
+            "finished setup, elapsed {:?}ms",
+            self.start_time.elapsed().as_millis()
+        );
+        Ok(batch)
+    }
+}
+
+/// Remove conflicting transactions -- tie breaking them by fee.
+fn make_txs_consistent(txs: &[TransactionDetails]) -> Vec<&TransactionDetails> {
+    let mut utxo_index: HashMap<OutPoint, &TransactionDetails> = HashMap::default();
+    for tx in txs {
+        for input in &tx.transaction.as_ref().unwrap().input {
+            utxo_index
+                .entry(input.previous_output)
+                .and_modify(|existing| match (tx.fee, existing.fee) {
+                    (Some(fee), Some(existing_fee)) if fee > existing_fee => *existing = tx,
+                    (Some(_), None) => *existing = tx,
+                    _ => { /* leave it the same */ }
+                })
+                .or_insert(tx);
+        }
+    }
+
+    utxo_index
+        .into_iter()
+        .map(|(_, tx)| (tx.txid, tx))
+        .collect::<HashMap<_, _>>()
+        .into_iter()
+        .map(|(_, tx)| tx)
+        .collect()
+}

src/blockchain/utils.rs

@@ -1,388 +0,0 @@
-// Bitcoin Dev Kit
-// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
-//
-// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
-//
-// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
-// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
-// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
-// You may not use this file except in accordance with one or both of these
-// licenses.
-
-use std::collections::{HashMap, HashSet};
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
-use rand::seq::SliceRandom;
-use rand::thread_rng;
-
-use bitcoin::{BlockHeader, OutPoint, Script, Transaction, Txid};
-
-use super::*;
-use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
-use crate::error::Error;
-use crate::types::{BlockTime, KeychainKind, LocalUtxo, TransactionDetails};
-use crate::wallet::time::Instant;
-use crate::wallet::utils::ChunksIterator;
-
-#[derive(Debug)]
-pub struct ElsGetHistoryRes {
-    pub height: i32,
-    pub tx_hash: Txid,
-}
-
-/// Implements the synchronization logic for an Electrum-like client.
-#[maybe_async]
-pub trait ElectrumLikeSync {
-    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script> + Clone>(
-        &self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error>;
-
-    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid> + Clone>(
-        &self,
-        txids: I,
-    ) -> Result<Vec<Transaction>, Error>;
-
-    fn els_batch_block_header<I: IntoIterator<Item = u32> + Clone>(
-        &self,
-        heights: I,
-    ) -> Result<Vec<BlockHeader>, Error>;
-
-    // Provided methods down here...
-
-    fn electrum_like_setup<D: BatchDatabase, P: Progress>(
-        &self,
-        stop_gap: usize,
-        db: &mut D,
-        _progress_update: P,
-    ) -> Result<(), Error> {
-        // TODO: progress
-        let start = Instant::new();
-        debug!("start setup");
-
-        let chunk_size = stop_gap;
-
-        let mut history_txs_id = HashSet::new();
-        let mut txid_height = HashMap::new();
-        let mut max_indexes = HashMap::new();
-
-        let mut wallet_chains = vec![KeychainKind::Internal, KeychainKind::External];
-        // shuffling improve privacy, the server doesn't know my first request is from my internal or external addresses
-        wallet_chains.shuffle(&mut thread_rng());
-        // download history of our internal and external script_pubkeys
-        for keychain in wallet_chains.iter() {
-            let script_iter = db.iter_script_pubkeys(Some(*keychain))?.into_iter();
-
-            for (i, chunk) in ChunksIterator::new(script_iter, stop_gap).enumerate() {
-                // TODO if i == last, should create another chunk of addresses in db
-                let call_result: Vec<Vec<ElsGetHistoryRes>> =
-                    maybe_await!(self.els_batch_script_get_history(chunk.iter()))?;
-                let max_index = call_result
-                    .iter()
-                    .enumerate()
-                    .filter_map(|(i, v)| v.first().map(|_| i as u32))
-                    .max();
-                if let Some(max) = max_index {
-                    max_indexes.insert(keychain, max + (i * chunk_size) as u32);
-                }
-                let flattened: Vec<ElsGetHistoryRes> = call_result.into_iter().flatten().collect();
-                debug!("#{} of {:?} results:{}", i, keychain, flattened.len());
-                if flattened.is_empty() {
-                    // Didn't find anything in the last `stop_gap` script_pubkeys, breaking
-                    break;
-                }
-
-                for el in flattened {
-                    // el.height = -1 means unconfirmed with unconfirmed parents
-                    // el.height =  0 means unconfirmed with confirmed parents
-                    // but we treat those tx the same
-                    if el.height <= 0 {
-                        txid_height.insert(el.tx_hash, None);
-                    } else {
-                        txid_height.insert(el.tx_hash, Some(el.height as u32));
-                    }
-                    history_txs_id.insert(el.tx_hash);
-                }
-            }
-        }
-
-        // saving max indexes
-        info!("max indexes are: {:?}", max_indexes);
-        for keychain in wallet_chains.iter() {
-            if let Some(index) = max_indexes.get(keychain) {
-                db.set_last_index(*keychain, *index)?;
-            }
-        }
-
-        // get db status
-        let txs_details_in_db: HashMap<Txid, TransactionDetails> = db
-            .iter_txs(false)?
-            .into_iter()
-            .map(|tx| (tx.txid, tx))
-            .collect();
-        let txs_raw_in_db: HashMap<Txid, Transaction> = db
-            .iter_raw_txs()?
-            .into_iter()
-            .map(|tx| (tx.txid(), tx))
-            .collect();
-        let utxos_deps = utxos_deps(db, &txs_raw_in_db)?;
-
-        // download new txs and headers
-        let new_txs = maybe_await!(self.download_and_save_needed_raw_txs(
-            &history_txs_id,
-            &txs_raw_in_db,
-            chunk_size,
-            db
-        ))?;
-        let new_timestamps = maybe_await!(self.download_needed_headers(
-            &txid_height,
-            &txs_details_in_db,
-            chunk_size
-        ))?;
-
-        let mut batch = db.begin_batch();
-
-        // save any tx details not in db but in history_txs_id or with different height/timestamp
-        for txid in history_txs_id.iter() {
-            let height = txid_height.get(txid).cloned().flatten();
-            let timestamp = new_timestamps.get(txid).cloned();
-            if let Some(tx_details) = txs_details_in_db.get(txid) {
-                // check if tx height matches, otherwise updates it. timestamp is not in the if clause
-                // because we are not asking headers for confirmed tx we know about
-                if tx_details.confirmation_time.as_ref().map(|c| c.height) != height {
-                    let confirmation_time = BlockTime::new(height, timestamp);
-                    let mut new_tx_details = tx_details.clone();
-                    new_tx_details.confirmation_time = confirmation_time;
-                    batch.set_tx(&new_tx_details)?;
-                }
-            } else {
-                save_transaction_details_and_utxos(
-                    txid,
-                    db,
-                    timestamp,
-                    height,
-                    &mut batch,
-                    &utxos_deps,
-                )?;
-            }
-        }
-
-        // remove any tx details in db but not in history_txs_id
-        for txid in txs_details_in_db.keys() {
-            if !history_txs_id.contains(txid) {
-                batch.del_tx(txid, false)?;
-            }
-        }
-
-        // remove any spent utxo
-        for new_tx in new_txs.iter() {
-            for input in new_tx.input.iter() {
-                batch.del_utxo(&input.previous_output)?;
-            }
-        }
-
-        db.commit_batch(batch)?;
-        info!("finish setup, elapsed {:?}ms", start.elapsed().as_millis());
-
-        Ok(())
-    }
-
-    /// download txs identified by `history_txs_id` and theirs previous outputs if not already present in db
-    fn download_and_save_needed_raw_txs<D: BatchDatabase>(
-        &self,
-        history_txs_id: &HashSet<Txid>,
-        txs_raw_in_db: &HashMap<Txid, Transaction>,
-        chunk_size: usize,
-        db: &mut D,
-    ) -> Result<Vec<Transaction>, Error> {
-        let mut txs_downloaded = vec![];
-        let txids_raw_in_db: HashSet<Txid> = txs_raw_in_db.keys().cloned().collect();
-        let txids_to_download: Vec<&Txid> = history_txs_id.difference(&txids_raw_in_db).collect();
-        if !txids_to_download.is_empty() {
-            info!("got {} txs to download", txids_to_download.len());
-            txs_downloaded.extend(maybe_await!(self.download_and_save_in_chunks(
-                txids_to_download,
-                chunk_size,
-                db,
-            ))?);
-            let mut prev_txids = HashSet::new();
-            let mut txids_downloaded = HashSet::new();
-            for tx in txs_downloaded.iter() {
-                txids_downloaded.insert(tx.txid());
-                // add every previous input tx, but skip coinbase
-                for input in tx.input.iter().filter(|i| !i.previous_output.is_null()) {
-                    prev_txids.insert(input.previous_output.txid);
-                }
-            }
-            let already_present: HashSet<Txid> =
-                txids_downloaded.union(&txids_raw_in_db).cloned().collect();
-            let prev_txs_to_download: Vec<&Txid> =
-                prev_txids.difference(&already_present).collect();
-            info!("{} previous txs to download", prev_txs_to_download.len());
-            txs_downloaded.extend(maybe_await!(self.download_and_save_in_chunks(
-                prev_txs_to_download,
-                chunk_size,
-                db,
-            ))?);
-        }
-
-        Ok(txs_downloaded)
-    }
-
-    /// download headers at heights in `txid_height` if tx details not already present, returns a map Txid -> timestamp
-    fn download_needed_headers(
-        &self,
-        txid_height: &HashMap<Txid, Option<u32>>,
-        txs_details_in_db: &HashMap<Txid, TransactionDetails>,
-        chunk_size: usize,
-    ) -> Result<HashMap<Txid, u64>, Error> {
-        let mut txid_timestamp = HashMap::new();
-        let txid_in_db_with_conf: HashSet<_> = txs_details_in_db
-            .values()
-            .filter_map(|details| details.confirmation_time.as_ref().map(|_| details.txid))
-            .collect();
-        let needed_txid_height: HashMap<&Txid, u32> = txid_height
-            .iter()
-            .filter(|(t, _)| !txid_in_db_with_conf.contains(*t))
-            .filter_map(|(t, o)| o.map(|h| (t, h)))
-            .collect();
-        let needed_heights: HashSet<u32> = needed_txid_height.values().cloned().collect();
-        if !needed_heights.is_empty() {
-            info!("{} headers to download for timestamp", needed_heights.len());
-            let mut height_timestamp: HashMap<u32, u64> = HashMap::new();
-            for chunk in ChunksIterator::new(needed_heights.into_iter(), chunk_size) {
-                let call_result: Vec<BlockHeader> =
-                    maybe_await!(self.els_batch_block_header(chunk.clone()))?;
-                height_timestamp.extend(
-                    chunk
-                        .into_iter()
-                        .zip(call_result.iter().map(|h| h.time as u64)),
-                );
-            }
-            for (txid, height) in needed_txid_height {
-                let timestamp = height_timestamp
-                    .get(&height)
-                    .ok_or_else(|| Error::Generic("timestamp missing".to_string()))?;
-                txid_timestamp.insert(*txid, *timestamp);
-            }
-        }
-
-        Ok(txid_timestamp)
-    }
-
-    fn download_and_save_in_chunks<D: BatchDatabase>(
-        &self,
-        to_download: Vec<&Txid>,
-        chunk_size: usize,
-        db: &mut D,
-    ) -> Result<Vec<Transaction>, Error> {
-        let mut txs_downloaded = vec![];
-        for chunk in ChunksIterator::new(to_download.into_iter(), chunk_size) {
-            let call_result: Vec<Transaction> =
-                maybe_await!(self.els_batch_transaction_get(chunk))?;
-            let mut batch = db.begin_batch();
-            for new_tx in call_result.iter() {
-                batch.set_raw_tx(new_tx)?;
-            }
-            db.commit_batch(batch)?;
-            txs_downloaded.extend(call_result);
-        }
-
-        Ok(txs_downloaded)
-    }
-}
-
-fn save_transaction_details_and_utxos<D: BatchDatabase>(
-    txid: &Txid,
-    db: &mut D,
-    timestamp: Option<u64>,
-    height: Option<u32>,
-    updates: &mut dyn BatchOperations,
-    utxo_deps: &HashMap<OutPoint, OutPoint>,
-) -> Result<(), Error> {
-    let tx = db.get_raw_tx(txid)?.ok_or(Error::TransactionNotFound)?;
-
-    let mut incoming: u64 = 0;
-    let mut outgoing: u64 = 0;
-
-    let mut inputs_sum: u64 = 0;
-    let mut outputs_sum: u64 = 0;
-
-    // look for our own inputs
-    for input in tx.input.iter() {
-        // skip coinbase inputs
-        if input.previous_output.is_null() {
-            continue;
-        }
-
-        // We already downloaded all previous output txs in the previous step
-        if let Some(previous_output) = db.get_previous_output(&input.previous_output)? {
-            inputs_sum += previous_output.value;
-
-            if db.is_mine(&previous_output.script_pubkey)? {
-                outgoing += previous_output.value;
-            }
-        } else {
-            // The input is not ours, but we still need to count it for the fees
-            let tx = db
-                .get_raw_tx(&input.previous_output.txid)?
-                .ok_or(Error::TransactionNotFound)?;
-            inputs_sum += tx.output[input.previous_output.vout as usize].value;
-        }
-
-        // removes conflicting UTXO if any (generated from same inputs, like for example RBF)
-        if let Some(outpoint) = utxo_deps.get(&input.previous_output) {
-            updates.del_utxo(outpoint)?;
-        }
-    }
-
-    for (i, output) in tx.output.iter().enumerate() {
-        // to compute the fees later
-        outputs_sum += output.value;
-
-        // this output is ours, we have a path to derive it
-        if let Some((keychain, _child)) = db.get_path_from_script_pubkey(&output.script_pubkey)? {
-            debug!("{} output #{} is mine, adding utxo", txid, i);
-            updates.set_utxo(&LocalUtxo {
-                outpoint: OutPoint::new(tx.txid(), i as u32),
-                txout: output.clone(),
-                keychain,
-            })?;
-
-            incoming += output.value;
-        }
-    }
-
-    let tx_details = TransactionDetails {
-        txid: tx.txid(),
-        transaction: Some(tx),
-        received: incoming,
-        sent: outgoing,
-        confirmation_time: BlockTime::new(height, timestamp),
-        fee: Some(inputs_sum.saturating_sub(outputs_sum)), /* if the tx is a coinbase, fees would be negative */
-        verified: height.is_some(),
-    };
-    updates.set_tx(&tx_details)?;
-
-    Ok(())
-}
-
-/// returns utxo dependency as the inputs needed for the utxo to exist
-/// `tx_raw_in_db` must contains utxo's generating txs or errors with [crate::Error::TransactionNotFound]
-fn utxos_deps<D: BatchDatabase>(
-    db: &mut D,
-    tx_raw_in_db: &HashMap<Txid, Transaction>,
-) -> Result<HashMap<OutPoint, OutPoint>, Error> {
-    let utxos = db.iter_utxos()?;
-    let mut utxos_deps = HashMap::new();
-    for utxo in utxos {
-        let from_tx = tx_raw_in_db
-            .get(&utxo.outpoint.txid)
-            .ok_or(Error::TransactionNotFound)?;
-        for input in from_tx.input.iter() {
-            utxos_deps.insert(input.previous_output, utxo.outpoint);
-        }
-    }
-    Ok(utxos_deps)
-}

src/database/any.rs

@@ -23,12 +23,12 @@
 //! # use bdk::database::{AnyDatabase, MemoryDatabase};
 //! # use bdk::{Wallet};
 //! let memory = MemoryDatabase::default();
-//! let wallet_memory = Wallet::new_offline("...", None, Network::Testnet, memory)?;
+//! let wallet_memory = Wallet::new("...", None, Network::Testnet, memory)?;
 //!
 //! # #[cfg(feature = "key-value-db")]
 //! # {
 //! let sled = sled::open("my-database")?.open_tree("default_tree")?;
-//! let wallet_sled = Wallet::new_offline("...", None, Network::Testnet, sled)?;
+//! let wallet_sled = Wallet::new("...", None, Network::Testnet, sled)?;
 //! # }
 //! # Ok::<(), bdk::Error>(())
 //! ```
@@ -42,7 +42,7 @@
 //! # use bdk::{Wallet};
 //! let config = serde_json::from_str("...")?;
 //! let database = AnyDatabase::from_config(&config)?;
-//! let wallet = Wallet::new_offline("...", None, Network::Testnet, database)?;
+//! let wallet = Wallet::new("...", None, Network::Testnet, database)?;
 //! # Ok::<(), bdk::Error>(())
 //! ```
 
@@ -61,6 +61,7 @@ macro_rules! impl_from {
 
 macro_rules! impl_inner_method {
     ( $enum_name:ident, $self:expr, $name:ident $(, $args:expr)* ) => {
+        #[allow(deprecated)]
         match $self {
             $enum_name::Memory(inner) => inner.$name( $($args, )* ),
             #[cfg(feature = "key-value-db")]
@@ -254,10 +255,6 @@ impl Database for AnyDatabase {
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
         impl_inner_method!(AnyDatabase, self, increment_last_index, keychain)
     }
-
-    fn flush(&mut self) -> Result<(), Error> {
-        impl_inner_method!(AnyDatabase, self, flush)
-    }
 }
 
 impl BatchOperations for AnyBatch {

src/database/keyvalue.rs

@@ -43,6 +43,7 @@ macro_rules! impl_batch_operations {
             let value = json!({
                 "t": utxo.txout,
                 "i": utxo.keychain,
+                "s": utxo.is_spent,
             });
             self.insert(key, serde_json::to_vec(&value)?)$($after_insert)*;
 
@@ -125,8 +126,9 @@ macro_rules! impl_batch_operations {
                     let mut val: serde_json::Value = serde_json::from_slice(&b)?;
                     let txout = serde_json::from_value(val["t"].take())?;
                     let keychain = serde_json::from_value(val["i"].take())?;
+                    let is_spent = val.get_mut("s").and_then(|s| s.take().as_bool()).unwrap_or(false);
 
-                    Ok(Some(LocalUtxo { outpoint: outpoint.clone(), txout, keychain }))
+                    Ok(Some(LocalUtxo { outpoint: outpoint.clone(), txout, keychain, is_spent, }))
                 }
             }
         }
@@ -164,16 +166,9 @@ macro_rules! impl_batch_operations {
         fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
             let key = MapKey::LastIndex(keychain).as_map_key();
             let res = self.remove(key);
-            let res = $process_delete!(res);
-
-            match res {
-                None => Ok(None),
-                Some(b) => {
-                    let array: [u8; 4] = b.as_ref().try_into().map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
-                    let val = u32::from_be_bytes(array);
-                    Ok(Some(val))
-                }
-            }
+            $process_delete!(res)
+            .map(ivec_to_u32)
+            .transpose()
         }
 
         fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
@@ -246,11 +241,16 @@ impl Database for Tree {
                 let mut val: serde_json::Value = serde_json::from_slice(&v)?;
                 let txout = serde_json::from_value(val["t"].take())?;
                 let keychain = serde_json::from_value(val["i"].take())?;
+                let is_spent = val
+                    .get_mut("s")
+                    .and_then(|s| s.take().as_bool())
+                    .unwrap_or(false);
 
                 Ok(LocalUtxo {
                     outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 })
             })
             .collect()
@@ -314,11 +314,16 @@ impl Database for Tree {
                 let mut val: serde_json::Value = serde_json::from_slice(&b)?;
                 let txout = serde_json::from_value(val["t"].take())?;
                 let keychain = serde_json::from_value(val["i"].take())?;
+                let is_spent = val
+                    .get_mut("s")
+                    .and_then(|s| s.take().as_bool())
+                    .unwrap_or(false);
 
                 Ok(LocalUtxo {
                     outpoint: *outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 })
             })
             .transpose()
@@ -345,16 +350,7 @@ impl Database for Tree {
 
     fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         let key = MapKey::LastIndex(keychain).as_map_key();
-        self.get(key)?
-            .map(|b| -> Result<_, Error> {
-                let array: [u8; 4] = b
-                    .as_ref()
-                    .try_into()
-                    .map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
-                let val = u32::from_be_bytes(array);
-                Ok(val)
-            })
-            .transpose()
+        self.get(key)?.map(ivec_to_u32).transpose()
     }
 
     fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
@@ -381,19 +377,17 @@ impl Database for Tree {
 
             Some(new.to_be_bytes().to_vec())
         })?
-        .map_or(Ok(0), |b| -> Result<_, Error> {
-            let array: [u8; 4] = b
-                .as_ref()
-                .try_into()
-                .map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
-            let val = u32::from_be_bytes(array);
-            Ok(val)
-        })
+        .map_or(Ok(0), ivec_to_u32)
     }
+}
 
-    fn flush(&mut self) -> Result<(), Error> {
-        Ok(Tree::flush(self).map(|_| ())?)
-    }
+fn ivec_to_u32(b: sled::IVec) -> Result<u32, Error> {
+    let array: [u8; 4] = b
+        .as_ref()
+        .try_into()
+        .map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
+    let val = u32::from_be_bytes(array);
+    Ok(val)
 }
 
 impl BatchDatabase for Tree {

src/database/memory.rs

@@ -109,7 +109,7 @@ fn after(key: &[u8]) -> Vec<u8> {
 /// Once it's dropped its content will be lost.
 ///
 /// If you are looking for a permanent storage solution, you can try with the default key-value
-/// database called [`sled`]. See the [`database`] module documentation for more defailts.
+/// database called [`sled`]. See the [`database`] module documentation for more details.
 ///
 /// [`database`]: crate::database
 #[derive(Debug, Default)]
@@ -150,8 +150,10 @@ impl BatchOperations for MemoryDatabase {
 
     fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error> {
         let key = MapKey::Utxo(Some(&utxo.outpoint)).as_map_key();
-        self.map
-            .insert(key, Box::new((utxo.txout.clone(), utxo.keychain)));
+        self.map.insert(
+            key,
+            Box::new((utxo.txout.clone(), utxo.keychain, utxo.is_spent)),
+        );
 
         Ok(())
     }
@@ -228,11 +230,12 @@ impl BatchOperations for MemoryDatabase {
         match res {
             None => Ok(None),
             Some(b) => {
-                let (txout, keychain) = b.downcast_ref().cloned().unwrap();
+                let (txout, keychain, is_spent) = b.downcast_ref().cloned().unwrap();
                 Ok(Some(LocalUtxo {
                     outpoint: *outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 }))
             }
         }
@@ -326,11 +329,12 @@ impl Database for MemoryDatabase {
             .range::<Vec<u8>, _>((Included(&key), Excluded(&after(&key))))
             .map(|(k, v)| {
                 let outpoint = deserialize(&k[1..]).unwrap();
-                let (txout, keychain) = v.downcast_ref().cloned().unwrap();
+                let (txout, keychain, is_spent) = v.downcast_ref().cloned().unwrap();
                 Ok(LocalUtxo {
                     outpoint,
                     txout,
                     keychain,
+                    is_spent,
                 })
             })
             .collect()
@@ -389,11 +393,12 @@ impl Database for MemoryDatabase {
     fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
         let key = MapKey::Utxo(Some(outpoint)).as_map_key();
         Ok(self.map.get(&key).map(|b| {
-            let (txout, keychain) = b.downcast_ref().cloned().unwrap();
+            let (txout, keychain, is_spent) = b.downcast_ref().cloned().unwrap();
             LocalUtxo {
                 outpoint: *outpoint,
                 txout,
                 keychain,
+                is_spent,
             }
         }))
     }
@@ -444,10 +449,6 @@ impl Database for MemoryDatabase {
 
         Ok(*value)
     }
-
-    fn flush(&mut self) -> Result<(), Error> {
-        Ok(())
-    }
 }
 
 impl BatchDatabase for MemoryDatabase {
@@ -481,15 +482,23 @@ impl ConfigurableDatabase for MemoryDatabase {
 /// don't have `test` set.
 macro_rules! populate_test_db {
     ($db:expr, $tx_meta:expr, $current_height:expr$(,)?) => {{
+        $crate::populate_test_db!($db, $tx_meta, $current_height, (@coinbase false))
+    }};
+    ($db:expr, $tx_meta:expr, $current_height:expr, (@coinbase $is_coinbase:expr)$(,)?) => {{
         use std::str::FromStr;
-        use $crate::database::BatchOperations;
+        use $crate::database::SyncTime;
+        use $crate::database::{BatchOperations, Database};
         let mut db = $db;
         let tx_meta = $tx_meta;
         let current_height: Option<u32> = $current_height;
+        let mut input = vec![$crate::bitcoin::TxIn::default()];
+        if !$is_coinbase {
+            input[0].previous_output.vout = 0;
+        }
         let tx = $crate::bitcoin::Transaction {
             version: 1,
             lock_time: 0,
-            input: vec![],
+            input,
             output: tx_meta
                 .output
                 .iter()
@@ -503,10 +512,31 @@ macro_rules! populate_test_db {
         };
 
         let txid = tx.txid();
-        let confirmation_time = tx_meta.min_confirmations.map(|conf| $crate::BlockTime {
-            height: current_height.unwrap().checked_sub(conf as u32).unwrap(),
-            timestamp: 0,
-        });
+        // Set Confirmation time only if current height is provided.
+        // panics if `tx_meta.min_confirmation` is Some, and current_height is None.
+        let confirmation_time = tx_meta
+            .min_confirmations
+            .and_then(|v| if v == 0 { None } else { Some(v) })
+            .map(|conf| $crate::BlockTime {
+                height: current_height.expect("Current height is needed for testing transaction with min-confirmation values").checked_sub(conf as u32).unwrap() + 1,
+                timestamp: 0,
+            });
+
+        // Set the database sync_time.
+        // Check if the current_height is less than already known sync height, apply the max
+        // If any of them is None, the other will be applied instead.
+        // If both are None, this will not be set.
+        if let Some(height) = db.get_sync_time().unwrap()
+                                .map(|sync_time| sync_time.block_time.height)
+                                .max(current_height) {
+            let sync_time = SyncTime {
+                block_time: BlockTime {
+                    height,
+                    timestamp: 0
+                }
+            };
+            db.set_sync_time(sync_time).unwrap();
+        }
 
         let tx_details = $crate::TransactionDetails {
             transaction: Some(tx.clone()),
@@ -515,7 +545,6 @@ macro_rules! populate_test_db {
             received: 0,
             sent: 0,
             confirmation_time,
-            verified: current_height.is_some(),
         };
 
         db.set_tx(&tx_details).unwrap();
@@ -527,6 +556,7 @@ macro_rules! populate_test_db {
                     vout: vout as u32,
                 },
                 keychain: $crate::KeychainKind::External,
+                is_spent: false,
             })
             .unwrap();
         }
@@ -555,7 +585,7 @@ macro_rules! doctest_wallet {
             Some(100),
         );
 
-        $crate::Wallet::new_offline(
+        $crate::Wallet::new(
             &descriptors.0,
             descriptors.1.as_ref(),
             Network::Regtest,

src/database/mod.rs

@@ -149,7 +149,7 @@ pub trait Database: BatchOperations {
     fn get_raw_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
     /// Fetch the transaction metadata and optionally also the raw transaction
     fn get_tx(&self, txid: &Txid, include_raw: bool) -> Result<Option<TransactionDetails>, Error>;
-    /// Return the last defivation index for a keychain.
+    /// Return the last derivation index for a keychain.
     fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error>;
     /// Return the sync time, if present
     fn get_sync_time(&self) -> Result<Option<SyncTime>, Error>;
@@ -158,9 +158,6 @@ pub trait Database: BatchOperations {
     ///
     /// It should insert and return `0` if not present in the database
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error>;
-
-    /// Force changes to be written to disk
-    fn flush(&mut self) -> Result<(), Error>;
 }
 
 /// Trait for a database that supports batch operations
@@ -196,8 +193,7 @@ pub(crate) trait DatabaseUtils: Database {
         D: FnOnce() -> Result<Option<Transaction>, Error>,
     {
         self.get_tx(txid, true)?
-            .map(|t| t.transaction)
-            .flatten()
+            .and_then(|t| t.transaction)
             .map_or_else(default, |t| Ok(Some(t)))
     }
 
@@ -316,10 +312,12 @@ pub mod test {
             txout,
             outpoint,
             keychain: KeychainKind::External,
+            is_spent: true,
         };
 
         tree.set_utxo(&utxo).unwrap();
-
+        tree.set_utxo(&utxo).unwrap();
+        assert_eq!(tree.iter_utxos().unwrap().len(), 1);
         assert_eq!(tree.get_utxo(&outpoint).unwrap(), Some(utxo));
     }
 
@@ -348,7 +346,6 @@ pub mod test {
                 timestamp: 123456,
                 height: 1000,
             }),
-            verified: true,
         };
 
         tree.set_tx(&tx_details).unwrap();
@@ -372,6 +369,34 @@ pub mod test {
         );
     }
 
+    pub fn test_list_transaction<D: Database>(mut tree: D) {
+        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
+        let tx: Transaction = deserialize(&hex_tx).unwrap();
+        let txid = tx.txid();
+        let mut tx_details = TransactionDetails {
+            transaction: Some(tx),
+            txid,
+            received: 1337,
+            sent: 420420,
+            fee: Some(140),
+            confirmation_time: Some(BlockTime {
+                timestamp: 123456,
+                height: 1000,
+            }),
+        };
+
+        tree.set_tx(&tx_details).unwrap();
+
+        // get raw tx
+        assert_eq!(tree.iter_txs(true).unwrap(), vec![tx_details.clone()]);
+
+        // now get without raw tx
+        tx_details.transaction = None;
+
+        // get not raw tx
+        assert_eq!(tree.iter_txs(false).unwrap(), vec![tx_details.clone()]);
+    }
+
     pub fn test_last_index<D: Database>(mut tree: D) {
         tree.set_last_index(KeychainKind::External, 1337).unwrap();
 

src/database/sqlite.rs

@@ -8,6 +8,8 @@
 // <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
 // You may not use this file except in accordance with one or both of these
 // licenses.
+use std::path::Path;
+use std::path::PathBuf;
 
 use bitcoin::consensus::encode::{deserialize, serialize};
 use bitcoin::hash_types::Txid;
@@ -35,7 +37,22 @@ static MIGRATIONS: &[&str] = &[
     "CREATE UNIQUE INDEX idx_indices_keychain ON last_derivation_indices(keychain);",
     "CREATE TABLE checksums (keychain TEXT, checksum BLOB);",
     "CREATE INDEX idx_checksums_keychain ON checksums(keychain);",
-    "CREATE TABLE sync_time (id INTEGER PRIMARY KEY, height INTEGER, timestamp INTEGER);"
+    "CREATE TABLE sync_time (id INTEGER PRIMARY KEY, height INTEGER, timestamp INTEGER);",
+    "ALTER TABLE transaction_details RENAME TO transaction_details_old;",
+    "CREATE TABLE transaction_details (txid BLOB, timestamp INTEGER, received INTEGER, sent INTEGER, fee INTEGER, height INTEGER);",
+    "INSERT INTO transaction_details SELECT txid, timestamp, received, sent, fee, height FROM transaction_details_old;",
+    "DROP TABLE transaction_details_old;",
+    "ALTER TABLE utxos ADD COLUMN is_spent;",
+    // drop all data due to possible inconsistencies with duplicate utxos, re-sync required
+    "DELETE FROM checksums;",
+    "DELETE FROM last_derivation_indices;",
+    "DELETE FROM script_pubkeys;",
+    "DELETE FROM sync_time;",
+    "DELETE FROM transaction_details;",
+    "DELETE FROM transactions;",
+    "DELETE FROM utxos;",
+    "DROP INDEX idx_txid_vout;",
+    "CREATE UNIQUE INDEX idx_utxos_txid_vout ON utxos(txid, vout);"
 ];
 
 /// Sqlite database stored on filesystem
@@ -45,7 +62,7 @@ static MIGRATIONS: &[&str] = &[
 #[derive(Debug)]
 pub struct SqliteDatabase {
     /// Path on the local filesystem to store the sqlite file
-    pub path: String,
+    pub path: PathBuf,
     /// A rusqlite connection object to the sqlite database
     pub connection: Connection,
 }
@@ -53,9 +70,12 @@ pub struct SqliteDatabase {
 impl SqliteDatabase {
     /// Instantiate a new SqliteDatabase instance by creating a connection
     /// to the database stored at path
-    pub fn new(path: String) -> Self {
+    pub fn new<T: AsRef<Path>>(path: T) -> Self {
         let connection = get_connection(&path).unwrap();
-        SqliteDatabase { path, connection }
+        SqliteDatabase {
+            path: PathBuf::from(path.as_ref()),
+            connection,
+        }
     }
     fn insert_script_pubkey(
         &self,
@@ -79,14 +99,16 @@ impl SqliteDatabase {
         vout: u32,
         txid: &[u8],
         script: &[u8],
+        is_spent: bool,
     ) -> Result<i64, Error> {
-        let mut statement = self.connection.prepare_cached("INSERT INTO utxos (value, keychain, vout, txid, script) VALUES (:value, :keychain, :vout, :txid, :script)")?;
+        let mut statement = self.connection.prepare_cached("INSERT INTO utxos (value, keychain, vout, txid, script, is_spent) VALUES (:value, :keychain, :vout, :txid, :script, :is_spent) ON CONFLICT(txid, vout) DO UPDATE SET value=:value, keychain=:keychain, script=:script, is_spent=:is_spent")?;
         statement.execute(named_params! {
             ":value": value,
             ":keychain": keychain,
             ":vout": vout,
             ":txid": txid,
-            ":script": script
+            ":script": script,
+            ":is_spent": is_spent,
         })?;
 
         Ok(self.connection.last_insert_rowid())
@@ -127,7 +149,7 @@ impl SqliteDatabase {
 
         let txid: &[u8] = &transaction.txid;
 
-        let mut statement = self.connection.prepare_cached("INSERT INTO transaction_details (txid, timestamp, received, sent, fee, height, verified) VALUES (:txid, :timestamp, :received, :sent, :fee, :height, :verified)")?;
+        let mut statement = self.connection.prepare_cached("INSERT INTO transaction_details (txid, timestamp, received, sent, fee, height) VALUES (:txid, :timestamp, :received, :sent, :fee, :height)")?;
 
         statement.execute(named_params! {
             ":txid": txid,
@@ -136,7 +158,6 @@ impl SqliteDatabase {
             ":sent": transaction.sent,
             ":fee": transaction.fee,
             ":height": height,
-            ":verified": transaction.verified
         })?;
 
         Ok(self.connection.last_insert_rowid())
@@ -153,7 +174,7 @@ impl SqliteDatabase {
 
         let txid: &[u8] = &transaction.txid;
 
-        let mut statement = self.connection.prepare_cached("UPDATE transaction_details SET timestamp=:timestamp, received=:received, sent=:sent, fee=:fee, height=:height, verified=:verified WHERE txid=:txid")?;
+        let mut statement = self.connection.prepare_cached("UPDATE transaction_details SET timestamp=:timestamp, received=:received, sent=:sent, fee=:fee, height=:height WHERE txid=:txid")?;
 
         statement.execute(named_params! {
             ":txid": txid,
@@ -162,7 +183,6 @@ impl SqliteDatabase {
             ":sent": transaction.sent,
             ":fee": transaction.fee,
             ":height": height,
-            ":verified": transaction.verified,
         })?;
 
         Ok(())
@@ -289,7 +309,7 @@ impl SqliteDatabase {
     fn select_utxos(&self) -> Result<Vec<LocalUtxo>, Error> {
         let mut statement = self
             .connection
-            .prepare_cached("SELECT value, keychain, vout, txid, script FROM utxos")?;
+            .prepare_cached("SELECT value, keychain, vout, txid, script, is_spent FROM utxos")?;
         let mut utxos: Vec<LocalUtxo> = vec![];
         let mut rows = statement.query([])?;
         while let Some(row) = rows.next()? {
@@ -298,6 +318,7 @@ impl SqliteDatabase {
             let vout = row.get(2)?;
             let txid: Vec<u8> = row.get(3)?;
             let script: Vec<u8> = row.get(4)?;
+            let is_spent: bool = row.get(5)?;
 
             let keychain: KeychainKind = serde_json::from_str(&keychain)?;
 
@@ -308,19 +329,16 @@ impl SqliteDatabase {
                     script_pubkey: script.into(),
                 },
                 keychain,
+                is_spent,
             })
         }
 
         Ok(utxos)
     }
 
-    fn select_utxo_by_outpoint(
-        &self,
-        txid: &[u8],
-        vout: u32,
-    ) -> Result<Option<(u64, KeychainKind, Script)>, Error> {
+    fn select_utxo_by_outpoint(&self, txid: &[u8], vout: u32) -> Result<Option<LocalUtxo>, Error> {
         let mut statement = self.connection.prepare_cached(
-            "SELECT value, keychain, script FROM utxos WHERE txid=:txid AND vout=:vout",
+            "SELECT value, keychain, script, is_spent FROM utxos WHERE txid=:txid AND vout=:vout",
         )?;
         let mut rows = statement.query(named_params! {":txid": txid,":vout": vout})?;
         match rows.next()? {
@@ -329,9 +347,18 @@ impl SqliteDatabase {
                 let keychain: String = row.get(1)?;
                 let keychain: KeychainKind = serde_json::from_str(&keychain)?;
                 let script: Vec<u8> = row.get(2)?;
-                let script: Script = script.into();
+                let script_pubkey: Script = script.into();
+                let is_spent: bool = row.get(3)?;
 
-                Ok(Some((value, keychain, script)))
+                Ok(Some(LocalUtxo {
+                    outpoint: OutPoint::new(deserialize(txid)?, vout),
+                    txout: TxOut {
+                        value,
+                        script_pubkey,
+                    },
+                    keychain,
+                    is_spent,
+                }))
             }
             None => Ok(None),
         }
@@ -367,7 +394,7 @@ impl SqliteDatabase {
     }
 
     fn select_transaction_details_with_raw(&self) -> Result<Vec<TransactionDetails>, Error> {
-        let mut statement = self.connection.prepare_cached("SELECT transaction_details.txid, transaction_details.timestamp, transaction_details.received, transaction_details.sent, transaction_details.fee, transaction_details.height, transaction_details.verified, transactions.raw_tx FROM transaction_details, transactions WHERE transaction_details.txid = transactions.txid")?;
+        let mut statement = self.connection.prepare_cached("SELECT transaction_details.txid, transaction_details.timestamp, transaction_details.received, transaction_details.sent, transaction_details.fee, transaction_details.height, transactions.raw_tx FROM transaction_details, transactions WHERE transaction_details.txid = transactions.txid")?;
         let mut transaction_details: Vec<TransactionDetails> = vec![];
         let mut rows = statement.query([])?;
         while let Some(row) = rows.next()? {
@@ -378,8 +405,7 @@ impl SqliteDatabase {
             let sent: u64 = row.get(3)?;
             let fee: Option<u64> = row.get(4)?;
             let height: Option<u32> = row.get(5)?;
-            let verified: bool = row.get(6)?;
-            let raw_tx: Option<Vec<u8>> = row.get(7)?;
+            let raw_tx: Option<Vec<u8>> = row.get(6)?;
             let tx: Option<Transaction> = match raw_tx {
                 Some(raw_tx) => {
                     let tx: Transaction = deserialize(&raw_tx)?;
@@ -400,7 +426,6 @@ impl SqliteDatabase {
                 sent,
                 fee,
                 confirmation_time,
-                verified,
             });
         }
         Ok(transaction_details)
@@ -408,7 +433,7 @@ impl SqliteDatabase {
 
     fn select_transaction_details(&self) -> Result<Vec<TransactionDetails>, Error> {
         let mut statement = self.connection.prepare_cached(
-            "SELECT txid, timestamp, received, sent, fee, height, verified FROM transaction_details",
+            "SELECT txid, timestamp, received, sent, fee, height FROM transaction_details",
         )?;
         let mut transaction_details: Vec<TransactionDetails> = vec![];
         let mut rows = statement.query([])?;
@@ -420,7 +445,6 @@ impl SqliteDatabase {
             let sent: u64 = row.get(3)?;
             let fee: Option<u64> = row.get(4)?;
             let height: Option<u32> = row.get(5)?;
-            let verified: bool = row.get(6)?;
 
             let confirmation_time = match (height, timestamp) {
                 (Some(height), Some(timestamp)) => Some(BlockTime { height, timestamp }),
@@ -434,7 +458,6 @@ impl SqliteDatabase {
                 sent,
                 fee,
                 confirmation_time,
-                verified,
             });
         }
         Ok(transaction_details)
@@ -444,7 +467,7 @@ impl SqliteDatabase {
         &self,
         txid: &[u8],
     ) -> Result<Option<TransactionDetails>, Error> {
-        let mut statement = self.connection.prepare_cached("SELECT transaction_details.timestamp, transaction_details.received, transaction_details.sent, transaction_details.fee, transaction_details.height, transaction_details.verified, transactions.raw_tx FROM transaction_details, transactions WHERE transaction_details.txid=transactions.txid AND transaction_details.txid=:txid")?;
+        let mut statement = self.connection.prepare_cached("SELECT transaction_details.timestamp, transaction_details.received, transaction_details.sent, transaction_details.fee, transaction_details.height, transactions.raw_tx FROM transaction_details, transactions WHERE transaction_details.txid=transactions.txid AND transaction_details.txid=:txid")?;
         let mut rows = statement.query(named_params! { ":txid": txid })?;
 
         match rows.next()? {
@@ -454,9 +477,8 @@ impl SqliteDatabase {
                 let sent: u64 = row.get(2)?;
                 let fee: Option<u64> = row.get(3)?;
                 let height: Option<u32> = row.get(4)?;
-                let verified: bool = row.get(5)?;
 
-                let raw_tx: Option<Vec<u8>> = row.get(6)?;
+                let raw_tx: Option<Vec<u8>> = row.get(5)?;
                 let tx: Option<Transaction> = match raw_tx {
                     Some(raw_tx) => {
                         let tx: Transaction = deserialize(&raw_tx)?;
@@ -477,7 +499,6 @@ impl SqliteDatabase {
                     sent,
                     fee,
                     confirmation_time,
-                    verified,
                 }))
             }
             None => Ok(None),
@@ -624,6 +645,7 @@ impl BatchOperations for SqliteDatabase {
             utxo.outpoint.vout,
             &utxo.outpoint.txid,
             utxo.txout.script_pubkey.as_bytes(),
+            utxo.is_spent,
         )?;
         Ok(())
     }
@@ -698,16 +720,9 @@ impl BatchOperations for SqliteDatabase {
 
     fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
         match self.select_utxo_by_outpoint(&outpoint.txid, outpoint.vout)? {
-            Some((value, keychain, script_pubkey)) => {
+            Some(local_utxo) => {
                 self.delete_utxo_by_outpoint(&outpoint.txid, outpoint.vout)?;
-                Ok(Some(LocalUtxo {
-                    outpoint: *outpoint,
-                    txout: TxOut {
-                        value,
-                        script_pubkey,
-                    },
-                    keychain,
-                }))
+                Ok(Some(local_utxo))
             }
             None => Ok(None),
         }
@@ -836,17 +851,7 @@ impl Database for SqliteDatabase {
     }
 
     fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
-        match self.select_utxo_by_outpoint(&outpoint.txid, outpoint.vout)? {
-            Some((value, keychain, script_pubkey)) => Ok(Some(LocalUtxo {
-                outpoint: *outpoint,
-                txout: TxOut {
-                    value,
-                    script_pubkey,
-                },
-                keychain,
-            })),
-            None => Ok(None),
-        }
+        self.select_utxo_by_outpoint(&outpoint.txid, outpoint.vout)
     }
 
     fn get_raw_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
@@ -891,10 +896,6 @@ impl Database for SqliteDatabase {
             }
         }
     }
-
-    fn flush(&mut self) -> Result<(), Error> {
-        Ok(())
-    }
 }
 
 impl BatchDatabase for SqliteDatabase {
@@ -912,7 +913,7 @@ impl BatchDatabase for SqliteDatabase {
     }
 }
 
-pub fn get_connection(path: &str) -> Result<Connection, Error> {
+pub fn get_connection<T: AsRef<Path>>(path: &T) -> Result<Connection, Error> {
     let connection = Connection::open(path)?;
     migrate(&connection)?;
     Ok(connection)
@@ -1030,4 +1031,9 @@ pub mod test {
     fn test_sync_time() {
         crate::database::test::test_sync_time(get_database());
     }
+
+    #[test]
+    fn test_txs() {
+        crate::database::test::test_list_transaction(get_database());
+    }
 }

src/descriptor/checksum.rs

@@ -14,12 +14,10 @@
 //! This module contains a re-implementation of the function used by Bitcoin Core to calculate the
 //! checksum of a descriptor
 
-use std::iter::FromIterator;
-
 use crate::descriptor::DescriptorError;
 
-const INPUT_CHARSET: &str =  "0123456789()[],'/*abcdefgh@:$%{}IJKLMNOPQRSTUVWXYZ&+-.;<=>?!^_|~ijklmnopqrstuvwxyzABCDEFGH`#\"\\ ";
-const CHECKSUM_CHARSET: &str = "qpzry9x8gf2tvdw0s3jn54khce6mua7l";
+const INPUT_CHARSET: &[u8] = b"0123456789()[],'/*abcdefgh@:$%{}IJKLMNOPQRSTUVWXYZ&+-.;<=>?!^_|~ijklmnopqrstuvwxyzABCDEFGH`#\"\\ ";
+const CHECKSUM_CHARSET: &[u8] = b"qpzry9x8gf2tvdw0s3jn54khce6mua7l";
 
 fn poly_mod(mut c: u64, val: u64) -> u64 {
     let c0 = c >> 35;
@@ -43,15 +41,17 @@ fn poly_mod(mut c: u64, val: u64) -> u64 {
     c
 }
 
-/// Compute the checksum of a descriptor
-pub fn get_checksum(desc: &str) -> Result<String, DescriptorError> {
+/// Computes the checksum bytes of a descriptor
+pub fn get_checksum_bytes(desc: &str) -> Result<[u8; 8], DescriptorError> {
     let mut c = 1;
     let mut cls = 0;
     let mut clscount = 0;
-    for ch in desc.chars() {
+
+    for ch in desc.as_bytes() {
         let pos = INPUT_CHARSET
-            .find(ch)
-            .ok_or(DescriptorError::InvalidDescriptorCharacter(ch))? as u64;
+            .iter()
+            .position(|b| b == ch)
+            .ok_or(DescriptorError::InvalidDescriptorCharacter(*ch))? as u64;
         c = poly_mod(c, pos & 31);
         cls = cls * 3 + (pos >> 5);
         clscount += 1;
@@ -67,17 +67,18 @@ pub fn get_checksum(desc: &str) -> Result<String, DescriptorError> {
     (0..8).for_each(|_| c = poly_mod(c, 0));
     c ^= 1;
 
-    let mut chars = Vec::with_capacity(8);
+    let mut checksum = [0_u8; 8];
     for j in 0..8 {
-        chars.push(
-            CHECKSUM_CHARSET
-                .chars()
-                .nth(((c >> (5 * (7 - j))) & 31) as usize)
-                .unwrap(),
-        );
+        checksum[j] = CHECKSUM_CHARSET[((c >> (5 * (7 - j))) & 31) as usize];
     }
 
-    Ok(String::from_iter(chars))
+    Ok(checksum)
+}
+
+/// Compute the checksum of a descriptor
+pub fn get_checksum(desc: &str) -> Result<String, DescriptorError> {
+    // unsafe is okay here as the checksum only uses bytes in `CHECKSUM_CHARSET`
+    get_checksum_bytes(desc).map(|b| unsafe { String::from_utf8_unchecked(b.to_vec()) })
 }
 
 #[cfg(test)]
@@ -97,17 +98,12 @@ mod test {
 
     #[test]
     fn test_get_checksum_invalid_character() {
-        let sparkle_heart = vec![240, 159, 146, 150];
-        let sparkle_heart = std::str::from_utf8(&sparkle_heart)
-            .unwrap()
-            .chars()
-            .next()
-            .unwrap();
+        let sparkle_heart = unsafe { std::str::from_utf8_unchecked(&[240, 159, 146, 150]) };
         let invalid_desc = format!("wpkh(tprv8ZgxMBicQKsPdpkqS7Eair4YxjcuuvDPNYmKX3sCniCf16tHEVrjjiSXEkFRnUH77yXc6ZcwHHcL{}fjdi5qUvw3VDfgYiH5mNsj5izuiu2N/1/2/*)", sparkle_heart);
 
         assert!(matches!(
             get_checksum(&invalid_desc).err(),
-            Some(DescriptorError::InvalidDescriptorCharacter(invalid_char)) if invalid_char == sparkle_heart
+            Some(DescriptorError::InvalidDescriptorCharacter(invalid_char)) if invalid_char == sparkle_heart.as_bytes()[0]
         ));
     }
 }

src/descriptor/derived.rs

@@ -10,6 +10,41 @@
 // licenses.
 
 //! Derived descriptor keys
+//!
+//! The [`DerivedDescriptorKey`] type is a wrapper over the standard [`DescriptorPublicKey`] which
+//! guarantees that all the extended keys have a fixed derivation path, i.e. all the wildcards have
+//! been replaced by actual derivation indexes.
+//!
+//! The [`AsDerived`] trait provides a quick way to derive descriptors to obtain a
+//! `Descriptor<DerivedDescriptorKey>` type. This, in turn, can be used to derive public
+//! keys for arbitrary derivation indexes.
+//!
+//! Combining this with [`Wallet::get_signers`], secret keys can also be derived.
+//!
+//! # Example
+//!
+//! ```
+//! # use std::str::FromStr;
+//! # use bitcoin::secp256k1::Secp256k1;
+//! use bdk::descriptor::{AsDerived, DescriptorPublicKey};
+//! use bdk::miniscript::{ToPublicKey, TranslatePk, MiniscriptKey};
+//!
+//! let secp = Secp256k1::gen_new();
+//!
+//! let key = DescriptorPublicKey::from_str("[aa600a45/84'/0'/0']tpubDCbDXFKoLTQp44wQuC12JgSn5g9CWGjZdpBHeTqyypZ4VvgYjTJmK9CkyR5bFvG9f4PutvwmvpYCLkFx2rpx25hiMs4sUgxJveW8ZzSAVAc/0/*")?;
+//! let (descriptor, _, _) = bdk::descriptor!(wpkh(key))?;
+//!
+//! // derived: wpkh([aa600a45/84'/0'/0']tpubDCbDXFKoLTQp44wQuC12JgSn5g9CWGjZdpBHeTqyypZ4VvgYjTJmK9CkyR5bFvG9f4PutvwmvpYCLkFx2rpx25hiMs4sUgxJveW8ZzSAVAc/0/42)#3ladd0t2
+//! let derived = descriptor.as_derived(42, &secp);
+//! println!("derived: {}", derived);
+//!
+//! // with_pks: wpkh(02373ecb54c5e83bd7e0d40adf78b65efaf12fafb13571f0261fc90364eee22e1e)#p4jjgvll
+//! let with_pks = derived.translate_pk_infallible(|pk| pk.to_public_key(), |pkh| pkh.to_public_key().to_pubkeyhash());
+//! println!("with_pks: {}", with_pks);
+//! # Ok::<(), Box<dyn std::error::Error>>(())
+//! ```
+//!
+//! [`Wallet::get_signers`]: crate::wallet::Wallet::get_signers
 
 use std::cmp::Ordering;
 use std::fmt;
@@ -17,13 +52,10 @@ use std::hash::{Hash, Hasher};
 use std::ops::Deref;
 
 use bitcoin::hashes::hash160;
-use bitcoin::PublicKey;
+use bitcoin::{PublicKey, XOnlyPublicKey};
 
-pub use miniscript::{
-    descriptor::KeyMap, descriptor::Wildcard, Descriptor, DescriptorPublicKey, Legacy, Miniscript,
-    ScriptContext, Segwitv0,
-};
-use miniscript::{MiniscriptKey, ToPublicKey, TranslatePk};
+use miniscript::descriptor::{DescriptorSinglePub, SinglePubKey, Wildcard};
+use miniscript::{Descriptor, DescriptorPublicKey, MiniscriptKey, ToPublicKey, TranslatePk};
 
 use crate::wallet::utils::SecpCtx;
 
@@ -96,21 +128,44 @@ impl<'s> MiniscriptKey for DerivedDescriptorKey<'s> {
     fn is_uncompressed(&self) -> bool {
         self.0.is_uncompressed()
     }
-    fn serialized_len(&self) -> usize {
-        self.0.serialized_len()
-    }
 }
 
 impl<'s> ToPublicKey for DerivedDescriptorKey<'s> {
     fn to_public_key(&self) -> PublicKey {
         match &self.0 {
-            DescriptorPublicKey::SinglePub(ref spub) => spub.key.to_public_key(),
-            DescriptorPublicKey::XPub(ref xpub) => {
+            DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+                key: SinglePubKey::XOnly(_),
+                ..
+            }) => panic!("Found x-only public key in non-tr descriptor"),
+            DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+                key: SinglePubKey::FullKey(ref pk),
+                ..
+            }) => *pk,
+            DescriptorPublicKey::XPub(ref xpub) => PublicKey::new(
                 xpub.xkey
                     .derive_pub(self.1, &xpub.derivation_path)
                     .expect("Shouldn't fail, only normal derivations")
-                    .public_key
-            }
+                    .public_key,
+            ),
+        }
+    }
+
+    fn to_x_only_pubkey(&self) -> XOnlyPublicKey {
+        match &self.0 {
+            DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+                key: SinglePubKey::XOnly(ref pk),
+                ..
+            }) => *pk,
+            DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+                key: SinglePubKey::FullKey(ref pk),
+                ..
+            }) => XOnlyPublicKey::from(pk.inner),
+            DescriptorPublicKey::XPub(ref xpub) => XOnlyPublicKey::from(
+                xpub.xkey
+                    .derive_pub(self.1, &xpub.derivation_path)
+                    .expect("Shouldn't fail, only normal derivations")
+                    .public_key,
+            ),
         }
     }
 
@@ -119,14 +174,19 @@ impl<'s> ToPublicKey for DerivedDescriptorKey<'s> {
     }
 }
 
-pub(crate) trait AsDerived {
-    // Derive a descriptor and transform all of its keys to `DerivedDescriptorKey`
+/// Utilities to derive descriptors
+///
+/// Check out the [module level] documentation for more.
+///
+/// [module level]: crate::descriptor::derived
+pub trait AsDerived {
+    /// Derive a descriptor and transform all of its keys to `DerivedDescriptorKey`
     fn as_derived<'s>(&self, index: u32, secp: &'s SecpCtx)
         -> Descriptor<DerivedDescriptorKey<'s>>;
 
-    // Transform the keys into `DerivedDescriptorKey`.
-    //
-    // Panics if the descriptor is not "fixed", i.e. if it's derivable
+    /// Transform the keys into `DerivedDescriptorKey`.
+    ///
+    /// Panics if the descriptor is not "fixed", i.e. if it's derivable
     fn as_derived_fixed<'s>(&self, secp: &'s SecpCtx) -> Descriptor<DerivedDescriptorKey<'s>>;
 }
 

src/descriptor/dsl.rs

@@ -73,6 +73,48 @@ macro_rules! impl_top_level_pk {
     }};
 }
 
+#[doc(hidden)]
+#[macro_export]
+macro_rules! impl_top_level_tr {
+    ( $internal_key:expr, $tap_tree:expr ) => {{
+        use $crate::miniscript::descriptor::{
+            Descriptor, DescriptorPublicKey, KeyMap, TapTree, Tr,
+        };
+        use $crate::miniscript::Tap;
+
+        #[allow(unused_imports)]
+        use $crate::keys::{DescriptorKey, IntoDescriptorKey, ValidNetworks};
+
+        let secp = $crate::bitcoin::secp256k1::Secp256k1::new();
+
+        $internal_key
+            .into_descriptor_key()
+            .and_then(|key: DescriptorKey<Tap>| key.extract(&secp))
+            .map_err($crate::descriptor::DescriptorError::Key)
+            .and_then(|(pk, mut key_map, mut valid_networks)| {
+                let tap_tree = $tap_tree.map(
+                    |(tap_tree, tree_keymap, tree_networks): (
+                        TapTree<DescriptorPublicKey>,
+                        KeyMap,
+                        ValidNetworks,
+                    )| {
+                        key_map.extend(tree_keymap.into_iter());
+                        valid_networks =
+                            $crate::keys::merge_networks(&valid_networks, &tree_networks);
+
+                        tap_tree
+                    },
+                );
+
+                Ok((
+                    Descriptor::<DescriptorPublicKey>::Tr(Tr::new(pk, tap_tree)?),
+                    key_map,
+                    valid_networks,
+                ))
+            })
+    }};
+}
+
 #[doc(hidden)]
 #[macro_export]
 macro_rules! impl_leaf_opcode {
@@ -84,7 +126,7 @@ macro_rules! impl_leaf_opcode {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_minsicript()?;
+            minisc.check_miniscript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -108,7 +150,7 @@ macro_rules! impl_leaf_opcode_value {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_minsicript()?;
+            minisc.check_miniscript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -132,7 +174,7 @@ macro_rules! impl_leaf_opcode_value_two {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_minsicript()?;
+            minisc.check_miniscript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -165,7 +207,7 @@ macro_rules! impl_node_opcode_two {
                     std::sync::Arc::new(b_minisc),
                 ))?;
 
-                minisc.check_minsicript()?;
+                minisc.check_miniscript()?;
 
                 Ok((minisc, a_keymap, $crate::keys::merge_networks(&a_networks, &b_networks)))
             })
@@ -197,7 +239,7 @@ macro_rules! impl_node_opcode_three {
                     std::sync::Arc::new(c_minisc),
                 ))?;
 
-                minisc.check_minsicript()?;
+                minisc.check_miniscript()?;
 
                 Ok((minisc, a_keymap, networks))
             })
@@ -228,6 +270,62 @@ macro_rules! impl_sortedmulti {
 
 }
 
+#[doc(hidden)]
+#[macro_export]
+macro_rules! parse_tap_tree {
+    ( @merge $tree_a:expr, $tree_b:expr) => {{
+        use std::sync::Arc;
+        use $crate::miniscript::descriptor::TapTree;
+
+        $tree_a
+            .and_then(|tree_a| Ok((tree_a, $tree_b?)))
+            .and_then(|((a_tree, mut a_keymap, a_networks), (b_tree, b_keymap, b_networks))| {
+                a_keymap.extend(b_keymap.into_iter());
+                Ok((TapTree::Tree(Arc::new(a_tree), Arc::new(b_tree)), a_keymap, $crate::keys::merge_networks(&a_networks, &b_networks)))
+            })
+
+    }};
+
+    // Two sub-trees
+    ( { { $( $tree_a:tt )* }, { $( $tree_b:tt )* } } ) => {{
+        let tree_a = $crate::parse_tap_tree!( { $( $tree_a )* } );
+        let tree_b = $crate::parse_tap_tree!( { $( $tree_b )* } );
+
+        $crate::parse_tap_tree!(@merge tree_a, tree_b)
+    }};
+
+    // One leaf and a sub-tree
+    ( { $op_a:ident ( $( $minisc_a:tt )* ), { $( $tree_b:tt )* } } ) => {{
+        let tree_a = $crate::parse_tap_tree!( $op_a ( $( $minisc_a )* ) );
+        let tree_b = $crate::parse_tap_tree!( { $( $tree_b )* } );
+
+        $crate::parse_tap_tree!(@merge tree_a, tree_b)
+    }};
+    ( { { $( $tree_a:tt )* }, $op_b:ident ( $( $minisc_b:tt )* ) } ) => {{
+        let tree_a = $crate::parse_tap_tree!( { $( $tree_a )* } );
+        let tree_b = $crate::parse_tap_tree!( $op_b ( $( $minisc_b )* ) );
+
+        $crate::parse_tap_tree!(@merge tree_a, tree_b)
+    }};
+
+    // Two leaves
+    ( { $op_a:ident ( $( $minisc_a:tt )* ), $op_b:ident ( $( $minisc_b:tt )* ) } ) => {{
+        let tree_a = $crate::parse_tap_tree!( $op_a ( $( $minisc_a )* ) );
+        let tree_b = $crate::parse_tap_tree!( $op_b ( $( $minisc_b )* ) );
+
+        $crate::parse_tap_tree!(@merge tree_a, tree_b)
+    }};
+
+    // Single leaf
+    ( $op:ident ( $( $minisc:tt )* ) ) => {{
+        use std::sync::Arc;
+        use $crate::miniscript::descriptor::TapTree;
+
+        $crate::fragment!( $op ( $( $minisc )* ) )
+            .map(|(a_minisc, a_keymap, a_networks)| (TapTree::Leaf(Arc::new(a_minisc)), a_keymap, a_networks))
+    }};
+}
+
 #[doc(hidden)]
 #[macro_export]
 macro_rules! apply_modifier {
@@ -243,7 +341,7 @@ macro_rules! apply_modifier {
                     ),
                 )?;
 
-                minisc.check_minsicript()?;
+                minisc.check_miniscript()?;
 
                 Ok((minisc, keymap, networks))
             })
@@ -336,7 +434,7 @@ macro_rules! apply_modifier {
 /// syntax is more suitable for a fixed number of items known at compile time, while the other accepts a
 /// [`Vec`] of items, which makes it more suitable for writing dynamic descriptors.
 ///
-/// They both produce the descriptor: `wsh(thresh(2,pk(...),s:pk(...),sdv:older(...)))`
+/// They both produce the descriptor: `wsh(thresh(2,pk(...),s:pk(...),sndv:older(...)))`
 ///
 /// ```
 /// # use std::str::FromStr;
@@ -349,7 +447,7 @@ macro_rules! apply_modifier {
 ///
 /// let (descriptor_a, key_map_a, networks) = bdk::descriptor! {
 ///     wsh (
-///         thresh(2, pk(my_key_1), s:pk(my_key_2), s:d:v:older(my_timelock))
+///         thresh(2, pk(my_key_1), s:pk(my_key_2), s:n:d:v:older(my_timelock))
 ///     )
 /// }?;
 ///
@@ -357,7 +455,7 @@ macro_rules! apply_modifier {
 /// let b_items = vec![
 ///     bdk::fragment!(pk(my_key_1))?,
 ///     bdk::fragment!(s:pk(my_key_2))?,
-///     bdk::fragment!(s:d:v:older(my_timelock))?,
+///     bdk::fragment!(s:n:d:v:older(my_timelock))?,
 /// ];
 /// let (descriptor_b, mut key_map_b, networks) = bdk::descriptor!(wsh(thresh_vec(2, b_items)))?;
 ///
@@ -441,6 +539,15 @@ macro_rules! descriptor {
     ( wsh ( $( $minisc:tt )* ) ) => ({
         $crate::impl_top_level_sh!(Wsh, new, new_sortedmulti, Segwitv0, $( $minisc )*)
     });
+
+    ( tr ( $internal_key:expr ) ) => ({
+        $crate::impl_top_level_tr!($internal_key, None)
+    });
+    ( tr ( $internal_key:expr, $( $taptree:tt )* ) ) => ({
+        let tap_tree = $crate::parse_tap_tree!( $( $taptree )* );
+        tap_tree
+            .and_then(|tap_tree| $crate::impl_top_level_tr!($internal_key, Some(tap_tree)))
+    });
 }
 
 #[doc(hidden)]
@@ -480,6 +587,23 @@ impl<A, B, C> From<(A, (B, (C, ())))> for TupleThree<A, B, C> {
     }
 }
 
+#[doc(hidden)]
+#[macro_export]
+macro_rules! group_multi_keys {
+    ( $( $key:expr ),+ ) => {{
+        use $crate::keys::IntoDescriptorKey;
+
+        let keys = vec![
+            $(
+                $key.into_descriptor_key(),
+            )*
+        ];
+
+        keys.into_iter().collect::<Result<Vec<_>, _>>()
+            .map_err($crate::descriptor::DescriptorError::Key)
+    }};
+}
+
 #[doc(hidden)]
 #[macro_export]
 macro_rules! fragment_internal {
@@ -521,7 +645,7 @@ macro_rules! fragment_internal {
     // three operands it's (X, (X, (X, ()))), etc.
     //
     // To check that the right number of arguments has been passed we can "cast" those tuples to
-    // more convenient structures like `TupleTwo`. If the conversion succedes, the right number of
+    // more convenient structures like `TupleTwo`. If the conversion succeeds, the right number of
     // args was passed. Otherwise the compilation fails entirely.
     ( @t $op:ident ( $( $args:tt )* ) $( $tail:tt )* ) => ({
         ($crate::fragment!( $op ( $( $args )* ) ), $crate::fragment_internal!( @t $( $tail )* ))
@@ -640,21 +764,22 @@ macro_rules! fragment {
             .and_then(|items| $crate::fragment!(thresh_vec($thresh, items)))
     });
     ( multi_vec ( $thresh:expr, $keys:expr ) ) => ({
-        $crate::keys::make_multi($thresh, $keys)
-    });
-    ( multi ( $thresh:expr $(, $key:expr )+ ) ) => ({
-        use $crate::keys::IntoDescriptorKey;
         let secp = $crate::bitcoin::secp256k1::Secp256k1::new();
 
-        let keys = vec![
-            $(
-                $key.into_descriptor_key(),
-            )*
-        ];
+        $crate::keys::make_multi($thresh, $crate::miniscript::Terminal::Multi, $keys, &secp)
+    });
+    ( multi ( $thresh:expr $(, $key:expr )+ ) ) => ({
+        $crate::group_multi_keys!( $( $key ),* )
+            .and_then(|keys| $crate::fragment!( multi_vec ( $thresh, keys ) ))
+    });
+    ( multi_a_vec ( $thresh:expr, $keys:expr ) ) => ({
+        let secp = $crate::bitcoin::secp256k1::Secp256k1::new();
 
-        keys.into_iter().collect::<Result<Vec<_>, _>>()
-            .map_err($crate::descriptor::DescriptorError::Key)
-            .and_then(|keys| $crate::keys::make_multi($thresh, keys, &secp))
+        $crate::keys::make_multi($thresh, $crate::miniscript::Terminal::MultiA, $keys, &secp)
+    });
+    ( multi_a ( $thresh:expr $(, $key:expr )+ ) ) => ({
+        $crate::group_multi_keys!( $( $key ),* )
+            .and_then(|keys| $crate::fragment!( multi_a_vec ( $thresh, keys ) ))
     });
 
     // `sortedmulti()` is handled separately
@@ -714,7 +839,7 @@ mod test {
         }
     }
 
-    // - at least one of each "type" of operator; ie. one modifier, one leaf_opcode, one leaf_opcode_value, etc.
+    // - at least one of each "type" of operator; i.e. one modifier, one leaf_opcode, one leaf_opcode_value, etc.
     // - mixing up key types that implement IntoDescriptorKey in multi() or thresh()
 
     // expected script for pk and bare manually created
@@ -1048,13 +1173,15 @@ mod test {
         let private_key =
             PrivateKey::from_wif("cSQPHDBwXGjVzWRqAHm6zfvQhaTuj1f2bFH58h55ghbjtFwvmeXR").unwrap();
         let (descriptor, _, _) =
-            descriptor!(wsh(thresh(2,d:v:older(1),s:pk(private_key),s:pk(private_key)))).unwrap();
+            descriptor!(wsh(thresh(2,n:d:v:older(1),s:pk(private_key),s:pk(private_key)))).unwrap();
 
-        assert_eq!(descriptor.to_string(), "wsh(thresh(2,dv:older(1),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c)))#cfdcqs3s")
+        assert_eq!(descriptor.to_string(), "wsh(thresh(2,ndv:older(1),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c),s:pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c)))#zzk3ux8g")
     }
 
     #[test]
-    #[should_panic(expected = "Miniscript(ContextError(CompressedOnly))")]
+    #[should_panic(
+        expected = "Miniscript(ContextError(CompressedOnly(\"04b4632d08485ff1df2db55b9dafd23347d1c47a457072a1e87be26896549a87378ec38ff91d43e8c2092ebda601780485263da089465619e0358a5c1be7ac91f4\")))"
+    )]
     fn test_dsl_miniscript_checks() {
         let mut uncompressed_pk =
             PrivateKey::from_wif("L5EZftvrYaSudiozVRzTqLcHLNDoVn7H5HSfM9BAN6tMJX8oTWz6").unwrap();
@@ -1062,4 +1189,35 @@ mod test {
 
         descriptor!(wsh(v: pk(uncompressed_pk))).unwrap();
     }
+
+    #[test]
+    fn test_dsl_tr_only_key() {
+        let private_key =
+            PrivateKey::from_wif("cSQPHDBwXGjVzWRqAHm6zfvQhaTuj1f2bFH58h55ghbjtFwvmeXR").unwrap();
+        let (descriptor, _, _) = descriptor!(tr(private_key)).unwrap();
+
+        assert_eq!(
+            descriptor.to_string(),
+            "tr(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c)#heq9m95v"
+        )
+    }
+
+    #[test]
+    fn test_dsl_tr_simple_tree() {
+        let private_key =
+            PrivateKey::from_wif("cSQPHDBwXGjVzWRqAHm6zfvQhaTuj1f2bFH58h55ghbjtFwvmeXR").unwrap();
+        let (descriptor, _, _) =
+            descriptor!(tr(private_key, { pk(private_key), pk(private_key) })).unwrap();
+
+        assert_eq!(descriptor.to_string(), "tr(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c,{pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c),pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c)})#xy5fjw6d")
+    }
+
+    #[test]
+    fn test_dsl_tr_single_leaf() {
+        let private_key =
+            PrivateKey::from_wif("cSQPHDBwXGjVzWRqAHm6zfvQhaTuj1f2bFH58h55ghbjtFwvmeXR").unwrap();
+        let (descriptor, _, _) = descriptor!(tr(private_key, pk(private_key))).unwrap();
+
+        assert_eq!(descriptor.to_string(), "tr(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c,pk(02e96fe52ef0e22d2f131dd425ce1893073a3c6ad20e8cac36726393dfb4856a4c))#lzl2vmc7")
+    }
 }

src/descriptor/error.rs

@@ -28,8 +28,8 @@ pub enum Error {
     /// Error while extracting and manipulating policies
     Policy(crate::descriptor::policy::PolicyError),
 
-    /// Invalid character found in the descriptor checksum
-    InvalidDescriptorCharacter(char),
+    /// Invalid byte found in the descriptor checksum
+    InvalidDescriptorCharacter(u8),
 
     /// BIP32 error
     Bip32(bitcoin::util::bip32::Error),

src/descriptor/mod.rs

@@ -14,23 +14,25 @@
 //! This module contains generic utilities to work with descriptors, plus some re-exported types
 //! from [`miniscript`].
 
-use std::collections::{BTreeMap, HashMap, HashSet};
+use std::collections::{BTreeMap, HashSet};
 use std::ops::Deref;
 
-use bitcoin::util::bip32::{
-    ChildNumber, DerivationPath, ExtendedPrivKey, ExtendedPubKey, Fingerprint, KeySource,
-};
-use bitcoin::util::psbt;
-use bitcoin::{Network, PublicKey, Script, TxOut};
+use bitcoin::util::bip32::{ChildNumber, DerivationPath, ExtendedPubKey, Fingerprint, KeySource};
+use bitcoin::util::{psbt, taproot};
+use bitcoin::{secp256k1, PublicKey, XOnlyPublicKey};
+use bitcoin::{Network, Script, TxOut};
 
-use miniscript::descriptor::{DescriptorPublicKey, DescriptorType, DescriptorXKey, Wildcard};
-pub use miniscript::{descriptor::KeyMap, Descriptor, Legacy, Miniscript, ScriptContext, Segwitv0};
+use miniscript::descriptor::{DescriptorType, InnerXKey, SinglePubKey};
+pub use miniscript::{
+    descriptor::DescriptorXKey, descriptor::KeyMap, descriptor::Wildcard, Descriptor,
+    DescriptorPublicKey, Legacy, Miniscript, ScriptContext, Segwitv0,
+};
 use miniscript::{DescriptorTrait, ForEachKey, TranslatePk};
 
 use crate::descriptor::policy::BuildSatisfaction;
 
 pub mod checksum;
-pub(crate) mod derived;
+pub mod derived;
 #[doc(hidden)]
 pub mod dsl;
 pub mod error;
@@ -38,8 +40,7 @@ pub mod policy;
 pub mod template;
 
 pub use self::checksum::get_checksum;
-use self::derived::AsDerived;
-pub use self::derived::DerivedDescriptorKey;
+pub use self::derived::{AsDerived, DerivedDescriptorKey};
 pub use self::error::Error as DescriptorError;
 pub use self::policy::Policy;
 use self::template::DescriptorTemplateOut;
@@ -58,7 +59,14 @@ pub type DerivedDescriptor<'s> = Descriptor<DerivedDescriptorKey<'s>>;
 ///
 /// [`psbt::Input`]: bitcoin::util::psbt::Input
 /// [`psbt::Output`]: bitcoin::util::psbt::Output
-pub type HdKeyPaths = BTreeMap<PublicKey, KeySource>;
+pub type HdKeyPaths = BTreeMap<secp256k1::PublicKey, KeySource>;
+
+/// Alias for the type of maps that represent taproot key origins in a [`psbt::Input`] or
+/// [`psbt::Output`]
+///
+/// [`psbt::Input`]: bitcoin::util::psbt::Input
+/// [`psbt::Output`]: bitcoin::util::psbt::Output
+pub type TapKeyOrigins = BTreeMap<bitcoin::XOnlyPublicKey, (Vec<taproot::TapLeafHash>, KeySource)>;
 
 /// Trait for types which can be converted into an [`ExtendedDescriptor`] and a [`KeyMap`] usable by a wallet in a specific [`Network`]
 pub trait IntoWalletDescriptor {
@@ -126,13 +134,13 @@ impl IntoWalletDescriptor for (ExtendedDescriptor, KeyMap) {
 
         let check_key = |pk: &DescriptorPublicKey| {
             let (pk, _, networks) = if self.0.is_witness() {
-                let desciptor_key: DescriptorKey<miniscript::Segwitv0> =
+                let descriptor_key: DescriptorKey<miniscript::Segwitv0> =
                     pk.clone().into_descriptor_key()?;
-                desciptor_key.extract(secp)?
+                descriptor_key.extract(secp)?
             } else {
-                let desciptor_key: DescriptorKey<miniscript::Legacy> =
+                let descriptor_key: DescriptorKey<miniscript::Legacy> =
                     pk.clone().into_descriptor_key()?;
-                desciptor_key.extract(secp)?
+                descriptor_key.extract(secp)?
             };
 
             if networks.contains(&network) {
@@ -238,13 +246,13 @@ pub(crate) fn into_wallet_descriptor_checked<T: IntoWalletDescriptor>(
 #[doc(hidden)]
 /// Used internally mainly by the `descriptor!()` and `fragment!()` macros
 pub trait CheckMiniscript<Ctx: miniscript::ScriptContext> {
-    fn check_minsicript(&self) -> Result<(), miniscript::Error>;
+    fn check_miniscript(&self) -> Result<(), miniscript::Error>;
 }
 
 impl<Ctx: miniscript::ScriptContext, Pk: miniscript::MiniscriptKey> CheckMiniscript<Ctx>
     for miniscript::Miniscript<Pk, Ctx>
 {
-    fn check_minsicript(&self) -> Result<(), miniscript::Error> {
+    fn check_miniscript(&self) -> Result<(), miniscript::Error> {
         Ctx::check_global_validity(self)?;
 
         Ok(())
@@ -267,41 +275,10 @@ pub(crate) trait XKeyUtils {
     fn root_fingerprint(&self, secp: &SecpCtx) -> Fingerprint;
 }
 
-// FIXME: `InnerXKey` was made private in rust-miniscript, so we have to implement this manually on
-// both `ExtendedPubKey` and `ExtendedPrivKey`.
-//
-// Revert back to using the trait once https://github.com/rust-bitcoin/rust-miniscript/pull/230 is
-// released
-impl XKeyUtils for DescriptorXKey<ExtendedPubKey> {
-    fn full_path(&self, append: &[ChildNumber]) -> DerivationPath {
-        let full_path = match self.origin {
-            Some((_, ref path)) => path
-                .into_iter()
-                .chain(self.derivation_path.into_iter())
-                .cloned()
-                .collect(),
-            None => self.derivation_path.clone(),
-        };
-
-        if self.wildcard != Wildcard::None {
-            full_path
-                .into_iter()
-                .chain(append.iter())
-                .cloned()
-                .collect()
-        } else {
-            full_path
-        }
-    }
-
-    fn root_fingerprint(&self, _: &SecpCtx) -> Fingerprint {
-        match self.origin {
-            Some((fingerprint, _)) => fingerprint,
-            None => self.xkey.fingerprint(),
-        }
-    }
-}
-impl XKeyUtils for DescriptorXKey<ExtendedPrivKey> {
+impl<T> XKeyUtils for DescriptorXKey<T>
+where
+    T: InnerXKey,
+{
     fn full_path(&self, append: &[ChildNumber]) -> DerivationPath {
         let full_path = match self.origin {
             Some((_, ref path)) => path
@@ -326,23 +303,35 @@ impl XKeyUtils for DescriptorXKey<ExtendedPrivKey> {
     fn root_fingerprint(&self, secp: &SecpCtx) -> Fingerprint {
         match self.origin {
             Some((fingerprint, _)) => fingerprint,
-            None => self.xkey.fingerprint(secp),
+            None => self.xkey.xkey_fingerprint(secp),
         }
     }
 }
 
 pub(crate) trait DerivedDescriptorMeta {
-    fn get_hd_keypaths(&self, secp: &SecpCtx) -> Result<HdKeyPaths, DescriptorError>;
+    fn get_hd_keypaths(&self, secp: &SecpCtx) -> HdKeyPaths;
+    fn get_tap_key_origins(&self, secp: &SecpCtx) -> TapKeyOrigins;
 }
 
 pub(crate) trait DescriptorMeta {
     fn is_witness(&self) -> bool;
+    fn is_taproot(&self) -> bool;
     fn get_extended_keys(&self) -> Result<Vec<DescriptorXKey<ExtendedPubKey>>, DescriptorError>;
     fn derive_from_hd_keypaths<'s>(
         &self,
         hd_keypaths: &HdKeyPaths,
         secp: &'s SecpCtx,
     ) -> Option<DerivedDescriptor<'s>>;
+    fn derive_from_tap_key_origins<'s>(
+        &self,
+        tap_key_origins: &TapKeyOrigins,
+        secp: &'s SecpCtx,
+    ) -> Option<DerivedDescriptor<'s>>;
+    fn derive_from_psbt_key_origins<'s>(
+        &self,
+        key_origins: BTreeMap<Fingerprint, (&DerivationPath, SinglePubKey)>,
+        secp: &'s SecpCtx,
+    ) -> Option<DerivedDescriptor<'s>>;
     fn derive_from_psbt_input<'s>(
         &self,
         psbt_input: &psbt::Input,
@@ -359,23 +348,34 @@ pub(crate) trait DescriptorScripts {
 impl<'s> DescriptorScripts for DerivedDescriptor<'s> {
     fn psbt_redeem_script(&self) -> Option<Script> {
         match self.desc_type() {
-            DescriptorType::ShWpkh => Some(self.explicit_script()),
-            DescriptorType::ShWsh => Some(self.explicit_script().to_v0_p2wsh()),
-            DescriptorType::Sh => Some(self.explicit_script()),
-            DescriptorType::Bare => Some(self.explicit_script()),
-            DescriptorType::ShSortedMulti => Some(self.explicit_script()),
-            _ => None,
+            DescriptorType::ShWpkh => Some(self.explicit_script().unwrap()),
+            DescriptorType::ShWsh => Some(self.explicit_script().unwrap().to_v0_p2wsh()),
+            DescriptorType::Sh => Some(self.explicit_script().unwrap()),
+            DescriptorType::Bare => Some(self.explicit_script().unwrap()),
+            DescriptorType::ShSortedMulti => Some(self.explicit_script().unwrap()),
+            DescriptorType::ShWshSortedMulti => Some(self.explicit_script().unwrap().to_v0_p2wsh()),
+            DescriptorType::Pkh
+            | DescriptorType::Wpkh
+            | DescriptorType::Tr
+            | DescriptorType::Wsh
+            | DescriptorType::WshSortedMulti => None,
         }
     }
 
     fn psbt_witness_script(&self) -> Option<Script> {
         match self.desc_type() {
-            DescriptorType::Wsh => Some(self.explicit_script()),
-            DescriptorType::ShWsh => Some(self.explicit_script()),
+            DescriptorType::Wsh => Some(self.explicit_script().unwrap()),
+            DescriptorType::ShWsh => Some(self.explicit_script().unwrap()),
             DescriptorType::WshSortedMulti | DescriptorType::ShWshSortedMulti => {
-                Some(self.explicit_script())
+                Some(self.explicit_script().unwrap())
             }
-            _ => None,
+            DescriptorType::Bare
+            | DescriptorType::Sh
+            | DescriptorType::Pkh
+            | DescriptorType::Wpkh
+            | DescriptorType::ShSortedMulti
+            | DescriptorType::Tr
+            | DescriptorType::ShWpkh => None,
         }
     }
 }
@@ -393,6 +393,10 @@ impl DescriptorMeta for ExtendedDescriptor {
         )
     }
 
+    fn is_taproot(&self) -> bool {
+        self.desc_type() == DescriptorType::Tr
+    }
+
     fn get_extended_keys(&self) -> Result<Vec<DescriptorXKey<ExtendedPubKey>>, DescriptorError> {
         let mut answer = Vec::new();
 
@@ -407,61 +411,124 @@ impl DescriptorMeta for ExtendedDescriptor {
         Ok(answer)
     }
 
-    fn derive_from_hd_keypaths<'s>(
+    fn derive_from_psbt_key_origins<'s>(
         &self,
-        hd_keypaths: &HdKeyPaths,
+        key_origins: BTreeMap<Fingerprint, (&DerivationPath, SinglePubKey)>,
         secp: &'s SecpCtx,
     ) -> Option<DerivedDescriptor<'s>> {
-        let index: HashMap<_, _> = hd_keypaths.values().map(|(a, b)| (a, b)).collect();
+        // Ensure that deriving `xpub` with `path` yields `expected`
+        let verify_key = |xpub: &DescriptorXKey<ExtendedPubKey>,
+                          path: &DerivationPath,
+                          expected: &SinglePubKey| {
+            let derived = xpub
+                .xkey
+                .derive_pub(secp, path)
+                .expect("The path should never contain hardened derivation steps")
+                .public_key;
+
+            match expected {
+                SinglePubKey::FullKey(pk) if &PublicKey::new(derived) == pk => true,
+                SinglePubKey::XOnly(pk) if &XOnlyPublicKey::from(derived) == pk => true,
+                _ => false,
+            }
+        };
 
         let mut path_found = None;
-        self.for_each_key(|key| {
-            if path_found.is_some() {
-                // already found a matching path, we are done
-                return true;
-            }
 
+        // using `for_any_key` should make this stop as soon as we return `true`
+        self.for_any_key(|key| {
             if let DescriptorPublicKey::XPub(xpub) = key.as_key().deref() {
-                // Check if the key matches one entry in our `index`. If it does, `matches()` will
+                // Check if the key matches one entry in our `key_origins`. If it does, `matches()` will
                 // return the "prefix" that matched, so we remove that prefix from the full path
-                // found in `index` and save it in `derive_path`. We expect this to be a derivation
+                // found in `key_origins` and save it in `derive_path`. We expect this to be a derivation
                 // path of length 1 if the key is `wildcard` and an empty path otherwise.
                 let root_fingerprint = xpub.root_fingerprint(secp);
-                let derivation_path: Option<Vec<ChildNumber>> = index
+                let derive_path = key_origins
                     .get_key_value(&root_fingerprint)
-                    .and_then(|(fingerprint, path)| {
-                        xpub.matches(&(**fingerprint, (*path).clone()), secp)
+                    .and_then(|(fingerprint, (path, expected))| {
+                        xpub.matches(&(*fingerprint, (*path).clone()), secp)
+                            .zip(Some((path, expected)))
                     })
-                    .map(|prefix| {
-                        index
-                            .get(&xpub.root_fingerprint(secp))
-                            .unwrap()
+                    .and_then(|(prefix, (full_path, expected))| {
+                        let derive_path = full_path
                             .into_iter()
                             .skip(prefix.into_iter().count())
                             .cloned()
-                            .collect()
+                            .collect::<DerivationPath>();
+
+                        // `derive_path` only contains the replacement index for the wildcard, if present, or
+                        // an empty path for fixed descriptors. To verify the key we also need the normal steps
+                        // that come before the wildcard, so we take them directly from `xpub` and then append
+                        // the final index
+                        if verify_key(
+                            xpub,
+                            &xpub.derivation_path.extend(derive_path.clone()),
+                            expected,
+                        ) {
+                            Some(derive_path)
+                        } else {
+                            log::debug!(
+                                "Key `{}` derived with {} yields an unexpected key",
+                                root_fingerprint,
+                                derive_path
+                            );
+                            None
+                        }
                     });
 
-                match derivation_path {
+                match derive_path {
                     Some(path) if xpub.wildcard != Wildcard::None && path.len() == 1 => {
                         // Ignore hardened wildcards
                         if let ChildNumber::Normal { index } = path[0] {
-                            path_found = Some(index)
+                            path_found = Some(index);
+                            return true;
                         }
                     }
                     Some(path) if xpub.wildcard == Wildcard::None && path.is_empty() => {
-                        path_found = Some(0)
+                        path_found = Some(0);
+                        return true;
                     }
                     _ => {}
                 }
             }
 
-            true
+            false
         });
 
         path_found.map(|path| self.as_derived(path, secp))
     }
 
+    fn derive_from_hd_keypaths<'s>(
+        &self,
+        hd_keypaths: &HdKeyPaths,
+        secp: &'s SecpCtx,
+    ) -> Option<DerivedDescriptor<'s>> {
+        // "Convert" an hd_keypaths map to the format required by `derive_from_psbt_key_origins`
+        let key_origins = hd_keypaths
+            .iter()
+            .map(|(pk, (fingerprint, path))| {
+                (
+                    *fingerprint,
+                    (path, SinglePubKey::FullKey(PublicKey::new(*pk))),
+                )
+            })
+            .collect();
+        self.derive_from_psbt_key_origins(key_origins, secp)
+    }
+
+    fn derive_from_tap_key_origins<'s>(
+        &self,
+        tap_key_origins: &TapKeyOrigins,
+        secp: &'s SecpCtx,
+    ) -> Option<DerivedDescriptor<'s>> {
+        // "Convert" a tap_key_origins map to the format required by `derive_from_psbt_key_origins`
+        let key_origins = tap_key_origins
+            .iter()
+            .map(|(pk, (_, (fingerprint, path)))| (*fingerprint, (path, SinglePubKey::XOnly(*pk))))
+            .collect();
+        self.derive_from_psbt_key_origins(key_origins, secp)
+    }
+
     fn derive_from_psbt_input<'s>(
         &self,
         psbt_input: &psbt::Input,
@@ -471,6 +538,9 @@ impl DescriptorMeta for ExtendedDescriptor {
         if let Some(derived) = self.derive_from_hd_keypaths(&psbt_input.bip32_derivation, secp) {
             return Some(derived);
         }
+        if let Some(derived) = self.derive_from_tap_key_origins(&psbt_input.tap_key_origins, secp) {
+            return Some(derived);
+        }
         if self.is_deriveable() {
             // We can't try to bruteforce the derivation index, exit here
             return None;
@@ -479,7 +549,10 @@ impl DescriptorMeta for ExtendedDescriptor {
         let descriptor = self.as_derived_fixed(secp);
         match descriptor.desc_type() {
             // TODO: add pk() here
-            DescriptorType::Pkh | DescriptorType::Wpkh | DescriptorType::ShWpkh
+            DescriptorType::Pkh
+            | DescriptorType::Wpkh
+            | DescriptorType::ShWpkh
+            | DescriptorType::Tr
                 if utxo.is_some()
                     && descriptor.script_pubkey() == utxo.as_ref().unwrap().script_pubkey =>
             {
@@ -487,7 +560,7 @@ impl DescriptorMeta for ExtendedDescriptor {
             }
             DescriptorType::Bare | DescriptorType::Sh | DescriptorType::ShSortedMulti
                 if psbt_input.redeem_script.is_some()
-                    && &descriptor.explicit_script()
+                    && &descriptor.explicit_script().unwrap()
                         == psbt_input.redeem_script.as_ref().unwrap() =>
             {
                 Some(descriptor)
@@ -497,7 +570,7 @@ impl DescriptorMeta for ExtendedDescriptor {
             | DescriptorType::ShWshSortedMulti
             | DescriptorType::WshSortedMulti
                 if psbt_input.witness_script.is_some()
-                    && &descriptor.explicit_script()
+                    && &descriptor.explicit_script().unwrap()
                         == psbt_input.witness_script.as_ref().unwrap() =>
             {
                 Some(descriptor)
@@ -508,7 +581,7 @@ impl DescriptorMeta for ExtendedDescriptor {
 }
 
 impl<'s> DerivedDescriptorMeta for DerivedDescriptor<'s> {
-    fn get_hd_keypaths(&self, secp: &SecpCtx) -> Result<HdKeyPaths, DescriptorError> {
+    fn get_hd_keypaths(&self, secp: &SecpCtx) -> HdKeyPaths {
         let mut answer = BTreeMap::new();
         self.for_each_key(|key| {
             if let DescriptorPublicKey::XPub(xpub) = key.as_key().deref() {
@@ -526,7 +599,64 @@ impl<'s> DerivedDescriptorMeta for DerivedDescriptor<'s> {
             true
         });
 
-        Ok(answer)
+        answer
+    }
+
+    fn get_tap_key_origins(&self, secp: &SecpCtx) -> TapKeyOrigins {
+        use miniscript::ToPublicKey;
+
+        let mut answer = BTreeMap::new();
+        let mut insert_path = |pk: &DerivedDescriptorKey<'_>, lh| {
+            let key_origin = match pk.deref() {
+                DescriptorPublicKey::XPub(xpub) => {
+                    Some((xpub.root_fingerprint(secp), xpub.full_path(&[])))
+                }
+                DescriptorPublicKey::SinglePub(_) => None,
+            };
+
+            // If this is the internal key, we only insert the key origin if it's not None.
+            // For keys found in the tap tree we always insert a key origin (because the signer
+            // looks for it to know which leaves to sign for), even though it may be None
+            match (lh, key_origin) {
+                (None, Some(ko)) => {
+                    answer
+                        .entry(pk.to_x_only_pubkey())
+                        .or_insert_with(|| (vec![], ko));
+                }
+                (Some(lh), origin) => {
+                    answer
+                        .entry(pk.to_x_only_pubkey())
+                        .or_insert_with(|| (vec![], origin.unwrap_or_default()))
+                        .0
+                        .push(lh);
+                }
+                _ => {}
+            }
+        };
+
+        if let Descriptor::Tr(tr) = &self {
+            // Internal key first, then iterate the scripts
+            insert_path(tr.internal_key(), None);
+
+            for (_, ms) in tr.iter_scripts() {
+                // Assume always the same leaf version
+                let leaf_hash = taproot::TapLeafHash::from_script(
+                    &ms.encode(),
+                    taproot::LeafVersion::TapScript,
+                );
+
+                for key in ms.iter_pk_pkh() {
+                    let key = match key {
+                        miniscript::miniscript::iter::PkPkh::PlainPubkey(pk) => pk,
+                        miniscript::miniscript::iter::PkPkh::HashedPubkey(pk) => pk,
+                    };
+
+                    insert_path(&key, Some(leaf_hash));
+                }
+            }
+        }
+
+        answer
     }
 }
 
@@ -538,6 +668,7 @@ mod test {
     use bitcoin::hashes::hex::FromHex;
     use bitcoin::secp256k1::Secp256k1;
     use bitcoin::util::{bip32, psbt};
+    use bitcoin::Script;
 
     use super::*;
     use crate::psbt::PsbtUtils;
@@ -667,7 +798,7 @@ mod test {
 
         // make a descriptor out of it
         let desc = crate::descriptor!(wpkh(key)).unwrap();
-        // this should conver the key that supports "any_network" to the right network (testnet)
+        // this should convert the key that supports "any_network" to the right network (testnet)
         let (wallet_desc, _) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
@@ -801,4 +932,22 @@ mod test {
             DescriptorError::DuplicatedKeys
         ));
     }
+
+    #[test]
+    fn test_sh_wsh_sortedmulti_redeemscript() {
+        use super::{AsDerived, DescriptorScripts};
+
+        let secp = Secp256k1::new();
+
+        let descriptor = "sh(wsh(sortedmulti(3,tpubDEsqS36T4DVsKJd9UH8pAKzrkGBYPLEt9jZMwpKtzh1G6mgYehfHt9WCgk7MJG5QGSFWf176KaBNoXbcuFcuadAFKxDpUdMDKGBha7bY3QM/0/*,tpubDF3cpwfs7fMvXXuoQbohXtLjNM6ehwYT287LWtmLsd4r77YLg6MZg4vTETx5MSJ2zkfigbYWu31VA2Z2Vc1cZugCYXgS7FQu6pE8V6TriEH/0/*,tpubDE1SKfcW76Tb2AASv5bQWMuScYNAdoqLHoexw13sNDXwmUhQDBbCD3QAedKGLhxMrWQdMDKENzYtnXPDRvexQPNuDrLj52wAjHhNEm8sJ4p/0/*,tpubDFLc6oXwJmhm3FGGzXkfJNTh2KitoY3WhmmQvuAjMhD8YbyWn5mAqckbxXfm2etM3p5J6JoTpSrMqRSTfMLtNW46poDaEZJ1kjd3csRSjwH/0/*,tpubDEWD9NBeWP59xXmdqSNt4VYdtTGwbpyP8WS962BuqpQeMZmX9Pur14dhXdZT5a7wR1pK6dPtZ9fP5WR493hPzemnBvkfLLYxnUjAKj1JCQV/0/*,tpubDEHyZkkwd7gZWCTgQuYQ9C4myF2hMEmyHsBCCmLssGqoqUxeT3gzohF5uEVURkf9TtmeepJgkSUmteac38FwZqirjApzNX59XSHLcwaTZCH/0/*,tpubDEqLouCekwnMUWN486kxGzD44qVgeyuqHyxUypNEiQt5RnUZNJe386TKPK99fqRV1vRkZjYAjtXGTECz98MCsdLcnkM67U6KdYRzVubeCgZ/0/*)))";
+        let (descriptor, _) =
+            into_wallet_descriptor_checked(descriptor, &secp, Network::Testnet).unwrap();
+
+        let descriptor = descriptor.as_derived(0, &secp);
+
+        let script = Script::from_str("5321022f533b667e2ea3b36e21961c9fe9dca340fbe0af5210173a83ae0337ab20a57621026bb53a98e810bd0ee61a0ed1164ba6c024786d76554e793e202dc6ce9c78c4ea2102d5b8a7d66a41ffdb6f4c53d61994022e886b4f45001fb158b95c9164d45f8ca3210324b75eead2c1f9c60e8adeb5e7009fec7a29afcdb30d829d82d09562fe8bae8521032d34f8932200833487bd294aa219dcbe000b9f9b3d824799541430009f0fa55121037468f8ea99b6c64788398b5ad25480cad08f4b0d65be54ce3a55fd206b5ae4722103f72d3d96663b0ea99b0aeb0d7f273cab11a8de37885f1dddc8d9112adb87169357ae").unwrap();
+
+        assert_eq!(descriptor.psbt_redeem_script(), Some(script.to_v0_p2wsh()));
+        assert_eq!(descriptor.psbt_witness_script(), Some(script));
+    }
 }

src/descriptor/policy.rs

@@ -21,6 +21,7 @@
 //! ```
 //! # use std::sync::Arc;
 //! # use bdk::descriptor::*;
+//! # use bdk::wallet::signer::*;
 //! # use bdk::bitcoin::secp256k1::Secp256k1;
 //! use bdk::descriptor::policy::BuildSatisfaction;
 //! let secp = Secp256k1::new();
@@ -29,7 +30,7 @@
 //! let (extended_desc, key_map) = ExtendedDescriptor::parse_descriptor(&secp, desc)?;
 //! println!("{:?}", extended_desc);
 //!
-//! let signers = Arc::new(key_map.into());
+//! let signers = Arc::new(SignersContainer::build(key_map, &extended_desc, &secp));
 //! let policy = extended_desc.extract_policy(&signers, BuildSatisfaction::None, &secp)?;
 //! println!("policy: {}", serde_json::to_string(&policy)?);
 //! # Ok::<(), bdk::Error>(())
@@ -44,59 +45,64 @@ use serde::{Serialize, Serializer};
 
 use bitcoin::hashes::*;
 use bitcoin::util::bip32::Fingerprint;
-use bitcoin::PublicKey;
+use bitcoin::{PublicKey, XOnlyPublicKey};
 
-use miniscript::descriptor::{DescriptorPublicKey, ShInner, SortedMultiVec, WshInner};
+use miniscript::descriptor::{
+    DescriptorPublicKey, DescriptorSinglePub, ShInner, SinglePubKey, SortedMultiVec, WshInner,
+};
 use miniscript::{Descriptor, Miniscript, MiniscriptKey, Satisfier, ScriptContext, Terminal};
 
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
 use crate::descriptor::ExtractPolicy;
+use crate::keys::ExtScriptContext;
 use crate::wallet::signer::{SignerId, SignersContainer};
 use crate::wallet::utils::{self, After, Older, SecpCtx};
 
 use super::checksum::get_checksum;
 use super::error::Error;
 use super::XKeyUtils;
-use bitcoin::util::psbt::PartiallySignedTransaction as Psbt;
+use bitcoin::util::psbt::{Input as PsbtInput, PartiallySignedTransaction as Psbt};
 use miniscript::psbt::PsbtInputSatisfier;
 
-/// Raw public key or extended key fingerprint
-#[derive(Debug, Clone, Default, Serialize)]
-pub struct PkOrF {
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pubkey: Option<PublicKey>,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pubkey_hash: Option<hash160::Hash>,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    fingerprint: Option<Fingerprint>,
+/// A unique identifier for a key
+#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize)]
+#[serde(rename_all = "snake_case")]
+pub enum PkOrF {
+    /// A legacy public key
+    Pubkey(PublicKey),
+    /// A x-only public key
+    XOnlyPubkey(XOnlyPublicKey),
+    /// An extended key fingerprint
+    Fingerprint(Fingerprint),
 }
 
 impl PkOrF {
     fn from_key(k: &DescriptorPublicKey, secp: &SecpCtx) -> Self {
         match k {
-            DescriptorPublicKey::SinglePub(pubkey) => PkOrF {
-                pubkey: Some(pubkey.key),
-                ..Default::default()
-            },
-            DescriptorPublicKey::XPub(xpub) => PkOrF {
-                fingerprint: Some(xpub.root_fingerprint(secp)),
-                ..Default::default()
-            },
+            DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+                key: SinglePubKey::FullKey(pk),
+                ..
+            }) => PkOrF::Pubkey(*pk),
+            DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+                key: SinglePubKey::XOnly(pk),
+                ..
+            }) => PkOrF::XOnlyPubkey(*pk),
+            DescriptorPublicKey::XPub(xpub) => PkOrF::Fingerprint(xpub.root_fingerprint(secp)),
         }
     }
 }
 
 /// An item that needs to be satisfied
-#[derive(Debug, Clone, Serialize)]
+#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
 #[serde(tag = "type", rename_all = "UPPERCASE")]
 pub enum SatisfiableItem {
     // Leaves
-    /// Signature for a raw public key
-    Signature(PkOrF),
-    /// Signature for an extended key fingerprint
-    SignatureKey(PkOrF),
+    /// ECDSA Signature for a raw public key
+    EcdsaSignature(PkOrF),
+    /// Schnorr Signature for a raw public key
+    SchnorrSignature(PkOrF),
     /// SHA256 preimage hash
     Sha256Preimage {
         /// The digest value
@@ -243,7 +249,7 @@ where
 }
 
 /// Represent if and how much a policy item is satisfied by the wallet's descriptor
-#[derive(Debug, Clone, Serialize)]
+#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
 #[serde(tag = "type", rename_all = "UPPERCASE")]
 pub enum Satisfaction {
     /// Only a partial satisfaction of some kind of threshold policy
@@ -354,10 +360,11 @@ impl Satisfaction {
                 indexes
                     .into_iter()
                     // .inspect(|x| println!("--- orig --- {:?}", x))
-                    // we map each of the combinations of elements into a tuple of ([choosen items], [conditions]). unfortunately, those items have potentially more than one
+                    // we map each of the combinations of elements into a tuple of ([chosen items], [conditions]). unfortunately, those items have potentially more than one
                     // condition (think about ORs), so we also use `mix` to expand those, i.e. [[0], [1, 2]] becomes [[0, 1], [0, 2]]. This is necessary to make sure that we
-                    // consider every possibile options and check whether or not they are compatible.
-                    .map(|i_vec| {
+                    // consider every possible options and check whether or not they are compatible.
+                    // since this step can turn one item of the iterator into multiple ones, we use `flat_map()` to expand them out
+                    .flat_map(|i_vec| {
                         mix(i_vec
                             .iter()
                             .map(|i| {
@@ -371,9 +378,6 @@ impl Satisfaction {
                         .map(|x| (i_vec.clone(), x))
                         .collect::<Vec<(Vec<usize>, Vec<Condition>)>>()
                     })
-                    // .inspect(|x: &Vec<(Vec<usize>, Vec<Condition>)>| println!("fetch {:?}", x))
-                    // since the previous step can turn one item of the iterator into multiple ones, we call flatten to expand them out
-                    .flatten()
                     // .inspect(|x| println!("flat {:?}", x))
                     // try to fold all the conditions for this specific combination of indexes/options. if they are not compatible, try_fold will be Err
                     .map(|(key, val)| {
@@ -419,7 +423,7 @@ impl From<bool> for Satisfaction {
 }
 
 /// Descriptor spending policy
-#[derive(Debug, Clone, Serialize)]
+#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
 pub struct Policy {
     /// Identifier for this policy node
     pub id: String,
@@ -567,7 +571,7 @@ impl Policy {
         Ok(Some(policy))
     }
 
-    fn make_multisig(
+    fn make_multisig<Ctx: ScriptContext + 'static>(
         keys: &[DescriptorPublicKey],
         signers: &SignersContainer,
         build_sat: BuildSatisfaction,
@@ -601,7 +605,7 @@ impl Policy {
             }
 
             if let Some(psbt) = build_sat.psbt() {
-                if signature_in_psbt(psbt, key, secp) {
+                if Ctx::find_signature(psbt, key, secp) {
                     satisfaction.add(
                         &Satisfaction::Complete {
                             condition: Default::default(),
@@ -717,18 +721,27 @@ impl From<SatisfiableItem> for Policy {
 
 fn signer_id(key: &DescriptorPublicKey, secp: &SecpCtx) -> SignerId {
     match key {
-        DescriptorPublicKey::SinglePub(pubkey) => pubkey.key.to_pubkeyhash().into(),
+        DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+            key: SinglePubKey::FullKey(pk),
+            ..
+        }) => pk.to_pubkeyhash().into(),
+        DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+            key: SinglePubKey::XOnly(pk),
+            ..
+        }) => pk.to_pubkeyhash().into(),
         DescriptorPublicKey::XPub(xpub) => xpub.root_fingerprint(secp).into(),
     }
 }
 
-fn signature(
+fn make_generic_signature<M: Fn() -> SatisfiableItem, F: Fn(&Psbt) -> bool>(
     key: &DescriptorPublicKey,
     signers: &SignersContainer,
     build_sat: BuildSatisfaction,
     secp: &SecpCtx,
+    make_policy: M,
+    find_sig: F,
 ) -> Policy {
-    let mut policy: Policy = SatisfiableItem::Signature(PkOrF::from_key(key, secp)).into();
+    let mut policy: Policy = make_policy().into();
 
     policy.contribution = if signers.find(signer_id(key, secp)).is_some() {
         Satisfaction::Complete {
@@ -739,7 +752,7 @@ fn signature(
     };
 
     if let Some(psbt) = build_sat.psbt() {
-        policy.satisfaction = if signature_in_psbt(psbt, key, secp) {
+        policy.satisfaction = if find_sig(psbt) {
             Satisfaction::Complete {
                 condition: Default::default(),
             }
@@ -751,26 +764,119 @@ fn signature(
     policy
 }
 
-fn signature_in_psbt(psbt: &Psbt, key: &DescriptorPublicKey, secp: &SecpCtx) -> bool {
+fn generic_sig_in_psbt<
+    // C is for "check", it's a closure we use to *check* if a psbt input contains the signature
+    // for a specific key
+    C: Fn(&PsbtInput, &SinglePubKey) -> bool,
+    // E is for "extract", it extracts a key from the bip32 derivations found in the psbt input
+    E: Fn(&PsbtInput, Fingerprint) -> Option<SinglePubKey>,
+>(
+    psbt: &Psbt,
+    key: &DescriptorPublicKey,
+    secp: &SecpCtx,
+    check: C,
+    extract: E,
+) -> bool {
     //TODO check signature validity
     psbt.inputs.iter().all(|input| match key {
-        DescriptorPublicKey::SinglePub(key) => input.partial_sigs.contains_key(&key.key),
+        DescriptorPublicKey::SinglePub(DescriptorSinglePub { key, .. }) => check(input, key),
         DescriptorPublicKey::XPub(xpub) => {
-            let pubkey = input
-                .bip32_derivation
-                .iter()
-                .find(|(_, (f, _))| *f == xpub.root_fingerprint(secp))
-                .map(|(p, _)| p);
             //TODO check actual derivation matches
-            match pubkey {
-                Some(pubkey) => input.partial_sigs.contains_key(pubkey),
+            match extract(input, xpub.root_fingerprint(secp)) {
+                Some(pubkey) => check(input, &pubkey),
                 None => false,
             }
         }
     })
 }
 
-impl<Ctx: ScriptContext> ExtractPolicy for Miniscript<DescriptorPublicKey, Ctx> {
+trait SigExt: ScriptContext {
+    fn make_signature(
+        key: &DescriptorPublicKey,
+        signers: &SignersContainer,
+        build_sat: BuildSatisfaction,
+        secp: &SecpCtx,
+    ) -> Policy;
+
+    fn find_signature(psbt: &Psbt, key: &DescriptorPublicKey, secp: &SecpCtx) -> bool;
+}
+
+impl<T: ScriptContext + 'static> SigExt for T {
+    fn make_signature(
+        key: &DescriptorPublicKey,
+        signers: &SignersContainer,
+        build_sat: BuildSatisfaction,
+        secp: &SecpCtx,
+    ) -> Policy {
+        if T::as_enum().is_taproot() {
+            make_generic_signature(
+                key,
+                signers,
+                build_sat,
+                secp,
+                || SatisfiableItem::SchnorrSignature(PkOrF::from_key(key, secp)),
+                |psbt| Self::find_signature(psbt, key, secp),
+            )
+        } else {
+            make_generic_signature(
+                key,
+                signers,
+                build_sat,
+                secp,
+                || SatisfiableItem::EcdsaSignature(PkOrF::from_key(key, secp)),
+                |psbt| Self::find_signature(psbt, key, secp),
+            )
+        }
+    }
+
+    fn find_signature(psbt: &Psbt, key: &DescriptorPublicKey, secp: &SecpCtx) -> bool {
+        if T::as_enum().is_taproot() {
+            generic_sig_in_psbt(
+                psbt,
+                key,
+                secp,
+                |input, pk| {
+                    let pk = match pk {
+                        SinglePubKey::XOnly(pk) => pk,
+                        _ => return false,
+                    };
+
+                    if input.tap_internal_key == Some(*pk) && input.tap_key_sig.is_some() {
+                        true
+                    } else {
+                        input.tap_script_sigs.keys().any(|(sk, _)| sk == pk)
+                    }
+                },
+                |input, fing| {
+                    input
+                        .tap_key_origins
+                        .iter()
+                        .find(|(_, (_, (f, _)))| f == &fing)
+                        .map(|(pk, _)| SinglePubKey::XOnly(*pk))
+                },
+            )
+        } else {
+            generic_sig_in_psbt(
+                psbt,
+                key,
+                secp,
+                |input, pk| match pk {
+                    SinglePubKey::FullKey(pk) => input.partial_sigs.contains_key(pk),
+                    _ => false,
+                },
+                |input, fing| {
+                    input
+                        .bip32_derivation
+                        .iter()
+                        .find(|(_, (f, _))| f == &fing)
+                        .map(|(pk, _)| SinglePubKey::FullKey(PublicKey::new(*pk)))
+                },
+            )
+        }
+    }
+}
+
+impl<Ctx: ScriptContext + 'static> ExtractPolicy for Miniscript<DescriptorPublicKey, Ctx> {
     fn extract_policy(
         &self,
         signers: &SignersContainer,
@@ -780,8 +886,10 @@ impl<Ctx: ScriptContext> ExtractPolicy for Miniscript<DescriptorPublicKey, Ctx>
         Ok(match &self.node {
             // Leaves
             Terminal::True | Terminal::False => None,
-            Terminal::PkK(pubkey) => Some(signature(pubkey, signers, build_sat, secp)),
-            Terminal::PkH(pubkey_hash) => Some(signature(pubkey_hash, signers, build_sat, secp)),
+            Terminal::PkK(pubkey) => Some(Ctx::make_signature(pubkey, signers, build_sat, secp)),
+            Terminal::PkH(pubkey_hash) => {
+                Some(Ctx::make_signature(pubkey_hash, signers, build_sat, secp))
+            }
             Terminal::After(value) => {
                 let mut policy: Policy = SatisfiableItem::AbsoluteTimelock { value: *value }.into();
                 policy.contribution = Satisfaction::Complete {
@@ -842,8 +950,8 @@ impl<Ctx: ScriptContext> ExtractPolicy for Miniscript<DescriptorPublicKey, Ctx>
             Terminal::Hash160(hash) => {
                 Some(SatisfiableItem::Hash160Preimage { hash: *hash }.into())
             }
-            Terminal::Multi(k, pks) => {
-                Policy::make_multisig(pks, signers, build_sat, *k, false, secp)?
+            Terminal::Multi(k, pks) | Terminal::MultiA(k, pks) => {
+                Policy::make_multisig::<Ctx>(pks, signers, build_sat, *k, false, secp)?
             }
             // Identities
             Terminal::Alt(inner)
@@ -934,13 +1042,13 @@ impl ExtractPolicy for Descriptor<DescriptorPublicKey> {
         build_sat: BuildSatisfaction,
         secp: &SecpCtx,
     ) -> Result<Option<Policy>, Error> {
-        fn make_sortedmulti<Ctx: ScriptContext>(
+        fn make_sortedmulti<Ctx: ScriptContext + 'static>(
             keys: &SortedMultiVec<DescriptorPublicKey, Ctx>,
             signers: &SignersContainer,
             build_sat: BuildSatisfaction,
             secp: &SecpCtx,
         ) -> Result<Option<Policy>, Error> {
-            Ok(Policy::make_multisig(
+            Ok(Policy::make_multisig::<Ctx>(
                 keys.pks.as_ref(),
                 signers,
                 build_sat,
@@ -951,10 +1059,25 @@ impl ExtractPolicy for Descriptor<DescriptorPublicKey> {
         }
 
         match self {
-            Descriptor::Pkh(pk) => Ok(Some(signature(pk.as_inner(), signers, build_sat, secp))),
-            Descriptor::Wpkh(pk) => Ok(Some(signature(pk.as_inner(), signers, build_sat, secp))),
+            Descriptor::Pkh(pk) => Ok(Some(miniscript::Legacy::make_signature(
+                pk.as_inner(),
+                signers,
+                build_sat,
+                secp,
+            ))),
+            Descriptor::Wpkh(pk) => Ok(Some(miniscript::Segwitv0::make_signature(
+                pk.as_inner(),
+                signers,
+                build_sat,
+                secp,
+            ))),
             Descriptor::Sh(sh) => match sh.as_inner() {
-                ShInner::Wpkh(pk) => Ok(Some(signature(pk.as_inner(), signers, build_sat, secp))),
+                ShInner::Wpkh(pk) => Ok(Some(miniscript::Segwitv0::make_signature(
+                    pk.as_inner(),
+                    signers,
+                    build_sat,
+                    secp,
+                ))),
                 ShInner::Ms(ms) => Ok(ms.extract_policy(signers, build_sat, secp)?),
                 ShInner::SortedMulti(ref keys) => make_sortedmulti(keys, signers, build_sat, secp),
                 ShInner::Wsh(wsh) => match wsh.as_inner() {
@@ -969,6 +1092,28 @@ impl ExtractPolicy for Descriptor<DescriptorPublicKey> {
                 WshInner::SortedMulti(ref keys) => make_sortedmulti(keys, signers, build_sat, secp),
             },
             Descriptor::Bare(ms) => Ok(ms.as_inner().extract_policy(signers, build_sat, secp)?),
+            Descriptor::Tr(tr) => {
+                // If there's no tap tree, treat this as a single sig, otherwise build a `Thresh`
+                // node with threshold = 1 and the key spend signature plus all the tree leaves
+                let key_spend_sig =
+                    miniscript::Tap::make_signature(tr.internal_key(), signers, build_sat, secp);
+
+                if tr.taptree().is_none() {
+                    Ok(Some(key_spend_sig))
+                } else {
+                    let mut items = vec![key_spend_sig];
+                    items.append(
+                        &mut tr
+                            .iter_scripts()
+                            .filter_map(|(_, ms)| {
+                                ms.extract_policy(signers, build_sat, secp).transpose()
+                            })
+                            .collect::<Result<Vec<_>, _>>()?,
+                    );
+
+                    Ok(Policy::make_thresh(items, 1)?)
+                }
+            }
         }
     }
 }
@@ -980,7 +1125,7 @@ mod test {
 
     use super::*;
     use crate::descriptor::derived::AsDerived;
-    use crate::descriptor::policy::SatisfiableItem::{Multisig, Signature, Thresh};
+    use crate::descriptor::policy::SatisfiableItem::{EcdsaSignature, Multisig, Thresh};
     use crate::keys::{DescriptorKey, IntoDescriptorKey};
     use crate::wallet::signer::SignersContainer;
     use bitcoin::secp256k1::Secp256k1;
@@ -1002,7 +1147,7 @@ mod test {
     ) -> (DescriptorKey<Ctx>, DescriptorKey<Ctx>, Fingerprint) {
         let path = bip32::DerivationPath::from_str(path).unwrap();
         let tprv = bip32::ExtendedPrivKey::from_str(tprv).unwrap();
-        let tpub = bip32::ExtendedPubKey::from_private(secp, &tprv);
+        let tpub = bip32::ExtendedPubKey::from_priv(secp, &tprv);
         let fingerprint = tprv.fingerprint(secp);
         let prvkey = (tprv, path.clone()).into_descriptor_key().unwrap();
         let pubkey = (tpub, path).into_descriptor_key().unwrap();
@@ -1021,30 +1166,26 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = wallet_desc
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
             .unwrap();
 
-        assert!(
-            matches!(&policy.item, Signature(pk_or_f) if pk_or_f.fingerprint.unwrap() == fingerprint)
-        );
+        assert!(matches!(&policy.item, EcdsaSignature(PkOrF::Fingerprint(f)) if f == &fingerprint));
         assert!(matches!(&policy.contribution, Satisfaction::None));
 
         let desc = descriptor!(wpkh(prvkey)).unwrap();
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = wallet_desc
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
             .unwrap();
 
-        assert!(
-            matches!(&policy.item, Signature(pk_or_f) if pk_or_f.fingerprint.unwrap() == fingerprint)
-        );
+        assert!(matches!(&policy.item, EcdsaSignature(PkOrF::Fingerprint(f)) if f == &fingerprint));
         assert!(
             matches!(&policy.contribution, Satisfaction::Complete {condition} if condition.csv == None && condition.timelock == None)
         );
@@ -1060,7 +1201,7 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = wallet_desc
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
@@ -1068,8 +1209,8 @@ mod test {
 
         assert!(
             matches!(&policy.item, Multisig { keys, threshold } if threshold == &2usize
-            && keys[0].fingerprint.unwrap() == fingerprint0
-            && keys[1].fingerprint.unwrap() == fingerprint1)
+            && keys[0] == PkOrF::Fingerprint(fingerprint0)
+            && keys[1] == PkOrF::Fingerprint(fingerprint1))
         );
         // TODO should this be "Satisfaction::None" since we have no prv keys?
         // TODO should items and conditions not be empty?
@@ -1092,15 +1233,15 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = wallet_desc
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
             .unwrap();
         assert!(
             matches!(&policy.item, Multisig { keys, threshold } if threshold == &2usize
-            && keys[0].fingerprint.unwrap() == fingerprint0
-            && keys[1].fingerprint.unwrap() == fingerprint1)
+            && keys[0] == PkOrF::Fingerprint(fingerprint0)
+            && keys[1] == PkOrF::Fingerprint(fingerprint1))
         );
 
         assert!(
@@ -1124,7 +1265,7 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = wallet_desc
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
@@ -1132,8 +1273,8 @@ mod test {
 
         assert!(
             matches!(&policy.item, Multisig { keys, threshold } if threshold == &1
-            && keys[0].fingerprint.unwrap() == fingerprint0
-            && keys[1].fingerprint.unwrap() == fingerprint1)
+            && keys[0] == PkOrF::Fingerprint(fingerprint0)
+            && keys[1] == PkOrF::Fingerprint(fingerprint1))
         );
         assert!(
             matches!(&policy.contribution, Satisfaction::PartialComplete { n, m, items, conditions, .. } if n == &2
@@ -1156,7 +1297,7 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = wallet_desc
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
@@ -1164,8 +1305,8 @@ mod test {
 
         assert!(
             matches!(&policy.item, Multisig { keys, threshold } if threshold == &2
-            && keys[0].fingerprint.unwrap() == fingerprint0
-            && keys[1].fingerprint.unwrap() == fingerprint1)
+            && keys[0] == PkOrF::Fingerprint(fingerprint0)
+            && keys[1] == PkOrF::Fingerprint(fingerprint1))
         );
 
         assert!(
@@ -1189,15 +1330,13 @@ mod test {
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
         let single_key = wallet_desc.derive(0);
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = single_key
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
             .unwrap();
 
-        assert!(
-            matches!(&policy.item, Signature(pk_or_f) if pk_or_f.fingerprint.unwrap() == fingerprint)
-        );
+        assert!(matches!(&policy.item, EcdsaSignature(PkOrF::Fingerprint(f)) if f == &fingerprint));
         assert!(matches!(&policy.contribution, Satisfaction::None));
 
         let desc = descriptor!(wpkh(prvkey)).unwrap();
@@ -1205,15 +1344,13 @@ mod test {
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
         let single_key = wallet_desc.derive(0);
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = single_key
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
             .unwrap();
 
-        assert!(
-            matches!(&policy.item, Signature(pk_or_f) if pk_or_f.fingerprint.unwrap() == fingerprint)
-        );
+        assert!(matches!(&policy.item, EcdsaSignature(PkOrF::Fingerprint(f)) if f == &fingerprint));
         assert!(
             matches!(&policy.contribution, Satisfaction::Complete {condition} if condition.csv == None && condition.timelock == None)
         );
@@ -1232,7 +1369,7 @@ mod test {
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
         let single_key = wallet_desc.derive(0);
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = single_key
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
@@ -1240,8 +1377,8 @@ mod test {
 
         assert!(
             matches!(&policy.item, Multisig { keys, threshold } if threshold == &1
-            && keys[0].fingerprint.unwrap() == fingerprint0
-            && keys[1].fingerprint.unwrap() == fingerprint1)
+            && keys[0] == PkOrF::Fingerprint(fingerprint0)
+            && keys[1] == PkOrF::Fingerprint(fingerprint1))
         );
         assert!(
             matches!(&policy.contribution, Satisfaction::PartialComplete { n, m, items, conditions, .. } if n == &2
@@ -1275,7 +1412,7 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = wallet_desc
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
@@ -1314,7 +1451,7 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = wallet_desc
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
@@ -1339,7 +1476,7 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = wallet_desc
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
@@ -1357,7 +1494,7 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
         let policy = wallet_desc
             .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
@@ -1379,10 +1516,10 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers = keymap.into();
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
 
         let policy = wallet_desc
-            .extract_policy(&signers, BuildSatisfaction::None, &secp)
+            .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
             .unwrap()
             .unwrap();
 
@@ -1445,7 +1582,7 @@ mod test {
             addr.to_string()
         );
 
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
 
         let psbt = Psbt::from_str(ALICE_SIGNED_PSBT).unwrap();
 
@@ -1501,19 +1638,19 @@ mod test {
         let (prvkey_bob, _, _) = setup_keys(BOB_TPRV_STR, ALICE_BOB_PATH, &secp);
 
         let desc =
-            descriptor!(wsh(thresh(2,d:v:older(2),s:pk(prvkey_alice),s:pk(prvkey_bob)))).unwrap();
+            descriptor!(wsh(thresh(2,n:d:v:older(2),s:pk(prvkey_alice),s:pk(prvkey_bob)))).unwrap();
 
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
 
         let addr = wallet_desc
             .as_derived(0, &secp)
             .address(Network::Testnet)
             .unwrap();
         assert_eq!(
-            "tb1qhpemaacpeu8ajlnh8k9v55ftg0px58r8630fz8t5mypxcwdk5d8sum522g",
+            "tb1qsydsey4hexagwkvercqsmes6yet0ndkyt6uzcphtqnygjd8hmzmsfxrv58",
             addr.to_string()
         );
 
@@ -1594,9 +1731,185 @@ mod test {
         let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
 
         let policy = wallet_desc.extract_policy(&signers_container, BuildSatisfaction::None, &secp);
         assert!(policy.is_ok());
     }
+
+    #[test]
+    fn test_extract_tr_key_spend() {
+        let secp = Secp256k1::new();
+
+        let (prvkey, _, fingerprint) = setup_keys(ALICE_TPRV_STR, ALICE_BOB_PATH, &secp);
+
+        let desc = descriptor!(tr(prvkey)).unwrap();
+        let (wallet_desc, keymap) = desc
+            .into_wallet_descriptor(&secp, Network::Testnet)
+            .unwrap();
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
+
+        let policy = wallet_desc
+            .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
+            .unwrap();
+        assert_eq!(
+            policy,
+            Some(Policy {
+                id: "48u0tz0n".to_string(),
+                item: SatisfiableItem::SchnorrSignature(PkOrF::Fingerprint(fingerprint)),
+                satisfaction: Satisfaction::None,
+                contribution: Satisfaction::Complete {
+                    condition: Condition::default()
+                }
+            })
+        );
+    }
+
+    #[test]
+    fn test_extract_tr_script_spend() {
+        let secp = Secp256k1::new();
+
+        let (alice_prv, _, alice_fing) = setup_keys(ALICE_TPRV_STR, ALICE_BOB_PATH, &secp);
+        let (_, bob_pub, bob_fing) = setup_keys(BOB_TPRV_STR, ALICE_BOB_PATH, &secp);
+
+        let desc = descriptor!(tr(bob_pub, pk(alice_prv))).unwrap();
+        let (wallet_desc, keymap) = desc
+            .into_wallet_descriptor(&secp, Network::Testnet)
+            .unwrap();
+        let signers_container = Arc::new(SignersContainer::build(keymap, &wallet_desc, &secp));
+
+        let policy = wallet_desc
+            .extract_policy(&signers_container, BuildSatisfaction::None, &secp)
+            .unwrap()
+            .unwrap();
+
+        assert!(
+            matches!(policy.item, SatisfiableItem::Thresh { ref items, threshold: 1 } if items.len() == 2)
+        );
+        assert!(
+            matches!(policy.contribution, Satisfaction::PartialComplete { n: 2, m: 1, items, .. } if items == vec![1])
+        );
+
+        let alice_sig = SatisfiableItem::SchnorrSignature(PkOrF::Fingerprint(alice_fing));
+        let bob_sig = SatisfiableItem::SchnorrSignature(PkOrF::Fingerprint(bob_fing));
+
+        let thresh_items = match policy.item {
+            SatisfiableItem::Thresh { items, .. } => items,
+            _ => unreachable!(),
+        };
+
+        assert_eq!(thresh_items[0].item, bob_sig);
+        assert_eq!(thresh_items[1].item, alice_sig);
+    }
+
+    #[test]
+    fn test_extract_tr_satisfaction_key_spend() {
+        const UNSIGNED_PSBT: &str = "cHNidP8BAFMBAAAAAUKgMCqtGLSiGYhsTols2UJ/VQQgQi/SXO38uXs2SahdAQAAAAD/////ARyWmAAAAAAAF6kU4R3W8CnGzZcSsaovTYu0X8vHt3WHAAAAAAABASuAlpgAAAAAACJRIEiEBFjbZa1xdjLfFjrKzuC1F1LeRyI/gL6IuGKNmUuSIRYnkGTDxwXMHP32fkDFoGJY28trxbkkVgR2z7jZa2pOJA0AyRF8LgAAAIADAAAAARcgJ5Bkw8cFzBz99n5AxaBiWNvLa8W5JFYEds+42WtqTiQAAA==";
+        const SIGNED_PSBT: &str = "cHNidP8BAFMBAAAAAUKgMCqtGLSiGYhsTols2UJ/VQQgQi/SXO38uXs2SahdAQAAAAD/////ARyWmAAAAAAAF6kU4R3W8CnGzZcSsaovTYu0X8vHt3WHAAAAAAABASuAlpgAAAAAACJRIEiEBFjbZa1xdjLfFjrKzuC1F1LeRyI/gL6IuGKNmUuSARNAIsRvARpRxuyQosVA7guRQT9vXr+S25W2tnP2xOGBsSgq7A4RL8yrbvwDmNlWw9R0Nc/6t+IsyCyy7dD/lbUGgyEWJ5Bkw8cFzBz99n5AxaBiWNvLa8W5JFYEds+42WtqTiQNAMkRfC4AAACAAwAAAAEXICeQZMPHBcwc/fZ+QMWgYljby2vFuSRWBHbPuNlrak4kAAA=";
+
+        let unsigned_psbt = Psbt::from_str(UNSIGNED_PSBT).unwrap();
+        let signed_psbt = Psbt::from_str(SIGNED_PSBT).unwrap();
+
+        let secp = Secp256k1::new();
+
+        let (_, pubkey, _) = setup_keys(ALICE_TPRV_STR, ALICE_BOB_PATH, &secp);
+
+        let desc = descriptor!(tr(pubkey)).unwrap();
+        let (wallet_desc, _) = desc
+            .into_wallet_descriptor(&secp, Network::Testnet)
+            .unwrap();
+
+        let policy_unsigned = wallet_desc
+            .extract_policy(
+                &SignersContainer::default(),
+                BuildSatisfaction::Psbt(&unsigned_psbt),
+                &secp,
+            )
+            .unwrap()
+            .unwrap();
+        let policy_signed = wallet_desc
+            .extract_policy(
+                &SignersContainer::default(),
+                BuildSatisfaction::Psbt(&signed_psbt),
+                &secp,
+            )
+            .unwrap()
+            .unwrap();
+
+        assert_eq!(policy_unsigned.satisfaction, Satisfaction::None);
+        assert_eq!(
+            policy_signed.satisfaction,
+            Satisfaction::Complete {
+                condition: Default::default()
+            }
+        );
+    }
+
+    #[test]
+    fn test_extract_tr_satisfaction_script_spend() {
+        const UNSIGNED_PSBT: &str = "cHNidP8BAFMBAAAAAWZalxaErOL7P3WPIUc8DsjgE68S+ww+uqiqEI2SAwlPAAAAAAD/////AQiWmAAAAAAAF6kU4R3W8CnGzZcSsaovTYu0X8vHt3WHAAAAAAABASuAlpgAAAAAACJRINa6bLPZwp3/CYWoxyI3mLYcSC5f9LInAMUng94nspa2IhXBgiPY+kcolS1Hp0niOK/+7VHz6F+nsz8JVxnzWzkgToYjIHhGyuexxtRVKevRx4YwWR/W0r7LPHt6oS6DLlzyuYQarMAhFnhGyuexxtRVKevRx4YwWR/W0r7LPHt6oS6DLlzyuYQaLQH2onWFc3UR6I9ZhuHVeJCi5LNAf4APVd7mHn4BhdViHRwu7j4AAACAAgAAACEWgiPY+kcolS1Hp0niOK/+7VHz6F+nsz8JVxnzWzkgToYNAMkRfC4AAACAAgAAAAEXIIIj2PpHKJUtR6dJ4jiv/u1R8+hfp7M/CVcZ81s5IE6GARgg9qJ1hXN1EeiPWYbh1XiQouSzQH+AD1Xe5h5+AYXVYh0AAA==";
+        const SIGNED_PSBT: &str = "cHNidP8BAFMBAAAAAWZalxaErOL7P3WPIUc8DsjgE68S+ww+uqiqEI2SAwlPAAAAAAD/////AQiWmAAAAAAAF6kU4R3W8CnGzZcSsaovTYu0X8vHt3WHAAAAAAABASuAlpgAAAAAACJRINa6bLPZwp3/CYWoxyI3mLYcSC5f9LInAMUng94nspa2AQcAAQhCAUALcP9w/+Ddly9DWdhHTnQ9uCDWLPZjR6vKbKePswW2Ee6W5KNfrklus/8z98n7BQ1U4vADHk0FbadeeL8rrbHlARNAC3D/cP/g3ZcvQ1nYR050Pbgg1iz2Y0erymynj7MFthHuluSjX65JbrP/M/fJ+wUNVOLwAx5NBW2nXni/K62x5UEUeEbK57HG1FUp69HHhjBZH9bSvss8e3qhLoMuXPK5hBr2onWFc3UR6I9ZhuHVeJCi5LNAf4APVd7mHn4BhdViHUAXNmWieJ80Fs+PMa2C186YOBPZbYG/ieEUkagMwzJ788SoCucNdp5wnxfpuJVygFhglDrXGzujFtC82PrMohwuIhXBgiPY+kcolS1Hp0niOK/+7VHz6F+nsz8JVxnzWzkgToYjIHhGyuexxtRVKevRx4YwWR/W0r7LPHt6oS6DLlzyuYQarMAhFnhGyuexxtRVKevRx4YwWR/W0r7LPHt6oS6DLlzyuYQaLQH2onWFc3UR6I9ZhuHVeJCi5LNAf4APVd7mHn4BhdViHRwu7j4AAACAAgAAACEWgiPY+kcolS1Hp0niOK/+7VHz6F+nsz8JVxnzWzkgToYNAMkRfC4AAACAAgAAAAEXIIIj2PpHKJUtR6dJ4jiv/u1R8+hfp7M/CVcZ81s5IE6GARgg9qJ1hXN1EeiPWYbh1XiQouSzQH+AD1Xe5h5+AYXVYh0AAA==";
+
+        let unsigned_psbt = Psbt::from_str(UNSIGNED_PSBT).unwrap();
+        let signed_psbt = Psbt::from_str(SIGNED_PSBT).unwrap();
+
+        let secp = Secp256k1::new();
+
+        let (_, alice_pub, _) = setup_keys(ALICE_TPRV_STR, ALICE_BOB_PATH, &secp);
+        let (_, bob_pub, _) = setup_keys(BOB_TPRV_STR, ALICE_BOB_PATH, &secp);
+
+        let desc = descriptor!(tr(bob_pub, pk(alice_pub))).unwrap();
+        let (wallet_desc, _) = desc
+            .into_wallet_descriptor(&secp, Network::Testnet)
+            .unwrap();
+
+        let policy_unsigned = wallet_desc
+            .extract_policy(
+                &SignersContainer::default(),
+                BuildSatisfaction::Psbt(&unsigned_psbt),
+                &secp,
+            )
+            .unwrap()
+            .unwrap();
+        let policy_signed = wallet_desc
+            .extract_policy(
+                &SignersContainer::default(),
+                BuildSatisfaction::Psbt(&signed_psbt),
+                &secp,
+            )
+            .unwrap()
+            .unwrap();
+
+        assert!(
+            matches!(policy_unsigned.item, SatisfiableItem::Thresh { ref items, threshold: 1 } if items.len() == 2)
+        );
+        assert!(
+            matches!(policy_unsigned.satisfaction, Satisfaction::Partial { n: 2, m: 1, items, .. } if items.is_empty())
+        );
+
+        assert!(
+            matches!(policy_signed.item, SatisfiableItem::Thresh { ref items, threshold: 1 } if items.len() == 2)
+        );
+        assert!(
+            matches!(policy_signed.satisfaction, Satisfaction::PartialComplete { n: 2, m: 1, items, .. } if items == vec![0, 1])
+        );
+
+        let satisfied_items = match policy_signed.item {
+            SatisfiableItem::Thresh { items, .. } => items,
+            _ => unreachable!(),
+        };
+
+        assert_eq!(
+            satisfied_items[0].satisfaction,
+            Satisfaction::Complete {
+                condition: Default::default()
+            }
+        );
+        assert_eq!(
+            satisfied_items[1].satisfaction,
+            Satisfaction::Complete {
+                condition: Default::default()
+            }
+        );
+    }
 }

src/descriptor/template.rs

@@ -40,18 +40,19 @@ pub type DescriptorTemplateOut = (ExtendedDescriptor, KeyMap, ValidNetworks);
 /// use bdk::keys::{IntoDescriptorKey, KeyError};
 /// use bdk::miniscript::Legacy;
 /// use bdk::template::{DescriptorTemplate, DescriptorTemplateOut};
+/// use bitcoin::Network;
 ///
 /// struct MyP2PKH<K: IntoDescriptorKey<Legacy>>(K);
 ///
 /// impl<K: IntoDescriptorKey<Legacy>> DescriptorTemplate for MyP2PKH<K> {
-///     fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
+///     fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
 ///         Ok(bdk::descriptor!(pkh(self.0))?)
 ///     }
 /// }
 /// ```
 pub trait DescriptorTemplate {
     /// Build the complete descriptor
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError>;
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError>;
 }
 
 /// Turns a [`DescriptorTemplate`] into a valid wallet descriptor by calling its
@@ -62,7 +63,7 @@ impl<T: DescriptorTemplate> IntoWalletDescriptor for T {
         secp: &SecpCtx,
         network: Network,
     ) -> Result<(ExtendedDescriptor, KeyMap), DescriptorError> {
-        self.build()?.into_wallet_descriptor(secp, network)
+        self.build(network)?.into_wallet_descriptor(secp, network)
     }
 }
 
@@ -79,7 +80,7 @@ impl<T: DescriptorTemplate> IntoWalletDescriptor for T {
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     P2Pkh(key),
 ///     None,
 ///     Network::Testnet,
@@ -95,7 +96,7 @@ impl<T: DescriptorTemplate> IntoWalletDescriptor for T {
 pub struct P2Pkh<K: IntoDescriptorKey<Legacy>>(pub K);
 
 impl<K: IntoDescriptorKey<Legacy>> DescriptorTemplate for P2Pkh<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
+    fn build(self, _network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
         descriptor!(pkh(self.0))
     }
 }
@@ -113,7 +114,7 @@ impl<K: IntoDescriptorKey<Legacy>> DescriptorTemplate for P2Pkh<K> {
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     P2Wpkh_P2Sh(key),
 ///     None,
 ///     Network::Testnet,
@@ -130,7 +131,7 @@ impl<K: IntoDescriptorKey<Legacy>> DescriptorTemplate for P2Pkh<K> {
 pub struct P2Wpkh_P2Sh<K: IntoDescriptorKey<Segwitv0>>(pub K);
 
 impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh_P2Sh<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
+    fn build(self, _network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
         descriptor!(sh(wpkh(self.0)))
     }
 }
@@ -148,7 +149,7 @@ impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh_P2Sh<K> {
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     P2Wpkh(key),
 ///     None,
 ///     Network::Testnet,
@@ -164,12 +165,12 @@ impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh_P2Sh<K> {
 pub struct P2Wpkh<K: IntoDescriptorKey<Segwitv0>>(pub K);
 
 impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
+    fn build(self, _network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
         descriptor!(wpkh(self.0))
     }
 }
 
-/// BIP44 template. Expands to `pkh(key/44'/0'/0'/{0,1}/*)`
+/// BIP44 template. Expands to `pkh(key/44'/{0,1}'/0'/{0,1}/*)`
 ///
 /// Since there are hardened derivation steps, this template requires a private derivable key (generally a `xprv`/`tprv`).
 ///
@@ -186,28 +187,28 @@ impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh<K> {
 /// use bdk::template::Bip44;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip44(key.clone(), KeychainKind::External),
 ///     Some(Bip44(key, KeychainKind::Internal)),
 ///     Network::Testnet,
 ///     MemoryDatabase::default()
 /// )?;
 ///
-/// assert_eq!(wallet.get_address(New)?.to_string(), "miNG7dJTzJqNbFS19svRdTCisC65dsubtR");
-/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "pkh([c55b303f/44'/0'/0']tpubDDDzQ31JkZB7VxUr9bjvBivDdqoFLrDPyLWtLapArAi51ftfmCb2DPxwLQzX65iNcXz1DGaVvyvo6JQ6rTU73r2gqdEo8uov9QKRb7nKCSU/0/*)#xgaaevjx");
+/// assert_eq!(wallet.get_address(New)?.to_string(), "mmogjc7HJEZkrLqyQYqJmxUqFaC7i4uf89");
+/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "pkh([c55b303f/44'/1'/0']tpubDCuorCpzvYS2LCD75BR46KHE8GdDeg1wsAgNZeNr6DaB5gQK1o14uErKwKLuFmeemkQ6N2m3rNgvctdJLyr7nwu2yia7413Hhg8WWE44cgT/0/*)#5wrnv0xt");
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
 pub struct Bip44<K: DerivableKey<Legacy>>(pub K, pub KeychainKind);
 
 impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        P2Pkh(legacy::make_bipxx_private(44, self.0, self.1)?).build()
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Pkh(legacy::make_bipxx_private(44, self.0, self.1, network)?).build(network)
     }
 }
 
 /// BIP44 public template. Expands to `pkh(key/{0,1}/*)`
 ///
-/// This assumes that the key used has already been derived with `m/44'/0'/0'`.
+/// This assumes that the key used has already been derived with `m/44'/0'/0'` for Mainnet or `m/44'/1'/0'` for Testnet.
 ///
 /// This template requires the parent fingerprint to populate correctly the metadata of PSBTs.
 ///
@@ -226,7 +227,7 @@ impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44<K> {
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDDDzQ31JkZB7VxUr9bjvBivDdqoFLrDPyLWtLapArAi51ftfmCb2DPxwLQzX65iNcXz1DGaVvyvo6JQ6rTU73r2gqdEo8uov9QKRb7nKCSU")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip44Public(key.clone(), fingerprint, KeychainKind::External),
 ///     Some(Bip44Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -240,12 +241,12 @@ impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44<K> {
 pub struct Bip44Public<K: DerivableKey<Legacy>>(pub K, pub bip32::Fingerprint, pub KeychainKind);
 
 impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44Public<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        P2Pkh(legacy::make_bipxx_public(44, self.0, self.1, self.2)?).build()
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Pkh(legacy::make_bipxx_public(44, self.0, self.1, self.2)?).build(network)
     }
 }
 
-/// BIP49 template. Expands to `sh(wpkh(key/49'/0'/0'/{0,1}/*))`
+/// BIP49 template. Expands to `sh(wpkh(key/49'/{0,1}'/0'/{0,1}/*))`
 ///
 /// Since there are hardened derivation steps, this template requires a private derivable key (generally a `xprv`/`tprv`).
 ///
@@ -262,22 +263,22 @@ impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44Public<K> {
 /// use bdk::template::Bip49;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip49(key.clone(), KeychainKind::External),
 ///     Some(Bip49(key, KeychainKind::Internal)),
 ///     Network::Testnet,
 ///     MemoryDatabase::default()
 /// )?;
 ///
-/// assert_eq!(wallet.get_address(New)?.to_string(), "2N3K4xbVAHoiTQSwxkZjWDfKoNC27pLkYnt");
-/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "sh(wpkh([c55b303f/49\'/0\'/0\']tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L/0/*))#gsmdv4xr");
+/// assert_eq!(wallet.get_address(New)?.to_string(), "2N4zkWAoGdUv4NXhSsU8DvS5MB36T8nKHEB");
+/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "sh(wpkh([c55b303f/49'/1'/0']tpubDDYr4kdnZgjjShzYNjZUZXUUtpXaofdkMaipyS8ThEh45qFmhT4hKYways7UXmg6V7het1QiFo9kf4kYUXyDvV4rHEyvSpys9pjCB3pukxi/0/*))#s9vxlc8e");
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
 pub struct Bip49<K: DerivableKey<Segwitv0>>(pub K, pub KeychainKind);
 
 impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        P2Wpkh_P2Sh(segwit_v0::make_bipxx_private(49, self.0, self.1)?).build()
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Wpkh_P2Sh(segwit_v0::make_bipxx_private(49, self.0, self.1, network)?).build(network)
     }
 }
 
@@ -302,7 +303,7 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49<K> {
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip49Public(key.clone(), fingerprint, KeychainKind::External),
 ///     Some(Bip49Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -310,18 +311,18 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49<K> {
 /// )?;
 ///
 /// assert_eq!(wallet.get_address(New)?.to_string(), "2N3K4xbVAHoiTQSwxkZjWDfKoNC27pLkYnt");
-/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "sh(wpkh([c55b303f/49\'/0\'/0\']tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L/0/*))#gsmdv4xr");
+/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "sh(wpkh([c55b303f/49'/0'/0']tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L/0/*))#gsmdv4xr");
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
 pub struct Bip49Public<K: DerivableKey<Segwitv0>>(pub K, pub bip32::Fingerprint, pub KeychainKind);
 
 impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49Public<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        P2Wpkh_P2Sh(segwit_v0::make_bipxx_public(49, self.0, self.1, self.2)?).build()
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Wpkh_P2Sh(segwit_v0::make_bipxx_public(49, self.0, self.1, self.2)?).build(network)
     }
 }
 
-/// BIP84 template. Expands to `wpkh(key/84'/0'/0'/{0,1}/*)`
+/// BIP84 template. Expands to `wpkh(key/84'/{0,1}'/0'/{0,1}/*)`
 ///
 /// Since there are hardened derivation steps, this template requires a private derivable key (generally a `xprv`/`tprv`).
 ///
@@ -338,22 +339,22 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49Public<K> {
 /// use bdk::template::Bip84;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip84(key.clone(), KeychainKind::External),
 ///     Some(Bip84(key, KeychainKind::Internal)),
 ///     Network::Testnet,
 ///     MemoryDatabase::default()
 /// )?;
 ///
-/// assert_eq!(wallet.get_address(New)?.to_string(), "tb1qedg9fdlf8cnnqfd5mks6uz5w4kgpk2pr6y4qc7");
-/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "wpkh([c55b303f/84\'/0\'/0\']tpubDC2Qwo2TFsaNC4ju8nrUJ9mqVT3eSgdmy1yPqhgkjwmke3PRXutNGRYAUo6RCHTcVQaDR3ohNU9we59brGHuEKPvH1ags2nevW5opEE9Z5Q/0/*)#nkk5dtkg");
+/// assert_eq!(wallet.get_address(New)?.to_string(), "tb1qhl85z42h7r4su5u37rvvw0gk8j2t3n9y7zsg4n");
+/// assert_eq!(wallet.public_descriptor(KeychainKind::External)?.unwrap().to_string(), "wpkh([c55b303f/84'/1'/0']tpubDDc5mum24DekpNw92t6fHGp8Gr2JjF9J7i4TZBtN6Vp8xpAULG5CFaKsfugWa5imhrQQUZKXe261asP5koDHo5bs3qNTmf3U3o4v9SaB8gg/0/*)#6kfecsmr");
 /// # Ok::<_, Box<dyn std::error::Error>>(())
 /// ```
 pub struct Bip84<K: DerivableKey<Segwitv0>>(pub K, pub KeychainKind);
 
 impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip84<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        P2Wpkh(segwit_v0::make_bipxx_private(84, self.0, self.1)?).build()
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Wpkh(segwit_v0::make_bipxx_private(84, self.0, self.1, network)?).build(network)
     }
 }
 
@@ -378,7 +379,7 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip84<K> {
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC2Qwo2TFsaNC4ju8nrUJ9mqVT3eSgdmy1yPqhgkjwmke3PRXutNGRYAUo6RCHTcVQaDR3ohNU9we59brGHuEKPvH1ags2nevW5opEE9Z5Q")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new_offline(
+/// let wallet = Wallet::new(
 ///     Bip84Public(key.clone(), fingerprint, KeychainKind::External),
 ///     Some(Bip84Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -392,8 +393,8 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip84<K> {
 pub struct Bip84Public<K: DerivableKey<Segwitv0>>(pub K, pub bip32::Fingerprint, pub KeychainKind);
 
 impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip84Public<K> {
-    fn build(self) -> Result<DescriptorTemplateOut, DescriptorError> {
-        P2Wpkh(segwit_v0::make_bipxx_public(84, self.0, self.1, self.2)?).build()
+    fn build(self, network: Network) -> Result<DescriptorTemplateOut, DescriptorError> {
+        P2Wpkh(segwit_v0::make_bipxx_public(84, self.0, self.1, self.2)?).build(network)
     }
 }
 
@@ -406,10 +407,19 @@ macro_rules! expand_make_bipxx {
                 bip: u32,
                 key: K,
                 keychain: KeychainKind,
+                network: Network,
             ) -> Result<impl IntoDescriptorKey<$ctx>, DescriptorError> {
                 let mut derivation_path = Vec::with_capacity(4);
                 derivation_path.push(bip32::ChildNumber::from_hardened_idx(bip)?);
-                derivation_path.push(bip32::ChildNumber::from_hardened_idx(0)?);
+
+                match network {
+                    Network::Bitcoin => {
+                        derivation_path.push(bip32::ChildNumber::from_hardened_idx(0)?);
+                    }
+                    _ => {
+                        derivation_path.push(bip32::ChildNumber::from_hardened_idx(1)?);
+                    }
+                }
                 derivation_path.push(bip32::ChildNumber::from_hardened_idx(0)?);
 
                 match keychain {
@@ -466,6 +476,40 @@ mod test {
     use miniscript::descriptor::{DescriptorPublicKey, DescriptorTrait, KeyMap};
     use miniscript::Descriptor;
 
+    // BIP44 `pkh(key/44'/{0,1}'/0'/{0,1}/*)`
+    #[test]
+    fn test_bip44_template_cointype() {
+        use bitcoin::util::bip32::ChildNumber::{self, Hardened};
+
+        let xprvkey = bitcoin::util::bip32::ExtendedPrivKey::from_str("xprv9s21ZrQH143K2fpbqApQL69a4oKdGVnVN52R82Ft7d1pSqgKmajF62acJo3aMszZb6qQ22QsVECSFxvf9uyxFUvFYQMq3QbtwtRSMjLAhMf").unwrap();
+        assert_eq!(Network::Bitcoin, xprvkey.network);
+        let xdesc = Bip44(xprvkey, KeychainKind::Internal)
+            .build(Network::Bitcoin)
+            .unwrap();
+
+        if let ExtendedDescriptor::Pkh(pkh) = xdesc.0 {
+            let path: Vec<ChildNumber> = pkh.into_inner().full_derivation_path().into();
+            let purpose = path.get(0).unwrap();
+            assert!(matches!(purpose, Hardened { index: 44 }));
+            let coin_type = path.get(1).unwrap();
+            assert!(matches!(coin_type, Hardened { index: 0 }));
+        }
+
+        let tprvkey = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
+        assert_eq!(Network::Testnet, tprvkey.network);
+        let tdesc = Bip44(tprvkey, KeychainKind::Internal)
+            .build(Network::Testnet)
+            .unwrap();
+
+        if let ExtendedDescriptor::Pkh(pkh) = tdesc.0 {
+            let path: Vec<ChildNumber> = pkh.into_inner().full_derivation_path().into();
+            let purpose = path.get(0).unwrap();
+            assert!(matches!(purpose, Hardened { index: 44 }));
+            let coin_type = path.get(1).unwrap();
+            assert!(matches!(coin_type, Hardened { index: 1 }));
+        }
+    }
+
     // verify template descriptor generates expected address(es)
     fn check(
         desc: Result<(Descriptor<DescriptorPublicKey>, KeyMap, ValidNetworks), DescriptorError>,
@@ -497,7 +541,7 @@ mod test {
             bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")
                 .unwrap();
         check(
-            P2Pkh(prvkey).build(),
+            P2Pkh(prvkey).build(Network::Bitcoin),
             false,
             true,
             &["mwJ8hxFYW19JLuc65RCTaP4v1rzVU8cVMT"],
@@ -508,7 +552,7 @@ mod test {
         )
         .unwrap();
         check(
-            P2Pkh(pubkey).build(),
+            P2Pkh(pubkey).build(Network::Bitcoin),
             false,
             true,
             &["muZpTpBYhxmRFuCjLc7C6BBDF32C8XVJUi"],
@@ -522,7 +566,7 @@ mod test {
             bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")
                 .unwrap();
         check(
-            P2Wpkh_P2Sh(prvkey).build(),
+            P2Wpkh_P2Sh(prvkey).build(Network::Bitcoin),
             true,
             true,
             &["2NB4ox5VDRw1ecUv6SnT3VQHPXveYztRqk5"],
@@ -533,7 +577,7 @@ mod test {
         )
         .unwrap();
         check(
-            P2Wpkh_P2Sh(pubkey).build(),
+            P2Wpkh_P2Sh(pubkey).build(Network::Bitcoin),
             true,
             true,
             &["2N5LiC3CqzxDamRTPG1kiNv1FpNJQ7x28sb"],
@@ -547,7 +591,7 @@ mod test {
             bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")
                 .unwrap();
         check(
-            P2Wpkh(prvkey).build(),
+            P2Wpkh(prvkey).build(Network::Bitcoin),
             true,
             true,
             &["bcrt1q4525hmgw265tl3drrl8jjta7ayffu6jfcwxx9y"],
@@ -558,7 +602,7 @@ mod test {
         )
         .unwrap();
         check(
-            P2Wpkh(pubkey).build(),
+            P2Wpkh(pubkey).build(Network::Bitcoin),
             true,
             true,
             &["bcrt1qngw83fg8dz0k749cg7k3emc7v98wy0c7azaa6h"],
@@ -570,7 +614,7 @@ mod test {
     fn test_bip44_template() {
         let prvkey = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
         check(
-            Bip44(prvkey, KeychainKind::External).build(),
+            Bip44(prvkey, KeychainKind::External).build(Network::Bitcoin),
             false,
             false,
             &[
@@ -580,7 +624,7 @@ mod test {
             ],
         );
         check(
-            Bip44(prvkey, KeychainKind::Internal).build(),
+            Bip44(prvkey, KeychainKind::Internal).build(Network::Bitcoin),
             false,
             false,
             &[
@@ -597,7 +641,7 @@ mod test {
         let pubkey = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDDDzQ31JkZB7VxUr9bjvBivDdqoFLrDPyLWtLapArAi51ftfmCb2DPxwLQzX65iNcXz1DGaVvyvo6JQ6rTU73r2gqdEo8uov9QKRb7nKCSU").unwrap();
         let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f").unwrap();
         check(
-            Bip44Public(pubkey, fingerprint, KeychainKind::External).build(),
+            Bip44Public(pubkey, fingerprint, KeychainKind::External).build(Network::Bitcoin),
             false,
             false,
             &[
@@ -607,7 +651,7 @@ mod test {
             ],
         );
         check(
-            Bip44Public(pubkey, fingerprint, KeychainKind::Internal).build(),
+            Bip44Public(pubkey, fingerprint, KeychainKind::Internal).build(Network::Bitcoin),
             false,
             false,
             &[
@@ -623,7 +667,7 @@ mod test {
     fn test_bip49_template() {
         let prvkey = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
         check(
-            Bip49(prvkey, KeychainKind::External).build(),
+            Bip49(prvkey, KeychainKind::External).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -633,7 +677,7 @@ mod test {
             ],
         );
         check(
-            Bip49(prvkey, KeychainKind::Internal).build(),
+            Bip49(prvkey, KeychainKind::Internal).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -650,7 +694,7 @@ mod test {
         let pubkey = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L").unwrap();
         let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f").unwrap();
         check(
-            Bip49Public(pubkey, fingerprint, KeychainKind::External).build(),
+            Bip49Public(pubkey, fingerprint, KeychainKind::External).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -660,7 +704,7 @@ mod test {
             ],
         );
         check(
-            Bip49Public(pubkey, fingerprint, KeychainKind::Internal).build(),
+            Bip49Public(pubkey, fingerprint, KeychainKind::Internal).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -676,7 +720,7 @@ mod test {
     fn test_bip84_template() {
         let prvkey = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy").unwrap();
         check(
-            Bip84(prvkey, KeychainKind::External).build(),
+            Bip84(prvkey, KeychainKind::External).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -686,7 +730,7 @@ mod test {
             ],
         );
         check(
-            Bip84(prvkey, KeychainKind::Internal).build(),
+            Bip84(prvkey, KeychainKind::Internal).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -703,7 +747,7 @@ mod test {
         let pubkey = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC2Qwo2TFsaNC4ju8nrUJ9mqVT3eSgdmy1yPqhgkjwmke3PRXutNGRYAUo6RCHTcVQaDR3ohNU9we59brGHuEKPvH1ags2nevW5opEE9Z5Q").unwrap();
         let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f").unwrap();
         check(
-            Bip84Public(pubkey, fingerprint, KeychainKind::External).build(),
+            Bip84Public(pubkey, fingerprint, KeychainKind::External).build(Network::Bitcoin),
             true,
             false,
             &[
@@ -713,7 +757,7 @@ mod test {
             ],
         );
         check(
-            Bip84Public(pubkey, fingerprint, KeychainKind::Internal).build(),
+            Bip84Public(pubkey, fingerprint, KeychainKind::Internal).build(Network::Bitcoin),
             true,
             false,
             &[

src/doctest.rs

@@ -9,6 +9,6 @@
 // You may not use this file except in accordance with one or both of these
 // licenses.
 
-#[doc(include = "../README.md")]
+#[doc = include_str!("../README.md")]
 #[cfg(doctest)]
 pub struct ReadmeDoctests;

src/error.rs

@@ -13,7 +13,7 @@ use std::fmt;
 
 use crate::bitcoin::Network;
 use crate::{descriptor, wallet, wallet::address_validator};
-use bitcoin::OutPoint;
+use bitcoin::{OutPoint, Txid};
 
 /// Errors that can be thrown by the [`Wallet`](crate::wallet::Wallet)
 #[derive(Debug)]
@@ -115,7 +115,7 @@ pub enum Error {
     Hex(bitcoin::hashes::hex::Error),
     /// Partially signed bitcoin transaction error
     Psbt(bitcoin::util::psbt::Error),
-    /// Partially signed bitcoin transaction parseerror
+    /// Partially signed bitcoin transaction parse error
     PsbtParse(bitcoin::util::psbt::PsbtParseError),
 
     //KeyMismatch(bitcoin::secp256k1::PublicKey, bitcoin::secp256k1::PublicKey),
@@ -125,6 +125,10 @@ pub enum Error {
     //DifferentDescriptorStructure,
     //Uncapable(crate::blockchain::Capability),
     //MissingCachedAddresses,
+    /// [`crate::blockchain::WalletSync`] sync attempt failed due to missing scripts in cache which
+    /// are needed to satisfy `stop_gap`.
+    MissingCachedScripts(MissingCachedScripts),
+
     #[cfg(feature = "electrum")]
     /// Electrum client error
     Electrum(electrum_client::Error),
@@ -145,6 +149,16 @@ pub enum Error {
     Rusqlite(rusqlite::Error),
 }
 
+/// Represents the last failed [`crate::blockchain::WalletSync`] sync attempt in which we were short
+/// on cached `scriptPubKey`s.
+#[derive(Debug)]
+pub struct MissingCachedScripts {
+    /// Number of scripts in which txs were requested during last request.
+    pub last_count: usize,
+    /// Minimum number of scripts to cache more of in order to satisfy `stop_gap`.
+    pub missing_count: usize,
+}
+
 impl fmt::Display for Error {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         write!(f, "{:?}", self)

src/keys/bip39.rs

@@ -19,7 +19,7 @@ use bitcoin::Network;
 
 use miniscript::ScriptContext;
 
-pub use bip39::{Language, Mnemonic};
+pub use bip39::{Error, Language, Mnemonic};
 
 type Seed = [u8; 64];
 
@@ -94,6 +94,23 @@ impl<Ctx: ScriptContext> DerivableKey<Ctx> for MnemonicWithPassphrase {
     }
 }
 
+#[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
+impl<Ctx: ScriptContext> DerivableKey<Ctx> for (GeneratedKey<Mnemonic, Ctx>, Option<String>) {
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        let (mnemonic, passphrase) = self;
+        (mnemonic.into_key(), passphrase).into_extended_key()
+    }
+
+    fn into_descriptor_key(
+        self,
+        source: Option<bip32::KeySource>,
+        derivation_path: bip32::DerivationPath,
+    ) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let (mnemonic, passphrase) = self;
+        (mnemonic.into_key(), passphrase).into_descriptor_key(source, derivation_path)
+    }
+}
+
 #[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
 impl<Ctx: ScriptContext> DerivableKey<Ctx> for Mnemonic {
     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {

src/keys/mod.rs

@@ -20,12 +20,12 @@ use std::str::FromStr;
 use bitcoin::secp256k1::{self, Secp256k1, Signing};
 
 use bitcoin::util::bip32;
-use bitcoin::{Network, PrivateKey, PublicKey};
+use bitcoin::{Network, PrivateKey, PublicKey, XOnlyPublicKey};
 
 use miniscript::descriptor::{Descriptor, DescriptorXKey, Wildcard};
 pub use miniscript::descriptor::{
     DescriptorPublicKey, DescriptorSecretKey, DescriptorSinglePriv, DescriptorSinglePub, KeyMap,
-    SortedMultiVec,
+    SinglePubKey, SortedMultiVec,
 };
 pub use miniscript::ScriptContext;
 use miniscript::{Miniscript, Terminal};
@@ -127,6 +127,8 @@ pub enum ScriptContextEnum {
     Legacy,
     /// Segwitv0 scripts
     Segwitv0,
+    /// Taproot scripts
+    Tap,
 }
 
 impl ScriptContextEnum {
@@ -139,6 +141,11 @@ impl ScriptContextEnum {
     pub fn is_segwit_v0(&self) -> bool {
         self == &ScriptContextEnum::Segwitv0
     }
+
+    /// Returns whether the script context is [`ScriptContextEnum::Tap`]
+    pub fn is_taproot(&self) -> bool {
+        self == &ScriptContextEnum::Tap
+    }
 }
 
 /// Trait that adds extra useful methods to [`ScriptContext`]s
@@ -155,6 +162,11 @@ pub trait ExtScriptContext: ScriptContext {
     fn is_segwit_v0() -> bool {
         Self::as_enum().is_segwit_v0()
     }
+
+    /// Returns whether the script context is [`Tap`](miniscript::Tap), aka Taproot or Segwit V1
+    fn is_taproot() -> bool {
+        Self::as_enum().is_taproot()
+    }
 }
 
 impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
@@ -162,6 +174,7 @@ impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
         match TypeId::of::<Ctx>() {
             t if t == TypeId::of::<miniscript::Legacy>() => ScriptContextEnum::Legacy,
             t if t == TypeId::of::<miniscript::Segwitv0>() => ScriptContextEnum::Segwitv0,
+            t if t == TypeId::of::<miniscript::Tap>() => ScriptContextEnum::Tap,
             _ => unimplemented!("Unknown ScriptContext type"),
         }
     }
@@ -212,7 +225,7 @@ impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
 ///
 /// use bdk::keys::{
 ///     mainnet_network, DescriptorKey, DescriptorPublicKey, DescriptorSinglePub,
-///     IntoDescriptorKey, KeyError, ScriptContext,
+///     IntoDescriptorKey, KeyError, ScriptContext, SinglePubKey,
 /// };
 ///
 /// pub struct MyKeyType {
@@ -224,7 +237,7 @@ impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
 ///         Ok(DescriptorKey::from_public(
 ///             DescriptorPublicKey::SinglePub(DescriptorSinglePub {
 ///                 origin: None,
-///                 key: self.pubkey,
+///                 key: SinglePubKey::FullKey(self.pubkey),
 ///             }),
 ///             mainnet_network(),
 ///         ))
@@ -333,7 +346,7 @@ impl<Ctx: ScriptContext> ExtendedKey<Ctx> {
         secp: &Secp256k1<C>,
     ) -> bip32::ExtendedPubKey {
         let mut xpub = match self {
-            ExtendedKey::Private((xprv, _)) => bip32::ExtendedPubKey::from_private(secp, &xprv),
+            ExtendedKey::Private((xprv, _)) => bip32::ExtendedPubKey::from_priv(secp, &xprv),
             ExtendedKey::Public((xpub, _)) => xpub,
         };
 
@@ -356,7 +369,7 @@ impl<Ctx: ScriptContext> From<bip32::ExtendedPrivKey> for ExtendedKey<Ctx> {
 
 /// Trait for keys that can be derived.
 ///
-/// When extra metadata are provided, a [`DerivableKey`] can be transofrmed into a
+/// When extra metadata are provided, a [`DerivableKey`] can be transformed into a
 /// [`DescriptorKey`]: the trait [`IntoDescriptorKey`] is automatically implemented
 /// for `(DerivableKey, DerivationPath)` and
 /// `(DerivableKey, KeySource, DerivationPath)` tuples.
@@ -387,7 +400,7 @@ impl<Ctx: ScriptContext> From<bip32::ExtendedPrivKey> for ExtendedKey<Ctx> {
 ///             network: self.network,
 ///             depth: 0,
 ///             parent_fingerprint: bip32::Fingerprint::default(),
-///             private_key: self.key_data,
+///             private_key: self.key_data.inner,
 ///             chain_code: bip32::ChainCode::from(self.chain_code.as_ref()),
 ///             child_number: bip32::ChildNumber::Normal { index: 0 },
 ///         };
@@ -419,7 +432,7 @@ impl<Ctx: ScriptContext> From<bip32::ExtendedPrivKey> for ExtendedKey<Ctx> {
 ///             network: bitcoin::Network::Bitcoin, // pick an arbitrary network here
 ///             depth: 0,
 ///             parent_fingerprint: bip32::Fingerprint::default(),
-///             private_key: self.key_data,
+///             private_key: self.key_data.inner,
 ///             chain_code: bip32::ChainCode::from(self.chain_code.as_ref()),
 ///             child_number: bip32::ChildNumber::Normal { index: 0 },
 ///         };
@@ -547,6 +560,16 @@ impl<K, Ctx: ScriptContext> Deref for GeneratedKey<K, Ctx> {
     }
 }
 
+impl<K: Clone, Ctx: ScriptContext> Clone for GeneratedKey<K, Ctx> {
+    fn clone(&self) -> GeneratedKey<K, Ctx> {
+        GeneratedKey {
+            key: self.key.clone(),
+            valid_networks: self.valid_networks.clone(),
+            phantom: self.phantom,
+        }
+    }
+}
+
 // Make generated "derivable" keys themselves "derivable". Also make sure they are assigned the
 // right `valid_networks`.
 impl<Ctx, K> DerivableKey<Ctx> for GeneratedKey<K, Ctx>
@@ -687,11 +710,11 @@ impl<Ctx: ScriptContext> GeneratableKey<Ctx> for PrivateKey {
         entropy: Self::Entropy,
     ) -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
         // pick a arbitrary network here, but say that we support all of them
-        let key = secp256k1::SecretKey::from_slice(&entropy)?;
+        let inner = secp256k1::SecretKey::from_slice(&entropy)?;
         let private_key = PrivateKey {
             compressed: options.compressed,
             network: Network::Bitcoin,
-            key,
+            inner,
         };
 
         Ok(GeneratedKey::new(private_key, any_network()))
@@ -748,7 +771,7 @@ pub fn make_pk<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
     let (key, key_map, valid_networks) = descriptor_key.into_descriptor_key()?.extract(secp)?;
     let minisc = Miniscript::from_ast(Terminal::PkK(key))?;
 
-    minisc.check_minsicript()?;
+    minisc.check_miniscript()?;
 
     Ok((minisc, key_map, valid_networks))
 }
@@ -762,22 +785,27 @@ pub fn make_pkh<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
     let (key, key_map, valid_networks) = descriptor_key.into_descriptor_key()?.extract(secp)?;
     let minisc = Miniscript::from_ast(Terminal::PkH(key))?;
 
-    minisc.check_minsicript()?;
+    minisc.check_miniscript()?;
 
     Ok((minisc, key_map, valid_networks))
 }
 
 // Used internally by `bdk::fragment!` to build `multi()` fragments
 #[doc(hidden)]
-pub fn make_multi<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
+pub fn make_multi<
+    Pk: IntoDescriptorKey<Ctx>,
+    Ctx: ScriptContext,
+    V: Fn(usize, Vec<DescriptorPublicKey>) -> Terminal<DescriptorPublicKey, Ctx>,
+>(
     thresh: usize,
+    variant: V,
     pks: Vec<Pk>,
     secp: &SecpCtx,
 ) -> Result<(Miniscript<DescriptorPublicKey, Ctx>, KeyMap, ValidNetworks), DescriptorError> {
     let (pks, key_map, valid_networks) = expand_multi_keys(pks, secp)?;
-    let minisc = Miniscript::from_ast(Terminal::Multi(thresh, pks))?;
+    let minisc = Miniscript::from_ast(variant(thresh, pks))?;
 
-    minisc.check_minsicript()?;
+    minisc.check_miniscript()?;
 
     Ok((minisc, key_map, valid_networks))
 }
@@ -830,7 +858,17 @@ impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for DescriptorPublicKey {
 impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for PublicKey {
     fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
         DescriptorPublicKey::SinglePub(DescriptorSinglePub {
-            key: self,
+            key: SinglePubKey::FullKey(self),
+            origin: None,
+        })
+        .into_descriptor_key()
+    }
+}
+
+impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for XOnlyPublicKey {
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        DescriptorPublicKey::SinglePub(DescriptorSinglePub {
+            key: SinglePubKey::XOnly(self),
             origin: None,
         })
         .into_descriptor_key()
@@ -931,4 +969,19 @@ pub mod test {
             "L2wTu6hQrnDMiFNWA5na6jB12ErGQqtXwqpSL7aWquJaZG8Ai3ch"
         );
     }
+
+    #[cfg(feature = "keys-bip39")]
+    #[test]
+    fn test_keys_wif_network_bip39() {
+        let xkey: ExtendedKey = bip39::Mnemonic::parse_in(
+            bip39::Language::English,
+            "jelly crash boy whisper mouse ecology tuna soccer memory million news short",
+        )
+        .unwrap()
+        .into_extended_key()
+        .unwrap();
+        let xprv = xkey.into_xprv(Network::Testnet).unwrap();
+
+        assert_eq!(xprv.network, Network::Testnet);
+    }
 }

src/lib.rs

@@ -14,6 +14,10 @@
 // only enables the `doc_cfg` feature when
 // the `docsrs` configuration attribute is defined
 #![cfg_attr(docsrs, feature(doc_cfg))]
+#![cfg_attr(
+    docsrs,
+    doc(html_logo_url = "https://github.com/bitcoindevkit/bdk/raw/master/static/bdk.png")
+)]
 
 //! A modern, lightweight, descriptor-based wallet library written in Rust.
 //!
@@ -39,32 +43,29 @@
 //! blockchain data and an [electrum](https://docs.rs/electrum-client/) blockchain client to
 //! interact with the bitcoin P2P network.
 //!
-//! ```toml
-//! bdk = "0.14.0"
-//! ```
+//! # Examples
 #![cfg_attr(
     feature = "electrum",
     doc = r##"
 ## Sync the balance of a descriptor
 
-### Example
 ```no_run
-use bdk::Wallet;
+use bdk::{Wallet, SyncOptions};
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::{noop_progress, ElectrumBlockchain};
+use bdk::blockchain::ElectrumBlockchain;
 use bdk::electrum_client::Client;
 
 fn main() -> Result<(), bdk::Error> {
     let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let blockchain = ElectrumBlockchain::from(client);
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
-        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(noop_progress(), None)?;
+    wallet.sync(&blockchain, SyncOptions::default())?;
 
     println!("Descriptor balance: {} SAT", wallet.get_balance()?);
 
@@ -83,7 +84,7 @@ fn main() -> Result<(), bdk::Error> {
 //! use bdk::wallet::AddressIndex::New;
 //!
 //! fn main() -> Result<(), bdk::Error> {
-//! let wallet = Wallet::new_offline(
+//! let wallet = Wallet::new(
 //!         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
 //!         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
 //!         bitcoin::Network::Testnet,
@@ -102,11 +103,10 @@ fn main() -> Result<(), bdk::Error> {
     doc = r##"
 ## Create a transaction
 
-### Example
 ```no_run
-use bdk::{FeeRate, Wallet};
+use bdk::{FeeRate, Wallet, SyncOptions};
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::{noop_progress, ElectrumBlockchain};
+use bdk::blockchain::ElectrumBlockchain;
 use bdk::electrum_client::Client;
 
 use bitcoin::consensus::serialize;
@@ -119,10 +119,10 @@ fn main() -> Result<(), bdk::Error> {
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
-        ElectrumBlockchain::from(client)
     )?;
+    let blockchain = ElectrumBlockchain::from(client);
 
-    wallet.sync(noop_progress(), None)?;
+    wallet.sync(&blockchain, SyncOptions::default())?;
 
     let send_to = wallet.get_address(New)?;
     let (psbt, details) = {
@@ -146,7 +146,6 @@ fn main() -> Result<(), bdk::Error> {
 //!
 //! ## Sign a transaction
 //!
-//! ### Example
 //! ```no_run
 //! use std::str::FromStr;
 //!
@@ -156,7 +155,7 @@ fn main() -> Result<(), bdk::Error> {
 //! use bdk::database::MemoryDatabase;
 //!
 //! fn main() -> Result<(), bdk::Error> {
-//!     let wallet = Wallet::new_offline(
+//!     let wallet = Wallet::new(
 //!         "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
 //!         Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
 //!         bitcoin::Network::Testnet,
@@ -188,7 +187,7 @@ fn main() -> Result<(), bdk::Error> {
 //! * `async-interface`: async functions in bdk traits
 //! * `keys-bip39`: [BIP-39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) mnemonic codes for generating deterministic keys
 //!
-//! ## Internal features
+//! # Internal features
 //!
 //! These features do not expose any new API, but influence internal implementation aspects of
 //! BDK.
@@ -247,6 +246,14 @@ pub extern crate sled;
 #[cfg(feature = "sqlite")]
 pub extern crate rusqlite;
 
+// We should consider putting this under a feature flag but we need the macro in doctests so we need
+// to wait until https://github.com/rust-lang/rust/issues/67295 is fixed.
+//
+// Stuff in here is too rough to document atm
+#[doc(hidden)]
+#[macro_use]
+pub mod testutils;
+
 #[allow(unused_imports)]
 #[macro_use]
 pub(crate) mod error;
@@ -268,16 +275,10 @@ pub use wallet::address_validator;
 pub use wallet::signer;
 pub use wallet::signer::SignOptions;
 pub use wallet::tx_builder::TxBuilder;
+pub use wallet::SyncOptions;
 pub use wallet::Wallet;
 
 /// Get the version of BDK at runtime
 pub fn version() -> &'static str {
     env!("CARGO_PKG_VERSION", "unknown")
 }
-
-// We should consider putting this under a feature flag but we need the macro in doctets so we need
-// to wait until https://github.com/rust-lang/rust/issues/67295 is fixed.
-//
-// Stuff in here is too rough to document atm
-#[doc(hidden)]
-pub mod testutils;

src/psbt/mod.rs

@@ -19,7 +19,7 @@ pub trait PsbtUtils {
 impl PsbtUtils for Psbt {
     #[allow(clippy::all)] // We want to allow `manual_map` but it is too new.
     fn get_utxo_for(&self, input_index: usize) -> Option<TxOut> {
-        let tx = &self.global.unsigned_tx;
+        let tx = &self.unsigned_tx;
 
         if input_index >= tx.input.len() {
             return None;
@@ -93,7 +93,7 @@ mod test {
         let mut builder = wallet.build_tx();
         builder.add_recipient(send_to.script_pubkey(), 10_000);
         let (mut psbt, _) = builder.finish().unwrap();
-        psbt.global.unsigned_tx.input.push(TxIn::default());
+        psbt.unsigned_tx.input.push(TxIn::default());
         let options = SignOptions {
             trust_witness_utxo: true,
             ..Default::default()
@@ -112,10 +112,9 @@ mod test {
 
         // add a finalized input
         psbt.inputs.push(psbt_bip.inputs[0].clone());
-        psbt.global
-            .unsigned_tx
+        psbt.unsigned_tx
             .input
-            .push(psbt_bip.global.unsigned_tx.input[0].clone());
+            .push(psbt_bip.unsigned_tx.input[0].clone());
 
         let _ = wallet.sign(&mut psbt, SignOptions::default()).unwrap();
     }

src/testutils/configurable_blockchain_tests.rs

@@ -0,0 +1,257 @@
+use bitcoin::Network;
+
+use crate::{
+    blockchain::ConfigurableBlockchain, database::MemoryDatabase, testutils, wallet::AddressIndex,
+    Wallet,
+};
+
+use super::blockchain_tests::TestClient;
+
+/// Trait for testing [`ConfigurableBlockchain`] implementations.
+pub trait ConfigurableBlockchainTester<B: ConfigurableBlockchain>: Sized {
+    /// Blockchain name for logging.
+    const BLOCKCHAIN_NAME: &'static str;
+
+    /// Generates a blockchain config with a given stop_gap.
+    ///
+    /// If this returns [`Option::None`], then the associated tests will not run.
+    fn config_with_stop_gap(
+        &self,
+        _test_client: &mut TestClient,
+        _stop_gap: usize,
+    ) -> Option<B::Config> {
+        None
+    }
+
+    /// Runs all avaliable tests.
+    fn run(&self) {
+        let test_client = &mut TestClient::default();
+
+        if self.config_with_stop_gap(test_client, 0).is_some() {
+            test_wallet_sync_with_stop_gaps(test_client, self);
+            test_wallet_sync_fulfills_missing_script_cache(test_client, self);
+            test_wallet_sync_self_transfer_tx(test_client, self);
+        } else {
+            println!(
+                "{}: Skipped tests requiring config_with_stop_gap.",
+                Self::BLOCKCHAIN_NAME
+            );
+        }
+    }
+}
+
+/// Test whether blockchain implementation syncs with expected behaviour given different `stop_gap`
+/// parameters.
+///
+/// For each test vector:
+/// * Fill wallet's derived addresses with balances (as specified by test vector).
+///    * [0..addrs_before]          => 1000sats for each address
+///    * [addrs_before..actual_gap] => empty addresses
+///    * [actual_gap..addrs_after]  => 1000sats for each address
+/// * Then, perform wallet sync and obtain wallet balance
+/// * Check balance is within expected range (we can compare `stop_gap` and `actual_gap` to
+///    determine this).
+fn test_wallet_sync_with_stop_gaps<T, B>(test_client: &mut TestClient, tester: &T)
+where
+    T: ConfigurableBlockchainTester<B>,
+    B: ConfigurableBlockchain,
+{
+    // Generates wallet descriptor
+    let descriptor_of_account = |account_index: usize| -> String {
+        format!("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/{account_index}/*)")
+    };
+
+    // Amount (in satoshis) provided to a single address (which expects to have a balance)
+    const AMOUNT_PER_TX: u64 = 1000;
+
+    // [stop_gap, actual_gap, addrs_before, addrs_after]
+    //
+    // [0]     stop_gap: Passed to [`ElectrumBlockchainConfig`]
+    // [1]   actual_gap: Range size of address indexes without a balance
+    // [2] addrs_before: Range size of address indexes (before gap) which contains a balance
+    // [3]  addrs_after: Range size of address indexes (after gap) which contains a balance
+    let test_vectors: Vec<[u64; 4]> = vec![
+        [0, 0, 0, 5],
+        [0, 0, 5, 5],
+        [0, 1, 5, 5],
+        [0, 2, 5, 5],
+        [1, 0, 5, 5],
+        [1, 1, 5, 5],
+        [1, 2, 5, 5],
+        [2, 1, 5, 5],
+        [2, 2, 5, 5],
+        [2, 3, 5, 5],
+    ];
+
+    for (account_index, vector) in test_vectors.into_iter().enumerate() {
+        let [stop_gap, actual_gap, addrs_before, addrs_after] = vector;
+        let descriptor = descriptor_of_account(account_index);
+
+        let blockchain = B::from_config(
+            &tester
+                .config_with_stop_gap(test_client, stop_gap as _)
+                .unwrap(),
+        )
+        .unwrap();
+
+        let wallet =
+            Wallet::new(&descriptor, None, Network::Regtest, MemoryDatabase::new()).unwrap();
+
+        // fill server-side with txs to specified address indexes
+        // return the max balance of the wallet (also the actual balance)
+        let max_balance = (0..addrs_before)
+            .chain(addrs_before + actual_gap..addrs_before + actual_gap + addrs_after)
+            .fold(0_u64, |sum, i| {
+                let address = wallet.get_address(AddressIndex::Peek(i as _)).unwrap();
+                test_client.receive(testutils! {
+                    @tx ( (@addr address.address) => AMOUNT_PER_TX )
+                });
+                sum + AMOUNT_PER_TX
+            });
+
+        // minimum allowed balance of wallet (based on stop gap)
+        let min_balance = if actual_gap > stop_gap {
+            addrs_before * AMOUNT_PER_TX
+        } else {
+            max_balance
+        };
+        let details = format!(
+            "test_vector: [stop_gap: {}, actual_gap: {}, addrs_before: {}, addrs_after: {}]",
+            stop_gap, actual_gap, addrs_before, addrs_after,
+        );
+        println!("{}", details);
+
+        // perform wallet sync
+        wallet.sync(&blockchain, Default::default()).unwrap();
+
+        let wallet_balance = wallet.get_balance().unwrap().get_total();
+        println!(
+            "max: {}, min: {}, actual: {}",
+            max_balance, min_balance, wallet_balance
+        );
+
+        assert!(
+            wallet_balance <= max_balance,
+            "wallet balance is greater than received amount: {}",
+            details
+        );
+        assert!(
+            wallet_balance >= min_balance,
+            "wallet balance is smaller than expected: {}",
+            details
+        );
+
+        // generate block to confirm new transactions
+        test_client.generate(1, None);
+    }
+}
+
+/// With a `stop_gap` of x and every x addresses having a balance of 1000 (for y addresses),
+/// we expect `Wallet::sync` to correctly self-cache addresses, so that the resulting balance,
+/// after sync, should be y * 1000.
+fn test_wallet_sync_fulfills_missing_script_cache<T, B>(test_client: &mut TestClient, tester: &T)
+where
+    T: ConfigurableBlockchainTester<B>,
+    B: ConfigurableBlockchain,
+{
+    // wallet descriptor
+    let descriptor = "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/200/*)";
+
+    // amount in sats per tx
+    const AMOUNT_PER_TX: u64 = 1000;
+
+    // addr constants
+    const ADDR_COUNT: usize = 6;
+    const ADDR_GAP: usize = 60;
+
+    let blockchain =
+        B::from_config(&tester.config_with_stop_gap(test_client, ADDR_GAP).unwrap()).unwrap();
+
+    let wallet = Wallet::new(descriptor, None, Network::Regtest, MemoryDatabase::new()).unwrap();
+
+    let expected_balance = (0..ADDR_COUNT).fold(0_u64, |sum, i| {
+        let addr_i = i * ADDR_GAP;
+        let address = wallet.get_address(AddressIndex::Peek(addr_i as _)).unwrap();
+
+        println!(
+            "tx: {} sats => [{}] {}",
+            AMOUNT_PER_TX,
+            addr_i,
+            address.to_string()
+        );
+
+        test_client.receive(testutils! {
+            @tx ( (@addr address.address) => AMOUNT_PER_TX )
+        });
+        test_client.generate(1, None);
+
+        sum + AMOUNT_PER_TX
+    });
+    println!("expected balance: {}, syncing...", expected_balance);
+
+    // perform sync
+    wallet.sync(&blockchain, Default::default()).unwrap();
+    println!("sync done!");
+
+    let balance = wallet.get_balance().unwrap().get_total();
+    assert_eq!(balance, expected_balance);
+}
+
+/// Given a `stop_gap`, a wallet with a 2 transactions, one sending to `scriptPubKey` at derivation
+/// index of `stop_gap`, and the other spending from the same `scriptPubKey` into another
+/// `scriptPubKey` at derivation index of `stop_gap * 2`, we expect `Wallet::sync` to perform
+/// correctly, so that we detect the total balance.
+fn test_wallet_sync_self_transfer_tx<T, B>(test_client: &mut TestClient, tester: &T)
+where
+    T: ConfigurableBlockchainTester<B>,
+    B: ConfigurableBlockchain,
+{
+    const TRANSFER_AMOUNT: u64 = 10_000;
+    const STOP_GAP: usize = 75;
+
+    let descriptor = "wpkh(tprv8i8F4EhYDMquzqiecEX8SKYMXqfmmb1Sm7deoA1Hokxzn281XgTkwsd6gL8aJevLE4aJugfVf9MKMvrcRvPawGMenqMBA3bRRfp4s1V7Eg3/*)";
+
+    let blockchain =
+        B::from_config(&tester.config_with_stop_gap(test_client, STOP_GAP).unwrap()).unwrap();
+
+    let wallet = Wallet::new(descriptor, None, Network::Regtest, MemoryDatabase::new()).unwrap();
+
+    let address1 = wallet
+        .get_address(AddressIndex::Peek(STOP_GAP as _))
+        .unwrap();
+    let address2 = wallet
+        .get_address(AddressIndex::Peek((STOP_GAP * 2) as _))
+        .unwrap();
+
+    test_client.receive(testutils! {
+        @tx ( (@addr address1.address) => TRANSFER_AMOUNT )
+    });
+    test_client.generate(1, None);
+
+    wallet.sync(&blockchain, Default::default()).unwrap();
+
+    let mut builder = wallet.build_tx();
+    builder.add_recipient(address2.script_pubkey(), TRANSFER_AMOUNT / 2);
+    let (mut psbt, details) = builder.finish().unwrap();
+    assert!(wallet.sign(&mut psbt, Default::default()).unwrap());
+    blockchain.broadcast(&psbt.extract_tx()).unwrap();
+
+    test_client.generate(1, None);
+
+    // obtain what is expected
+    let fee = details.fee.unwrap();
+    let expected_balance = TRANSFER_AMOUNT - fee;
+    println!("fee={}, expected_balance={}", fee, expected_balance);
+
+    // actually test the wallet
+    wallet.sync(&blockchain, Default::default()).unwrap();
+    let balance = wallet.get_balance().unwrap().get_total();
+    assert_eq!(balance, expected_balance);
+
+    // now try with a fresh wallet
+    let fresh_wallet =
+        Wallet::new(descriptor, None, Network::Regtest, MemoryDatabase::new()).unwrap();
+    fresh_wallet.sync(&blockchain, Default::default()).unwrap();
+    let fresh_balance = fresh_wallet.get_balance().unwrap().get_total();
+    assert_eq!(fresh_balance, expected_balance);
+}

src/testutils/mod.rs

@@ -14,11 +14,37 @@
 #[cfg(feature = "test-blockchains")]
 pub mod blockchain_tests;
 
-use bitcoin::secp256k1::{Secp256k1, Verification};
-use bitcoin::{Address, PublicKey};
+#[cfg(test)]
+#[cfg(feature = "test-blockchains")]
+pub mod configurable_blockchain_tests;
 
-use miniscript::descriptor::DescriptorPublicKey;
-use miniscript::{Descriptor, MiniscriptKey, TranslatePk};
+use bitcoin::{Address, Txid};
+
+#[derive(Clone, Debug)]
+pub struct TestIncomingInput {
+    pub txid: Txid,
+    pub vout: u32,
+    pub sequence: Option<u32>,
+}
+
+impl TestIncomingInput {
+    pub fn new(txid: Txid, vout: u32, sequence: Option<u32>) -> Self {
+        Self {
+            txid,
+            vout,
+            sequence,
+        }
+    }
+
+    #[cfg(feature = "test-blockchains")]
+    pub fn into_raw_tx_input(self) -> bitcoincore_rpc::json::CreateRawTransactionInput {
+        bitcoincore_rpc::json::CreateRawTransactionInput {
+            txid: self.txid,
+            vout: self.vout,
+            sequence: self.sequence,
+        }
+    }
+}
 
 #[derive(Clone, Debug)]
 pub struct TestIncomingOutput {
@@ -37,6 +63,7 @@ impl TestIncomingOutput {
 
 #[derive(Clone, Debug)]
 pub struct TestIncomingTx {
+    pub input: Vec<TestIncomingInput>,
     pub output: Vec<TestIncomingOutput>,
     pub min_confirmations: Option<u64>,
     pub locktime: Option<i64>,
@@ -45,12 +72,14 @@ pub struct TestIncomingTx {
 
 impl TestIncomingTx {
     pub fn new(
+        input: Vec<TestIncomingInput>,
         output: Vec<TestIncomingOutput>,
         min_confirmations: Option<u64>,
         locktime: Option<i64>,
         replaceable: Option<bool>,
     ) -> Self {
         Self {
+            input,
             output,
             min_confirmations,
             locktime,
@@ -58,44 +87,15 @@ impl TestIncomingTx {
         }
     }
 
+    pub fn add_input(&mut self, input: TestIncomingInput) {
+        self.input.push(input);
+    }
+
     pub fn add_output(&mut self, output: TestIncomingOutput) {
         self.output.push(output);
     }
 }
 
-#[doc(hidden)]
-pub trait TranslateDescriptor {
-    // derive and translate a `Descriptor<DescriptorPublicKey>` into a `Descriptor<PublicKey>`
-    fn derive_translated<C: Verification>(
-        &self,
-        secp: &Secp256k1<C>,
-        index: u32,
-    ) -> Descriptor<PublicKey>;
-}
-
-impl TranslateDescriptor for Descriptor<DescriptorPublicKey> {
-    fn derive_translated<C: Verification>(
-        &self,
-        secp: &Secp256k1<C>,
-        index: u32,
-    ) -> Descriptor<PublicKey> {
-        let translate = |key: &DescriptorPublicKey| -> PublicKey {
-            match key {
-                DescriptorPublicKey::XPub(xpub) => {
-                    xpub.xkey
-                        .derive_pub(secp, &xpub.derivation_path)
-                        .expect("hardened derivation steps")
-                        .public_key
-                }
-                DescriptorPublicKey::SinglePub(key) => key.key,
-            }
-        };
-
-        self.derive(index)
-            .translate_pk_infallible(|pk| translate(pk), |pkh| translate(pkh).to_pubkeyhash())
-    }
-}
-
 #[doc(hidden)]
 #[macro_export]
 macro_rules! testutils {
@@ -103,36 +103,41 @@ macro_rules! testutils {
         use $crate::bitcoin::secp256k1::Secp256k1;
         use $crate::miniscript::descriptor::{Descriptor, DescriptorPublicKey, DescriptorTrait};
 
-        use $crate::testutils::TranslateDescriptor;
+        use $crate::descriptor::AsDerived;
 
         let secp = Secp256k1::new();
 
         let parsed = Descriptor::<DescriptorPublicKey>::parse_descriptor(&secp, &$descriptors.0).expect("Failed to parse descriptor in `testutils!(@external)`").0;
-        parsed.derive_translated(&secp, $child).address(bitcoin::Network::Regtest).expect("No address form")
+        parsed.as_derived($child, &secp).address(bitcoin::Network::Regtest).expect("No address form")
     });
     ( @internal $descriptors:expr, $child:expr ) => ({
         use $crate::bitcoin::secp256k1::Secp256k1;
         use $crate::miniscript::descriptor::{Descriptor, DescriptorPublicKey, DescriptorTrait};
 
-        use $crate::testutils::TranslateDescriptor;
+        use $crate::descriptor::AsDerived;
 
         let secp = Secp256k1::new();
 
         let parsed = Descriptor::<DescriptorPublicKey>::parse_descriptor(&secp, &$descriptors.1.expect("Missing internal descriptor")).expect("Failed to parse descriptor in `testutils!(@internal)`").0;
-        parsed.derive_translated(&secp, $child).address($crate::bitcoin::Network::Regtest).expect("No address form")
+        parsed.as_derived($child, &secp).address($crate::bitcoin::Network::Regtest).expect("No address form")
     });
     ( @e $descriptors:expr, $child:expr ) => ({ testutils!(@external $descriptors, $child) });
     ( @i $descriptors:expr, $child:expr ) => ({ testutils!(@internal $descriptors, $child) });
+    ( @addr $addr:expr ) => ({ $addr });
 
-    ( @tx ( $( ( $( $addr:tt )* ) => $amount:expr ),+ ) $( ( @locktime $locktime:expr ) )? $( ( @confirmations $confirmations:expr ) )? $( ( @replaceable $replaceable:expr ) )? ) => ({
+    ( @tx ( $( ( $( $addr:tt )* ) => $amount:expr ),+ ) $( ( @inputs $( ($txid:expr, $vout:expr) ),+ ) )? $( ( @locktime $locktime:expr ) )? $( ( @confirmations $confirmations:expr ) )? $( ( @replaceable $replaceable:expr ) )? ) => ({
         let outs = vec![$( $crate::testutils::TestIncomingOutput::new($amount, testutils!( $($addr)* ))),+];
+        let _ins: Vec<$crate::testutils::TestIncomingInput> = vec![];
+        $(
+            let _ins = vec![$( $crate::testutils::TestIncomingInput { txid: $txid, vout: $vout, sequence: None }),+];
+        )?
 
         let locktime = None::<i64>$(.or(Some($locktime)))?;
 
         let min_confirmations = None::<u64>$(.or(Some($confirmations)))?;
         let replaceable = None::<bool>$(.or(Some($replaceable)))?;
 
-        $crate::testutils::TestIncomingTx::new(outs, min_confirmations, locktime, replaceable)
+        $crate::testutils::TestIncomingTx::new(_ins, outs, min_confirmations, locktime, replaceable)
     });
 
     ( @literal $key:expr ) => ({

src/types.rs

@@ -51,14 +51,44 @@ impl AsRef<[u8]> for KeychainKind {
 pub struct FeeRate(f32);
 
 impl FeeRate {
+    /// Create a new instance checking the value provided
+    ///
+    /// ## Panics
+    ///
+    /// Panics if the value is not [normal](https://doc.rust-lang.org/std/primitive.f32.html#method.is_normal) (except if it's a positive zero) or negative.
+    fn new_checked(value: f32) -> Self {
+        assert!(value.is_normal() || value == 0.0);
+        assert!(value.is_sign_positive());
+
+        FeeRate(value)
+    }
+
+    /// Create a new instance of [`FeeRate`] given a float fee rate in sats/kwu
+    pub fn from_sat_per_kwu(sat_per_kwu: f32) -> Self {
+        FeeRate::new_checked(sat_per_kwu / 250.0_f32)
+    }
+
+    /// Create a new instance of [`FeeRate`] given a float fee rate in sats/kvb
+    pub fn from_sat_per_kvb(sat_per_kvb: f32) -> Self {
+        FeeRate::new_checked(sat_per_kvb / 1000.0_f32)
+    }
+
     /// Create a new instance of [`FeeRate`] given a float fee rate in btc/kvbytes
+    ///
+    /// ## Panics
+    ///
+    /// Panics if the value is not [normal](https://doc.rust-lang.org/std/primitive.f32.html#method.is_normal) (except if it's a positive zero) or negative.
     pub fn from_btc_per_kvb(btc_per_kvb: f32) -> Self {
-        FeeRate(btc_per_kvb * 1e5)
+        FeeRate::new_checked(btc_per_kvb * 1e5)
     }
 
     /// Create a new instance of [`FeeRate`] given a float fee rate in satoshi/vbyte
-    pub const fn from_sat_per_vb(sat_per_vb: f32) -> Self {
-        FeeRate(sat_per_vb)
+    ///
+    /// ## Panics
+    ///
+    /// Panics if the value is not [normal](https://doc.rust-lang.org/std/primitive.f32.html#method.is_normal) (except if it's a positive zero) or negative.
+    pub fn from_sat_per_vb(sat_per_vb: f32) -> Self {
+        FeeRate::new_checked(sat_per_vb)
     }
 
     /// Create a new [`FeeRate`] with the default min relay fee value
@@ -78,7 +108,7 @@ impl FeeRate {
     }
 
     /// Return the value as satoshi/vbyte
-    pub fn as_sat_vb(&self) -> f32 {
+    pub fn as_sat_per_vb(&self) -> f32 {
         self.0
     }
 
@@ -89,7 +119,7 @@ impl FeeRate {
 
     /// Calculate absolute fee in Satoshis using size in virtual bytes.
     pub fn fee_vb(&self, vbytes: usize) -> u64 {
-        (self.as_sat_vb() * vbytes as f32).ceil() as u64
+        (self.as_sat_per_vb() * vbytes as f32).ceil() as u64
     }
 }
 
@@ -131,6 +161,8 @@ pub struct LocalUtxo {
     pub txout: TxOut,
     /// Type of keychain
     pub keychain: KeychainKind,
+    /// Whether this UTXO is spent or not
+    pub is_spent: bool,
 }
 
 /// A [`Utxo`] with its `satisfaction_weight`.
@@ -200,10 +232,12 @@ pub struct TransactionDetails {
     pub txid: Txid,
 
     /// Received value (sats)
+    /// Sum of owned outputs of this transaction.
     pub received: u64,
     /// Sent value (sats)
+    /// Sum of owned inputs of this transaction.
     pub sent: u64,
-    /// Fee value (sats) if available.
+    /// Fee value (sats) if confirmed.
     /// The availability of the fee depends on the backend. It's never `None` with an Electrum
     /// Server backend, but it could be `None` with a Bitcoin RPC node without txindex that receive
     /// funds while offline.
@@ -211,15 +245,6 @@ pub struct TransactionDetails {
     /// If the transaction is confirmed, contains height and timestamp of the block containing the
     /// transaction, unconfirmed transaction contains `None`.
     pub confirmation_time: Option<BlockTime>,
-    /// Whether the tx has been verified against the consensus rules
-    ///
-    /// Confirmed txs are considered "verified" by default, while unconfirmed txs are checked to
-    /// ensure an unstrusted [`Blockchain`](crate::blockchain::Blockchain) backend can't trick the
-    /// wallet into using an invalid tx as an RBF template.
-    ///
-    /// The check is only perfomed when the `verify` feature is enabled.
-    #[serde(default = "bool::default")] // default to `false` if not specified
-    pub verified: bool,
 }
 
 /// Block height and timestamp of a block
@@ -247,13 +272,130 @@ impl BlockTime {
     }
 }
 
+/// Balance differentiated in various categories
+#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone, Default)]
+pub struct Balance {
+    /// All coinbase outputs not yet matured
+    pub immature: u64,
+    /// Unconfirmed UTXOs generated by a wallet tx
+    pub trusted_pending: u64,
+    /// Unconfirmed UTXOs received from an external wallet
+    pub untrusted_pending: u64,
+    /// Confirmed and immediately spendable balance
+    pub confirmed: u64,
+}
+
+impl Balance {
+    /// Get sum of trusted_pending and confirmed coins
+    pub fn get_spendable(&self) -> u64 {
+        self.confirmed + self.trusted_pending
+    }
+
+    /// Get the whole balance visible to the wallet
+    pub fn get_total(&self) -> u64 {
+        self.confirmed + self.trusted_pending + self.untrusted_pending + self.immature
+    }
+}
+
+impl std::fmt::Display for Balance {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(
+            f,
+            "{{ immature: {}, trusted_pending: {}, untrusted_pending: {}, confirmed: {} }}",
+            self.immature, self.trusted_pending, self.untrusted_pending, self.confirmed
+        )
+    }
+}
+
+impl std::ops::Add for Balance {
+    type Output = Self;
+
+    fn add(self, other: Self) -> Self {
+        Self {
+            immature: self.immature + other.immature,
+            trusted_pending: self.trusted_pending + other.trusted_pending,
+            untrusted_pending: self.untrusted_pending + other.untrusted_pending,
+            confirmed: self.confirmed + other.confirmed,
+        }
+    }
+}
+
+impl std::iter::Sum for Balance {
+    fn sum<I: Iterator<Item = Self>>(iter: I) -> Self {
+        iter.fold(
+            Balance {
+                ..Default::default()
+            },
+            |a, b| a + b,
+        )
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
     #[test]
     fn can_store_feerate_in_const() {
-        const _MY_RATE: FeeRate = FeeRate::from_sat_per_vb(10.0);
         const _MIN_RELAY: FeeRate = FeeRate::default_min_relay_fee();
     }
+
+    #[test]
+    #[should_panic]
+    fn test_invalid_feerate_neg_zero() {
+        let _ = FeeRate::from_sat_per_vb(-0.0);
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_invalid_feerate_neg_value() {
+        let _ = FeeRate::from_sat_per_vb(-5.0);
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_invalid_feerate_nan() {
+        let _ = FeeRate::from_sat_per_vb(f32::NAN);
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_invalid_feerate_inf() {
+        let _ = FeeRate::from_sat_per_vb(f32::INFINITY);
+    }
+
+    #[test]
+    fn test_valid_feerate_pos_zero() {
+        let _ = FeeRate::from_sat_per_vb(0.0);
+    }
+
+    #[test]
+    fn test_fee_from_btc_per_kvb() {
+        let fee = FeeRate::from_btc_per_kvb(1e-5);
+        assert!((fee.as_sat_per_vb() - 1.0).abs() < f32::EPSILON);
+    }
+
+    #[test]
+    fn test_fee_from_sat_per_vbyte() {
+        let fee = FeeRate::from_sat_per_vb(1.0);
+        assert!((fee.as_sat_per_vb() - 1.0).abs() < f32::EPSILON);
+    }
+
+    #[test]
+    fn test_fee_default_min_relay_fee() {
+        let fee = FeeRate::default_min_relay_fee();
+        assert!((fee.as_sat_per_vb() - 1.0).abs() < f32::EPSILON);
+    }
+
+    #[test]
+    fn test_fee_from_sat_per_kvb() {
+        let fee = FeeRate::from_sat_per_kvb(1000.0);
+        assert!((fee.as_sat_per_vb() - 1.0).abs() < f32::EPSILON);
+    }
+
+    #[test]
+    fn test_fee_from_sat_per_kwu() {
+        let fee = FeeRate::from_sat_per_kwu(250.0);
+        assert!((fee.as_sat_per_vb() - 1.0).abs() < f32::EPSILON);
+    }
 }

src/wallet/address_validator.rs

@@ -55,7 +55,7 @@
 //! }
 //!
 //! let descriptor = "wpkh(tpubD6NzVbkrYhZ4Xferm7Pz4VnjdcDPFyjVu5K4iZXQ4pVN8Cks4pHVowTBXBKRhX64pkRyJZJN5xAKj4UDNnLPb5p2sSKXhewoYx5GbTdUFWq/*)";
-//! let mut wallet = Wallet::new_offline(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
+//! let mut wallet = Wallet::new(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
 //! wallet.add_address_validator(Arc::new(PrintAddressAndContinue));
 //!
 //! let address = wallet.get_address(New)?;
@@ -100,6 +100,7 @@ impl std::error::Error for AddressValidatorError {}
 /// validator will be propagated up to the original caller that triggered the address generation.
 ///
 /// For a usage example see [this module](crate::address_validator)'s documentation.
+#[deprecated = "AddressValidator was rarely used. Address validation can occur outside of BDK"]
 pub trait AddressValidator: Send + Sync + fmt::Debug {
     /// Validate or inspect an address
     fn validate(
@@ -120,6 +121,7 @@ mod test {
 
     #[derive(Debug)]
     struct TestValidator;
+    #[allow(deprecated)]
     impl AddressValidator for TestValidator {
         fn validate(
             &self,
@@ -135,6 +137,7 @@ mod test {
     #[should_panic(expected = "InvalidScript")]
     fn test_address_validator_external() {
         let (mut wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        #[allow(deprecated)]
         wallet.add_address_validator(Arc::new(TestValidator));
 
         wallet.get_address(New).unwrap();
@@ -144,6 +147,7 @@ mod test {
     #[should_panic(expected = "InvalidScript")]
     fn test_address_validator_internal() {
         let (mut wallet, descriptors, _) = get_funded_wallet(get_test_wpkh());
+        #[allow(deprecated)]
         wallet.add_address_validator(Arc::new(TestValidator));
 
         let addr = crate::testutils!(@external descriptors, 10);

src/wallet/export.rs

@@ -29,8 +29,8 @@
 //!     "label":"testnet"
 //! }"#;
 //!
-//! let import = WalletExport::from_str(import)?;
-//! let wallet = Wallet::new_offline(
+//! let import = FullyNodedExport::from_str(import)?;
+//! let wallet = Wallet::new(
 //!     &import.descriptor(),
 //!     import.change_descriptor().as_ref(),
 //!     Network::Testnet,
@@ -45,13 +45,13 @@
 //! # use bdk::database::*;
 //! # use bdk::wallet::export::*;
 //! # use bdk::*;
-//! let wallet = Wallet::new_offline(
+//! let wallet = Wallet::new(
 //!     "wpkh([c258d2e4/84h/1h/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe/0/*)",
 //!     Some("wpkh([c258d2e4/84h/1h/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe/1/*)"),
 //!     Network::Testnet,
 //!     MemoryDatabase::default()
 //! )?;
-//! let export = WalletExport::export_wallet(&wallet, "exported wallet", true)
+//! let export = FullyNodedExport::export_wallet(&wallet, "exported wallet", true)
 //!     .map_err(ToString::to_string)
 //!     .map_err(bdk::Error::Generic)?;
 //!
@@ -64,16 +64,21 @@ use std::str::FromStr;
 use serde::{Deserialize, Serialize};
 
 use miniscript::descriptor::{ShInner, WshInner};
-use miniscript::{Descriptor, DescriptorPublicKey, ScriptContext, Terminal};
+use miniscript::{Descriptor, ScriptContext, Terminal};
 
 use crate::database::BatchDatabase;
+use crate::types::KeychainKind;
 use crate::wallet::Wallet;
 
+/// Alias for [`FullyNodedExport`]
+#[deprecated(since = "0.18.0", note = "Please use [`FullyNodedExport`] instead")]
+pub type WalletExport = FullyNodedExport;
+
 /// Structure that contains the export of a wallet
 ///
 /// For a usage example see [this module](crate::wallet::export)'s documentation.
 #[derive(Debug, Serialize, Deserialize)]
-pub struct WalletExport {
+pub struct FullyNodedExport {
     descriptor: String,
     /// Earliest block to rescan when looking for the wallet's transactions
     pub blockheight: u32,
@@ -81,13 +86,13 @@ pub struct WalletExport {
     pub label: String,
 }
 
-impl ToString for WalletExport {
+impl ToString for FullyNodedExport {
     fn to_string(&self) -> String {
         serde_json::to_string(self).unwrap()
     }
 }
 
-impl FromStr for WalletExport {
+impl FromStr for FullyNodedExport {
     type Err = serde_json::Error;
 
     fn from_str(s: &str) -> Result<Self, Self::Err> {
@@ -96,10 +101,10 @@ impl FromStr for WalletExport {
 }
 
 fn remove_checksum(s: String) -> String {
-    s.splitn(2, '#').next().map(String::from).unwrap()
+    s.split_once('#').map(|(a, _)| String::from(a)).unwrap()
 }
 
-impl WalletExport {
+impl FullyNodedExport {
     /// Export a wallet
     ///
     /// This function returns an error if it determines that the `wallet`'s descriptor(s) are not
@@ -111,14 +116,18 @@ impl WalletExport {
     ///
     /// If the database is empty or `include_blockheight` is false, the `blockheight` field
     /// returned will be `0`.
-    pub fn export_wallet<B, D: BatchDatabase>(
-        wallet: &Wallet<B, D>,
+    pub fn export_wallet<D: BatchDatabase>(
+        wallet: &Wallet<D>,
         label: &str,
         include_blockheight: bool,
     ) -> Result<Self, &'static str> {
         let descriptor = wallet
-            .descriptor
-            .to_string_with_secret(&wallet.signers.as_key_map(wallet.secp_ctx()));
+            .get_descriptor_for_keychain(KeychainKind::External)
+            .to_string_with_secret(
+                &wallet
+                    .get_signers(KeychainKind::External)
+                    .as_key_map(wallet.secp_ctx()),
+            );
         let descriptor = remove_checksum(descriptor);
         Self::is_compatible_with_core(&descriptor)?;
 
@@ -136,18 +145,30 @@ impl WalletExport {
             }
         };
 
-        let export = WalletExport {
+        let export = FullyNodedExport {
             descriptor,
             label: label.into(),
             blockheight,
         };
 
-        let desc_to_string = |d: &Descriptor<DescriptorPublicKey>| {
-            let descriptor =
-                d.to_string_with_secret(&wallet.change_signers.as_key_map(wallet.secp_ctx()));
-            remove_checksum(descriptor)
+        let change_descriptor = match wallet
+            .public_descriptor(KeychainKind::Internal)
+            .map_err(|_| "Invalid change descriptor")?
+            .is_some()
+        {
+            false => None,
+            true => {
+                let descriptor = wallet
+                    .get_descriptor_for_keychain(KeychainKind::Internal)
+                    .to_string_with_secret(
+                        &wallet
+                            .get_signers(KeychainKind::Internal)
+                            .as_key_map(wallet.secp_ctx()),
+                    );
+                Some(remove_checksum(descriptor))
+            }
         };
-        if export.change_descriptor() != wallet.change_descriptor.as_ref().map(desc_to_string) {
+        if export.change_descriptor() != change_descriptor {
             return Err("Incompatible change descriptor");
         }
 
@@ -230,7 +251,6 @@ mod test {
                 timestamp: 12345678,
                 height: 5000,
             }),
-            verified: true,
         })
         .unwrap();
 
@@ -242,14 +262,14 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
             get_test_db(),
         )
         .unwrap();
-        let export = WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
 
         assert_eq!(export.descriptor(), descriptor);
         assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));
@@ -266,9 +286,8 @@ mod test {
 
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
 
-        let wallet =
-            Wallet::new_offline(descriptor, None, Network::Bitcoin, get_test_db()).unwrap();
-        WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let wallet = Wallet::new(descriptor, None, Network::Bitcoin, get_test_db()).unwrap();
+        FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
     }
 
     #[test]
@@ -280,14 +299,14 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/50'/0'/1/*)";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
             get_test_db(),
         )
         .unwrap();
-        WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
     }
 
     #[test]
@@ -303,14 +322,14 @@ mod test {
                                        [c98b1535/48'/0'/0'/2']tpubDCDi5W4sP6zSnzJeowy8rQDVhBdRARaPhK1axABi8V1661wEPeanpEXj4ZLAUEoikVtoWcyK26TKKJSecSfeKxwHCcRrge9k1ybuiL71z4a/1/*\
                                  ))";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Testnet,
             get_test_db(),
         )
         .unwrap();
-        let export = WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
 
         assert_eq!(export.descriptor(), descriptor);
         assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));
@@ -323,14 +342,14 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
-        let wallet = Wallet::new_offline(
+        let wallet = Wallet::new(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
             get_test_db(),
         )
         .unwrap();
-        let export = WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
 
         assert_eq!(export.to_string(), "{\"descriptor\":\"wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44\'/0\'/0\'/0/*)\",\"blockheight\":5000,\"label\":\"Test Label\"}");
     }
@@ -341,7 +360,7 @@ mod test {
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
         let import_str = "{\"descriptor\":\"wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44\'/0\'/0\'/0/*)\",\"blockheight\":5000,\"label\":\"Test Label\"}";
-        let export = WalletExport::from_str(import_str).unwrap();
+        let export = FullyNodedExport::from_str(import_str).unwrap();
 
         assert_eq!(export.descriptor(), descriptor);
         assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));

src/wallet/hardwaresigner.rs

@@ -0,0 +1,64 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! HWI Signer
+//!
+//! This module contains a simple implementation of a Custom signer for rust-hwi
+
+use bitcoin::psbt::PartiallySignedTransaction;
+use bitcoin::secp256k1::{All, Secp256k1};
+use bitcoin::util::bip32::Fingerprint;
+
+use hwi::error::Error;
+use hwi::types::{HWIChain, HWIDevice};
+use hwi::HWIClient;
+
+use crate::signer::{SignerCommon, SignerError, SignerId, TransactionSigner};
+
+#[derive(Debug)]
+/// Custom signer for Hardware Wallets
+///
+/// This ignores `sign_options` and leaves the decisions up to the hardware wallet.
+pub struct HWISigner {
+    fingerprint: Fingerprint,
+    client: HWIClient,
+}
+
+impl HWISigner {
+    /// Create a instance from the specified device and chain
+    pub fn from_device(device: &HWIDevice, chain: HWIChain) -> Result<HWISigner, Error> {
+        let client = HWIClient::get_client(device, false, chain)?;
+        Ok(HWISigner {
+            fingerprint: device.fingerprint,
+            client,
+        })
+    }
+}
+
+impl SignerCommon for HWISigner {
+    fn id(&self, _secp: &Secp256k1<All>) -> SignerId {
+        SignerId::Fingerprint(self.fingerprint)
+    }
+}
+
+/// This implementation ignores `sign_options`
+impl TransactionSigner for HWISigner {
+    fn sign_transaction(
+        &self,
+        psbt: &mut PartiallySignedTransaction,
+        _sign_options: &crate::SignOptions,
+        _secp: &crate::wallet::utils::SecpCtx,
+    ) -> Result<(), SignerError> {
+        psbt.combine(self.client.sign_tx(psbt)?.psbt)
+            .expect("Failed to combine HW signed psbt with passed PSBT");
+        Ok(())
+    }
+}

src/wallet/signer.rs

@@ -26,7 +26,7 @@
 //! # #[derive(Debug)]
 //! # struct CustomHSM;
 //! # impl CustomHSM {
-//! #     fn sign_input(&self, _psbt: &mut psbt::PartiallySignedTransaction, _input: usize) -> Result<(), SignerError> {
+//! #     fn hsm_sign_input(&self, _psbt: &mut psbt::PartiallySignedTransaction, _input: usize) -> Result<(), SignerError> {
 //! #         Ok(())
 //! #     }
 //! #     fn connect() -> Self {
@@ -47,32 +47,30 @@
 //!     }
 //! }
 //!
-//! impl Signer for CustomSigner {
-//!     fn sign(
-//!         &self,
-//!         psbt: &mut psbt::PartiallySignedTransaction,
-//!         input_index: Option<usize>,
-//!         _secp: &Secp256k1<All>,
-//!     ) -> Result<(), SignerError> {
-//!         let input_index = input_index.ok_or(SignerError::InputIndexOutOfRange)?;
-//!         self.device.sign_input(psbt, input_index)?;
-//!
-//!         Ok(())
-//!     }
-//!
+//! impl SignerCommon for CustomSigner {
 //!     fn id(&self, _secp: &Secp256k1<All>) -> SignerId {
 //!         self.device.get_id()
 //!     }
+//! }
 //!
-//!     fn sign_whole_tx(&self) -> bool {
-//!         false
+//! impl InputSigner for CustomSigner {
+//!     fn sign_input(
+//!         &self,
+//!         psbt: &mut psbt::PartiallySignedTransaction,
+//!         input_index: usize,
+//!         _sign_options: &SignOptions,
+//!         _secp: &Secp256k1<All>,
+//!     ) -> Result<(), SignerError> {
+//!         self.device.hsm_sign_input(psbt, input_index)?;
+//!
+//!         Ok(())
 //!     }
 //! }
 //!
 //! let custom_signer = CustomSigner::connect();
 //!
 //! let descriptor = "wpkh(tpubD6NzVbkrYhZ4Xferm7Pz4VnjdcDPFyjVu5K4iZXQ4pVN8Cks4pHVowTBXBKRhX64pkRyJZJN5xAKj4UDNnLPb5p2sSKXhewoYx5GbTdUFWq/*)";
-//! let mut wallet = Wallet::new_offline(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
+//! let mut wallet = Wallet::new(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
 //! wallet.add_signer(
 //!     KeychainKind::External,
 //!     SignerOrdering(200),
@@ -85,22 +83,26 @@
 use std::cmp::Ordering;
 use std::collections::BTreeMap;
 use std::fmt;
-use std::ops::Bound::Included;
+use std::ops::{Bound::Included, Deref};
 use std::sync::Arc;
 
 use bitcoin::blockdata::opcodes;
 use bitcoin::blockdata::script::Builder as ScriptBuilder;
 use bitcoin::hashes::{hash160, Hash};
-use bitcoin::secp256k1::{Message, Secp256k1};
+use bitcoin::secp256k1::Message;
 use bitcoin::util::bip32::{ChildNumber, DerivationPath, ExtendedPrivKey, Fingerprint};
-use bitcoin::util::{bip143, psbt};
-use bitcoin::{PrivateKey, Script, SigHash, SigHashType};
+use bitcoin::util::{ecdsa, psbt, schnorr, sighash, taproot};
+use bitcoin::{secp256k1, XOnlyPublicKey};
+use bitcoin::{EcdsaSighashType, PrivateKey, PublicKey, SchnorrSighashType, Script};
 
-use miniscript::descriptor::{DescriptorSecretKey, DescriptorSinglePriv, DescriptorXKey, KeyMap};
-use miniscript::{Legacy, MiniscriptKey, Segwitv0};
+use miniscript::descriptor::{
+    Descriptor, DescriptorPublicKey, DescriptorSecretKey, DescriptorSinglePriv, DescriptorXKey,
+    KeyMap, SinglePubKey,
+};
+use miniscript::{Legacy, MiniscriptKey, Segwitv0, Tap};
 
 use super::utils::SecpCtx;
-use crate::descriptor::XKeyUtils;
+use crate::descriptor::{DescriptorMeta, XKeyUtils};
 
 /// Identifier of a signer in the `SignersContainers`. Used as a key to find the right signer among
 /// multiple of them
@@ -143,7 +145,7 @@ pub enum SignerError {
     InvalidNonWitnessUtxo,
     /// The `witness_utxo` field of the transaction is required to sign this input
     MissingWitnessUtxo,
-    /// The `witness_script` field of the transaction is requied to sign this input
+    /// The `witness_script` field of the transaction is required to sign this input
     MissingWitnessScript,
     /// The fingerprint and derivation path are missing from the psbt input
     MissingHdKeypath,
@@ -153,6 +155,26 @@ pub enum SignerError {
     /// To enable signing transactions with non-standard sighashes set
     /// [`SignOptions::allow_all_sighashes`] to `true`.
     NonStandardSighash,
+    /// Invalid SIGHASH for the signing context in use
+    InvalidSighash,
+    /// Error while computing the hash to sign
+    SighashError(sighash::Error),
+    /// Error while signing using hardware wallets
+    #[cfg(feature = "hardware-signer")]
+    HWIError(hwi::error::Error),
+}
+
+#[cfg(feature = "hardware-signer")]
+impl From<hwi::error::Error> for SignerError {
+    fn from(e: hwi::error::Error) -> Self {
+        SignerError::HWIError(e)
+    }
+}
+
+impl From<sighash::Error> for SignerError {
+    fn from(e: sighash::Error) -> Self {
+        SignerError::SighashError(e)
+    }
 }
 
 impl fmt::Display for SignerError {
@@ -163,27 +185,46 @@ impl fmt::Display for SignerError {
 
 impl std::error::Error for SignerError {}
 
-/// Trait for signers
+/// Signing context
 ///
-/// This trait can be implemented to provide customized signers to the wallet. For an example see
-/// [`this module`](crate::wallet::signer)'s documentation.
-pub trait Signer: fmt::Debug + Send + Sync {
-    /// Sign a PSBT
-    ///
-    /// The `input_index` argument is only provided if the wallet doesn't declare to sign the whole
-    /// transaction in one go (see [`Signer::sign_whole_tx`]). Otherwise its value is `None` and
-    /// can be ignored.
-    fn sign(
-        &self,
-        psbt: &mut psbt::PartiallySignedTransaction,
-        input_index: Option<usize>,
-        secp: &SecpCtx,
-    ) -> Result<(), SignerError>;
+/// Used by our software signers to determine the type of signatures to make
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum SignerContext {
+    /// Legacy context
+    Legacy,
+    /// Segwit v0 context (BIP 143)
+    Segwitv0,
+    /// Taproot context (BIP 340)
+    Tap {
+        /// Whether the signer can sign for the internal key or not
+        is_internal_key: bool,
+    },
+}
 
-    /// Return whether or not the signer signs the whole transaction in one go instead of every
-    /// input individually
-    fn sign_whole_tx(&self) -> bool;
+/// Wrapper structure to pair a signer with its context
+#[derive(Debug, Clone)]
+pub struct SignerWrapper<S: Sized + fmt::Debug + Clone> {
+    signer: S,
+    ctx: SignerContext,
+}
 
+impl<S: Sized + fmt::Debug + Clone> SignerWrapper<S> {
+    /// Create a wrapped signer from a signer and a context
+    pub fn new(signer: S, ctx: SignerContext) -> Self {
+        SignerWrapper { signer, ctx }
+    }
+}
+
+impl<S: Sized + fmt::Debug + Clone> Deref for SignerWrapper<S> {
+    type Target = S;
+
+    fn deref(&self) -> &Self::Target {
+        &self.signer
+    }
+}
+
+/// Common signer methods
+pub trait SignerCommon: fmt::Debug + Send + Sync {
     /// Return the [`SignerId`] for this signer
     ///
     /// The [`SignerId`] can be used to lookup a signer in the [`Wallet`](crate::Wallet)'s signers map or to
@@ -200,14 +241,69 @@ pub trait Signer: fmt::Debug + Send + Sync {
     }
 }
 
-impl Signer for DescriptorXKey<ExtendedPrivKey> {
-    fn sign(
+/// PSBT Input signer
+///
+/// This trait can be implemented to provide custom signers to the wallet. If the signer supports signing
+/// individual inputs, this trait should be implemented and BDK will provide automatically an implementation
+/// for [`TransactionSigner`].
+pub trait InputSigner: SignerCommon {
+    /// Sign a single psbt input
+    fn sign_input(
         &self,
         psbt: &mut psbt::PartiallySignedTransaction,
-        input_index: Option<usize>,
+        input_index: usize,
+        sign_options: &SignOptions,
+        secp: &SecpCtx,
+    ) -> Result<(), SignerError>;
+}
+
+/// PSBT signer
+///
+/// This trait can be implemented when the signer can't sign inputs individually, but signs the whole transaction
+/// at once.
+pub trait TransactionSigner: SignerCommon {
+    /// Sign all the inputs of the psbt
+    fn sign_transaction(
+        &self,
+        psbt: &mut psbt::PartiallySignedTransaction,
+        sign_options: &SignOptions,
+        secp: &SecpCtx,
+    ) -> Result<(), SignerError>;
+}
+
+impl<T: InputSigner> TransactionSigner for T {
+    fn sign_transaction(
+        &self,
+        psbt: &mut psbt::PartiallySignedTransaction,
+        sign_options: &SignOptions,
+        secp: &SecpCtx,
+    ) -> Result<(), SignerError> {
+        for input_index in 0..psbt.inputs.len() {
+            self.sign_input(psbt, input_index, sign_options, secp)?;
+        }
+
+        Ok(())
+    }
+}
+
+impl SignerCommon for SignerWrapper<DescriptorXKey<ExtendedPrivKey>> {
+    fn id(&self, secp: &SecpCtx) -> SignerId {
+        SignerId::from(self.root_fingerprint(secp))
+    }
+
+    fn descriptor_secret_key(&self) -> Option<DescriptorSecretKey> {
+        Some(DescriptorSecretKey::XPrv(self.signer.clone()))
+    }
+}
+
+impl InputSigner for SignerWrapper<DescriptorXKey<ExtendedPrivKey>> {
+    fn sign_input(
+        &self,
+        psbt: &mut psbt::PartiallySignedTransaction,
+        input_index: usize,
+        sign_options: &SignOptions,
         secp: &SecpCtx,
     ) -> Result<(), SignerError> {
-        let input_index = input_index.unwrap();
         if input_index >= psbt.inputs.len() {
             return Err(SignerError::InputIndexOutOfRange);
         }
@@ -218,19 +314,23 @@ impl Signer for DescriptorXKey<ExtendedPrivKey> {
             return Ok(());
         }
 
+        let tap_key_origins = psbt.inputs[input_index]
+            .tap_key_origins
+            .iter()
+            .map(|(pk, (_, keysource))| (SinglePubKey::XOnly(*pk), keysource));
         let (public_key, full_path) = match psbt.inputs[input_index]
             .bip32_derivation
             .iter()
-            .filter_map(|(pk, &(fingerprint, ref path))| {
-                if self.matches(&(fingerprint, path.clone()), secp).is_some() {
-                    Some((pk, path))
+            .map(|(pk, keysource)| (SinglePubKey::FullKey(PublicKey::new(*pk)), keysource))
+            .chain(tap_key_origins)
+            .find_map(|(pk, keysource)| {
+                if self.matches(keysource, secp).is_some() {
+                    Some((pk, keysource.1.clone()))
                 } else {
                     None
                 }
-            })
-            .next()
-        {
-            Some((pk, full_path)) => (pk, full_path.clone()),
+            }) {
+            Some((pk, full_path)) => (pk, full_path),
             None => return Ok(()),
         };
 
@@ -245,35 +345,49 @@ impl Signer for DescriptorXKey<ExtendedPrivKey> {
             None => self.xkey.derive_priv(secp, &full_path).unwrap(),
         };
 
-        if &derived_key.private_key.public_key(secp) != public_key {
+        let computed_pk = secp256k1::PublicKey::from_secret_key(secp, &derived_key.private_key);
+        let valid_key = match public_key {
+            SinglePubKey::FullKey(pk) if pk.inner == computed_pk => true,
+            SinglePubKey::XOnly(x_only) if XOnlyPublicKey::from(computed_pk) == x_only => true,
+            _ => false,
+        };
+        if !valid_key {
             Err(SignerError::InvalidKey)
         } else {
-            derived_key.private_key.sign(psbt, Some(input_index), secp)
+            // HD wallets imply compressed keys
+            let priv_key = PrivateKey {
+                compressed: true,
+                network: self.xkey.network,
+                inner: derived_key.private_key,
+            };
+
+            SignerWrapper::new(priv_key, self.ctx).sign_input(psbt, input_index, sign_options, secp)
         }
     }
-
-    fn sign_whole_tx(&self) -> bool {
-        false
-    }
-
-    fn id(&self, secp: &SecpCtx) -> SignerId {
-        SignerId::from(self.root_fingerprint(secp))
-    }
-
-    fn descriptor_secret_key(&self) -> Option<DescriptorSecretKey> {
-        Some(DescriptorSecretKey::XPrv(self.clone()))
-    }
 }
 
-impl Signer for PrivateKey {
-    fn sign(
+impl SignerCommon for SignerWrapper<PrivateKey> {
+    fn id(&self, secp: &SecpCtx) -> SignerId {
+        SignerId::from(self.public_key(secp).to_pubkeyhash())
+    }
+
+    fn descriptor_secret_key(&self) -> Option<DescriptorSecretKey> {
+        Some(DescriptorSecretKey::SinglePriv(DescriptorSinglePriv {
+            key: self.signer,
+            origin: None,
+        }))
+    }
+}
+
+impl InputSigner for SignerWrapper<PrivateKey> {
+    fn sign_input(
         &self,
         psbt: &mut psbt::PartiallySignedTransaction,
-        input_index: Option<usize>,
+        input_index: usize,
+        sign_options: &SignOptions,
         secp: &SecpCtx,
     ) -> Result<(), SignerError> {
-        let input_index = input_index.unwrap();
-        if input_index >= psbt.inputs.len() || input_index >= psbt.global.unsigned_tx.input.len() {
+        if input_index >= psbt.inputs.len() || input_index >= psbt.unsigned_tx.input.len() {
             return Err(SignerError::InputIndexOutOfRange);
         }
 
@@ -283,49 +397,136 @@ impl Signer for PrivateKey {
             return Ok(());
         }
 
-        let pubkey = self.public_key(secp);
+        let pubkey = PublicKey::from_private_key(secp, self);
+        let x_only_pubkey = XOnlyPublicKey::from(pubkey.inner);
+
+        if let SignerContext::Tap { is_internal_key } = self.ctx {
+            if is_internal_key
+                && psbt.inputs[input_index].tap_key_sig.is_none()
+                && sign_options.sign_with_tap_internal_key
+            {
+                let (hash, hash_ty) = Tap::sighash(psbt, input_index, None)?;
+                sign_psbt_schnorr(
+                    &self.inner,
+                    x_only_pubkey,
+                    None,
+                    &mut psbt.inputs[input_index],
+                    hash,
+                    hash_ty,
+                    secp,
+                );
+            }
+
+            if let Some((leaf_hashes, _)) =
+                psbt.inputs[input_index].tap_key_origins.get(&x_only_pubkey)
+            {
+                let leaf_hashes = leaf_hashes
+                    .iter()
+                    .filter(|lh| {
+                        // Removing the leaves we shouldn't sign for
+                        let should_sign = match &sign_options.tap_leaves_options {
+                            TapLeavesOptions::All => true,
+                            TapLeavesOptions::Include(v) => v.contains(lh),
+                            TapLeavesOptions::Exclude(v) => !v.contains(lh),
+                            TapLeavesOptions::None => false,
+                        };
+                        // Filtering out the leaves without our key
+                        should_sign
+                            && !psbt.inputs[input_index]
+                                .tap_script_sigs
+                                .contains_key(&(x_only_pubkey, **lh))
+                    })
+                    .cloned()
+                    .collect::<Vec<_>>();
+                for lh in leaf_hashes {
+                    let (hash, hash_ty) = Tap::sighash(psbt, input_index, Some(lh))?;
+                    sign_psbt_schnorr(
+                        &self.inner,
+                        x_only_pubkey,
+                        Some(lh),
+                        &mut psbt.inputs[input_index],
+                        hash,
+                        hash_ty,
+                        secp,
+                    );
+                }
+            }
+
+            return Ok(());
+        }
+
         if psbt.inputs[input_index].partial_sigs.contains_key(&pubkey) {
             return Ok(());
         }
 
-        // FIXME: use the presence of `witness_utxo` as an indication that we should make a bip143
-        // sig. Does this make sense? Should we add an extra argument to explicitly swith between
-        // these? The original idea was to declare sign() as sign<Ctx: ScriptContex>() and use Ctx,
-        // but that violates the rules for trait-objects, so we can't do it.
-        let (hash, sighash) = match psbt.inputs[input_index].witness_utxo {
-            Some(_) => Segwitv0::sighash(psbt, input_index)?,
-            None => Legacy::sighash(psbt, input_index)?,
+        let (hash, hash_ty) = match self.ctx {
+            SignerContext::Segwitv0 => Segwitv0::sighash(psbt, input_index, ())?,
+            SignerContext::Legacy => Legacy::sighash(psbt, input_index, ())?,
+            _ => return Ok(()), // handled above
         };
-
-        let signature = secp.sign(
-            &Message::from_slice(&hash.into_inner()[..]).unwrap(),
-            &self.key,
+        sign_psbt_ecdsa(
+            &self.inner,
+            pubkey,
+            &mut psbt.inputs[input_index],
+            hash,
+            hash_ty,
+            secp,
         );
 
-        let mut final_signature = Vec::with_capacity(75);
-        final_signature.extend_from_slice(&signature.serialize_der());
-        final_signature.push(sighash.as_u32() as u8);
-
-        psbt.inputs[input_index]
-            .partial_sigs
-            .insert(pubkey, final_signature);
-
         Ok(())
     }
+}
 
-    fn sign_whole_tx(&self) -> bool {
-        false
-    }
+fn sign_psbt_ecdsa(
+    secret_key: &secp256k1::SecretKey,
+    pubkey: PublicKey,
+    psbt_input: &mut psbt::Input,
+    hash: bitcoin::Sighash,
+    hash_ty: EcdsaSighashType,
+    secp: &SecpCtx,
+) {
+    let msg = &Message::from_slice(&hash.into_inner()[..]).unwrap();
+    let sig = secp.sign_ecdsa(msg, secret_key);
+    secp.verify_ecdsa(msg, &sig, &pubkey.inner)
+        .expect("invalid or corrupted ecdsa signature");
 
-    fn id(&self, secp: &SecpCtx) -> SignerId {
-        SignerId::from(self.public_key(secp).to_pubkeyhash())
-    }
+    let final_signature = ecdsa::EcdsaSig { sig, hash_ty };
+    psbt_input.partial_sigs.insert(pubkey, final_signature);
+}
 
-    fn descriptor_secret_key(&self) -> Option<DescriptorSecretKey> {
-        Some(DescriptorSecretKey::SinglePriv(DescriptorSinglePriv {
-            key: *self,
-            origin: None,
-        }))
+// Calling this with `leaf_hash` = `None` will sign for key-spend
+fn sign_psbt_schnorr(
+    secret_key: &secp256k1::SecretKey,
+    pubkey: XOnlyPublicKey,
+    leaf_hash: Option<taproot::TapLeafHash>,
+    psbt_input: &mut psbt::Input,
+    hash: taproot::TapSighashHash,
+    hash_ty: SchnorrSighashType,
+    secp: &SecpCtx,
+) {
+    use schnorr::TapTweak;
+
+    let keypair = secp256k1::KeyPair::from_seckey_slice(secp, secret_key.as_ref()).unwrap();
+    let keypair = match leaf_hash {
+        None => keypair
+            .tap_tweak(secp, psbt_input.tap_merkle_root)
+            .into_inner(),
+        Some(_) => keypair, // no tweak for script spend
+    };
+
+    let msg = &Message::from_slice(&hash.into_inner()[..]).unwrap();
+    let sig = secp.sign_schnorr(msg, &keypair);
+    secp.verify_schnorr(&sig, msg, &XOnlyPublicKey::from_keypair(&keypair))
+        .expect("invalid or corrupted schnorr signature");
+
+    let final_signature = schnorr::SchnorrSig { sig, hash_ty };
+
+    if let Some(lh) = leaf_hash {
+        psbt_input
+            .tap_script_sigs
+            .insert((pubkey, lh), final_signature);
+    } else {
+        psbt_input.tap_key_sig = Some(final_signature);
     }
 }
 
@@ -360,7 +561,7 @@ impl From<(SignerId, SignerOrdering)> for SignersContainerKey {
 
 /// Container for multiple signers
 #[derive(Debug, Default, Clone)]
-pub struct SignersContainer(BTreeMap<SignersContainerKey, Arc<dyn Signer>>);
+pub struct SignersContainer(BTreeMap<SignersContainerKey, Arc<dyn TransactionSigner>>);
 
 impl SignersContainer {
     /// Create a map of public keys to secret keys
@@ -371,24 +572,37 @@ impl SignersContainer {
             .filter_map(|secret| secret.as_public(secp).ok().map(|public| (public, secret)))
             .collect()
     }
-}
 
-impl From<KeyMap> for SignersContainer {
-    fn from(keymap: KeyMap) -> SignersContainer {
-        let secp = Secp256k1::new();
+    /// Build a new signer container from a [`KeyMap`]
+    ///
+    /// Also looks at the corresponding descriptor to determine the [`SignerContext`] to attach to
+    /// the signers
+    pub fn build(
+        keymap: KeyMap,
+        descriptor: &Descriptor<DescriptorPublicKey>,
+        secp: &SecpCtx,
+    ) -> SignersContainer {
         let mut container = SignersContainer::new();
 
-        for (_, secret) in keymap {
+        for (pubkey, secret) in keymap {
+            let ctx = match descriptor {
+                Descriptor::Tr(tr) => SignerContext::Tap {
+                    is_internal_key: tr.internal_key() == &pubkey,
+                },
+                _ if descriptor.is_witness() => SignerContext::Segwitv0,
+                _ => SignerContext::Legacy,
+            };
+
             match secret {
                 DescriptorSecretKey::SinglePriv(private_key) => container.add_external(
-                    SignerId::from(private_key.key.public_key(&secp).to_pubkeyhash()),
+                    SignerId::from(private_key.key.public_key(secp).to_pubkeyhash()),
                     SignerOrdering::default(),
-                    Arc::new(private_key.key),
+                    Arc::new(SignerWrapper::new(private_key.key, ctx)),
                 ),
                 DescriptorSecretKey::XPrv(xprv) => container.add_external(
-                    SignerId::from(xprv.root_fingerprint(&secp)),
+                    SignerId::from(xprv.root_fingerprint(secp)),
                     SignerOrdering::default(),
-                    Arc::new(xprv),
+                    Arc::new(SignerWrapper::new(xprv, ctx)),
                 ),
             };
         }
@@ -409,13 +623,17 @@ impl SignersContainer {
         &mut self,
         id: SignerId,
         ordering: SignerOrdering,
-        signer: Arc<dyn Signer>,
-    ) -> Option<Arc<dyn Signer>> {
+        signer: Arc<dyn TransactionSigner>,
+    ) -> Option<Arc<dyn TransactionSigner>> {
         self.0.insert((id, ordering).into(), signer)
     }
 
     /// Removes a signer from the container and returns it
-    pub fn remove(&mut self, id: SignerId, ordering: SignerOrdering) -> Option<Arc<dyn Signer>> {
+    pub fn remove(
+        &mut self,
+        id: SignerId,
+        ordering: SignerOrdering,
+    ) -> Option<Arc<dyn TransactionSigner>> {
         self.0.remove(&(id, ordering).into())
     }
 
@@ -428,12 +646,12 @@ impl SignersContainer {
     }
 
     /// Returns the list of signers in the container, sorted by lowest to highest `ordering`
-    pub fn signers(&self) -> Vec<&Arc<dyn Signer>> {
+    pub fn signers(&self) -> Vec<&Arc<dyn TransactionSigner>> {
         self.0.values().collect()
     }
 
     /// Finds the signer with lowest ordering for a given id in the container.
-    pub fn find(&self, id: SignerId) -> Option<&Arc<dyn Signer>> {
+    pub fn find(&self, id: SignerId) -> Option<&Arc<dyn TransactionSigner>> {
         self.0
             .range((
                 Included(&(id.clone(), SignerOrdering(0)).into()),
@@ -477,38 +695,99 @@ pub struct SignOptions {
     ///
     /// Defaults to `false` which will only allow signing using `SIGHASH_ALL`.
     pub allow_all_sighashes: bool,
+
+    /// Whether to remove partial_sigs from psbt inputs while finalizing psbt.
+    ///
+    /// Defaults to `true` which will remove partial_sigs after finalizing.
+    pub remove_partial_sigs: bool,
+
+    /// Whether to try finalizing psbt input after the inputs are signed.
+    ///
+    /// Defaults to `true` which will try fianlizing psbt after inputs are signed.
+    pub try_finalize: bool,
+
+    /// Specifies which Taproot script-spend leaves we should sign for. This option is
+    /// ignored if we're signing a non-taproot PSBT.
+    ///
+    /// Defaults to All, i.e., the wallet will sign all the leaves it has a key for.
+    pub tap_leaves_options: TapLeavesOptions,
+
+    /// Whether we should try to sign a taproot transaction with the taproot internal key
+    /// or not. This option is ignored if we're signing a non-taproot PSBT.
+    ///
+    /// Defaults to `true`, i.e., we always try to sign with the taproot internal key.
+    pub sign_with_tap_internal_key: bool,
 }
 
+/// Customize which taproot script-path leaves the signer should sign.
+#[derive(Debug, Clone, PartialEq)]
+pub enum TapLeavesOptions {
+    /// The signer will sign all the leaves it has a key for.
+    All,
+    /// The signer won't sign leaves other than the ones specified. Note that it could still ignore
+    /// some of the specified leaves, if it doesn't have the right key to sign them.
+    Include(Vec<taproot::TapLeafHash>),
+    /// The signer won't sign the specified leaves.
+    Exclude(Vec<taproot::TapLeafHash>),
+    /// The signer won't sign any leaf.
+    None,
+}
+
+impl Default for TapLeavesOptions {
+    fn default() -> Self {
+        TapLeavesOptions::All
+    }
+}
+
+#[allow(clippy::derivable_impls)]
 impl Default for SignOptions {
     fn default() -> Self {
         SignOptions {
             trust_witness_utxo: false,
             assume_height: None,
             allow_all_sighashes: false,
+            remove_partial_sigs: true,
+            try_finalize: true,
+            tap_leaves_options: TapLeavesOptions::default(),
+            sign_with_tap_internal_key: true,
         }
     }
 }
 
 pub(crate) trait ComputeSighash {
+    type Extra;
+    type Sighash;
+    type SighashType;
+
     fn sighash(
         psbt: &psbt::PartiallySignedTransaction,
         input_index: usize,
-    ) -> Result<(SigHash, SigHashType), SignerError>;
+        extra: Self::Extra,
+    ) -> Result<(Self::Sighash, Self::SighashType), SignerError>;
 }
 
 impl ComputeSighash for Legacy {
+    type Extra = ();
+    type Sighash = bitcoin::Sighash;
+    type SighashType = EcdsaSighashType;
+
     fn sighash(
         psbt: &psbt::PartiallySignedTransaction,
         input_index: usize,
-    ) -> Result<(SigHash, SigHashType), SignerError> {
-        if input_index >= psbt.inputs.len() || input_index >= psbt.global.unsigned_tx.input.len() {
+        _extra: (),
+    ) -> Result<(Self::Sighash, Self::SighashType), SignerError> {
+        if input_index >= psbt.inputs.len() || input_index >= psbt.unsigned_tx.input.len() {
             return Err(SignerError::InputIndexOutOfRange);
         }
 
         let psbt_input = &psbt.inputs[input_index];
-        let tx_input = &psbt.global.unsigned_tx.input[input_index];
+        let tx_input = &psbt.unsigned_tx.input[input_index];
 
-        let sighash = psbt_input.sighash_type.unwrap_or(SigHashType::All);
+        let sighash = psbt_input
+            .sighash_type
+            .unwrap_or_else(|| EcdsaSighashType::All.into())
+            .ecdsa_hash_ty()
+            .map_err(|_| SignerError::InvalidSighash)?;
         let script = match psbt_input.redeem_script {
             Some(ref redeem_script) => redeem_script.clone(),
             None => {
@@ -526,9 +805,11 @@ impl ComputeSighash for Legacy {
         };
 
         Ok((
-            psbt.global
-                .unsigned_tx
-                .signature_hash(input_index, &script, sighash.as_u32()),
+            sighash::SighashCache::new(&psbt.unsigned_tx).legacy_signature_hash(
+                input_index,
+                &script,
+                sighash.to_u32(),
+            )?,
             sighash,
         ))
     }
@@ -545,18 +826,27 @@ fn p2wpkh_script_code(script: &Script) -> Script {
 }
 
 impl ComputeSighash for Segwitv0 {
+    type Extra = ();
+    type Sighash = bitcoin::Sighash;
+    type SighashType = EcdsaSighashType;
+
     fn sighash(
         psbt: &psbt::PartiallySignedTransaction,
         input_index: usize,
-    ) -> Result<(SigHash, SigHashType), SignerError> {
-        if input_index >= psbt.inputs.len() || input_index >= psbt.global.unsigned_tx.input.len() {
+        _extra: (),
+    ) -> Result<(Self::Sighash, Self::SighashType), SignerError> {
+        if input_index >= psbt.inputs.len() || input_index >= psbt.unsigned_tx.input.len() {
             return Err(SignerError::InputIndexOutOfRange);
         }
 
         let psbt_input = &psbt.inputs[input_index];
-        let tx_input = &psbt.global.unsigned_tx.input[input_index];
+        let tx_input = &psbt.unsigned_tx.input[input_index];
 
-        let sighash = psbt_input.sighash_type.unwrap_or(SigHashType::All);
+        let sighash = psbt_input
+            .sighash_type
+            .unwrap_or_else(|| EcdsaSighashType::All.into())
+            .ecdsa_hash_ty()
+            .map_err(|_| SignerError::InvalidSighash)?;
 
         // Always try first with the non-witness utxo
         let utxo = if let Some(prev_tx) = &psbt_input.non_witness_utxo {
@@ -599,17 +889,72 @@ impl ComputeSighash for Segwitv0 {
         };
 
         Ok((
-            bip143::SigHashCache::new(&psbt.global.unsigned_tx).signature_hash(
+            sighash::SighashCache::new(&psbt.unsigned_tx).segwit_signature_hash(
                 input_index,
                 &script,
                 value,
                 sighash,
-            ),
+            )?,
             sighash,
         ))
     }
 }
 
+impl ComputeSighash for Tap {
+    type Extra = Option<taproot::TapLeafHash>;
+    type Sighash = taproot::TapSighashHash;
+    type SighashType = SchnorrSighashType;
+
+    fn sighash(
+        psbt: &psbt::PartiallySignedTransaction,
+        input_index: usize,
+        extra: Self::Extra,
+    ) -> Result<(Self::Sighash, SchnorrSighashType), SignerError> {
+        if input_index >= psbt.inputs.len() || input_index >= psbt.unsigned_tx.input.len() {
+            return Err(SignerError::InputIndexOutOfRange);
+        }
+
+        let psbt_input = &psbt.inputs[input_index];
+
+        let sighash_type = psbt_input
+            .sighash_type
+            .unwrap_or_else(|| SchnorrSighashType::Default.into())
+            .schnorr_hash_ty()
+            .map_err(|_| SignerError::InvalidSighash)?;
+        let witness_utxos = psbt
+            .inputs
+            .iter()
+            .cloned()
+            .map(|i| i.witness_utxo)
+            .collect::<Vec<_>>();
+        let mut all_witness_utxos = vec![];
+
+        let mut cache = sighash::SighashCache::new(&psbt.unsigned_tx);
+        let is_anyone_can_pay = psbt::PsbtSighashType::from(sighash_type).to_u32() & 0x80 != 0;
+        let prevouts = if is_anyone_can_pay {
+            sighash::Prevouts::One(
+                input_index,
+                witness_utxos[input_index]
+                    .as_ref()
+                    .ok_or(SignerError::MissingWitnessUtxo)?,
+            )
+        } else if witness_utxos.iter().all(Option::is_some) {
+            all_witness_utxos.extend(witness_utxos.iter().filter_map(|x| x.as_ref()));
+            sighash::Prevouts::All(&all_witness_utxos)
+        } else {
+            return Err(SignerError::MissingWitnessUtxo);
+        };
+
+        // Assume no OP_CODESEPARATOR
+        let extra = extra.map(|leaf_hash| (leaf_hash, 0xFFFFFFFF));
+
+        Ok((
+            cache.taproot_signature_hash(input_index, &prevouts, None, extra, sighash_type)?,
+            sighash_type,
+        ))
+    }
+}
+
 impl PartialOrd for SignersContainerKey {
     fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
         Some(self.cmp(other))
@@ -640,12 +985,11 @@ mod signers_container_tests {
     use crate::keys::{DescriptorKey, IntoDescriptorKey};
     use bitcoin::secp256k1::{All, Secp256k1};
     use bitcoin::util::bip32;
-    use bitcoin::util::psbt::PartiallySignedTransaction;
     use bitcoin::Network;
     use miniscript::ScriptContext;
     use std::str::FromStr;
 
-    fn is_equal(this: &Arc<dyn Signer>, that: &Arc<DummySigner>) -> bool {
+    fn is_equal(this: &Arc<dyn TransactionSigner>, that: &Arc<DummySigner>) -> bool {
         let secp = Secp256k1::new();
         this.id(&secp) == that.id(&secp)
     }
@@ -660,11 +1004,11 @@ mod signers_container_tests {
         let (prvkey1, _, _) = setup_keys(TPRV0_STR);
         let (prvkey2, _, _) = setup_keys(TPRV1_STR);
         let desc = descriptor!(sh(multi(2, prvkey1, prvkey2))).unwrap();
-        let (_, keymap) = desc
+        let (wallet_desc, keymap) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();
 
-        let signers = SignersContainer::from(keymap);
+        let signers = SignersContainer::build(keymap, &wallet_desc, &secp);
         assert_eq!(signers.ids().len(), 2);
 
         let signers = signers.signers();
@@ -726,22 +1070,20 @@ mod signers_container_tests {
         number: u64,
     }
 
-    impl Signer for DummySigner {
-        fn sign(
-            &self,
-            _psbt: &mut PartiallySignedTransaction,
-            _input_index: Option<usize>,
-            _secp: &SecpCtx,
-        ) -> Result<(), SignerError> {
-            Ok(())
-        }
-
+    impl SignerCommon for DummySigner {
         fn id(&self, _secp: &SecpCtx) -> SignerId {
             SignerId::Dummy(self.number)
         }
+    }
 
-        fn sign_whole_tx(&self) -> bool {
-            true
+    impl TransactionSigner for DummySigner {
+        fn sign_transaction(
+            &self,
+            _psbt: &mut psbt::PartiallySignedTransaction,
+            _sign_options: &SignOptions,
+            _secp: &SecpCtx,
+        ) -> Result<(), SignerError> {
+            Ok(())
         }
     }
 
@@ -756,7 +1098,7 @@ mod signers_container_tests {
         let secp: Secp256k1<All> = Secp256k1::new();
         let path = bip32::DerivationPath::from_str(PATH).unwrap();
         let tprv = bip32::ExtendedPrivKey::from_str(tprv).unwrap();
-        let tpub = bip32::ExtendedPubKey::from_private(&secp, &tprv);
+        let tpub = bip32::ExtendedPubKey::from_priv(&secp, &tprv);
         let fingerprint = tprv.fingerprint(&secp);
         let prvkey = (tprv, path.clone()).into_descriptor_key().unwrap();
         let pubkey = (tpub, path).into_descriptor_key().unwrap();

src/wallet/tx_builder.rs

@@ -42,7 +42,7 @@ use std::default::Default;
 use std::marker::PhantomData;
 
 use bitcoin::util::psbt::{self, PartiallySignedTransaction as Psbt};
-use bitcoin::{OutPoint, Script, SigHashType, Transaction};
+use bitcoin::{OutPoint, Script, Transaction};
 
 use miniscript::descriptor::DescriptorTrait;
 
@@ -103,10 +103,7 @@ impl TxBuilderContext for BumpFee {}
 ///     builder.finish()?
 /// };
 ///
-/// assert_eq!(
-///     psbt1.global.unsigned_tx.output[..2],
-///     psbt2.global.unsigned_tx.output[..2]
-/// );
+/// assert_eq!(psbt1.unsigned_tx.output[..2], psbt2.unsigned_tx.output[..2]);
 /// # Ok::<(), bdk::Error>(())
 /// ```
 ///
@@ -120,8 +117,8 @@ impl TxBuilderContext for BumpFee {}
 /// [`finish`]: Self::finish
 /// [`coin_selection`]: Self::coin_selection
 #[derive(Debug)]
-pub struct TxBuilder<'a, B, D, Cs, Ctx> {
-    pub(crate) wallet: &'a Wallet<B, D>,
+pub struct TxBuilder<'a, D, Cs, Ctx> {
+    pub(crate) wallet: &'a Wallet<D>,
     pub(crate) params: TxParams,
     pub(crate) coin_selection: Cs,
     pub(crate) phantom: PhantomData<Ctx>,
@@ -140,7 +137,7 @@ pub(crate) struct TxParams {
     pub(crate) utxos: Vec<WeightedUtxo>,
     pub(crate) unspendable: HashSet<OutPoint>,
     pub(crate) manually_selected_only: bool,
-    pub(crate) sighash: Option<SigHashType>,
+    pub(crate) sighash: Option<psbt::PsbtSighashType>,
     pub(crate) ordering: TxOrdering,
     pub(crate) locktime: Option<u32>,
     pub(crate) rbf: Option<RbfValue>,
@@ -150,6 +147,8 @@ pub(crate) struct TxParams {
     pub(crate) add_global_xpubs: bool,
     pub(crate) include_output_redeem_witness_script: bool,
     pub(crate) bumping_fee: Option<PreviousFee>,
+    pub(crate) current_height: Option<u32>,
+    pub(crate) allow_dust: bool,
 }
 
 #[derive(Clone, Copy, Debug)]
@@ -170,7 +169,7 @@ impl std::default::Default for FeePolicy {
     }
 }
 
-impl<'a, Cs: Clone, Ctx, B, D> Clone for TxBuilder<'a, B, D, Cs, Ctx> {
+impl<'a, Cs: Clone, Ctx, D> Clone for TxBuilder<'a, D, Cs, Ctx> {
     fn clone(&self) -> Self {
         TxBuilder {
             wallet: self.wallet,
@@ -182,8 +181,8 @@ impl<'a, Cs: Clone, Ctx, B, D> Clone for TxBuilder<'a, B, D, Cs, Ctx> {
 }
 
 // methods supported by both contexts, for any CoinSelectionAlgorithm
-impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
-    TxBuilder<'a, B, D, Cs, Ctx>
+impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
+    TxBuilder<'a, D, Cs, Ctx>
 {
     /// Set a custom fee rate
     pub fn fee_rate(&mut self, fee_rate: FeeRate) -> &mut Self {
@@ -310,7 +309,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     /// 2. `psbt_input`: To know the value.
     /// 3. `satisfaction_weight`: To know how much weight/vbytes the input will add to the transaction for fee calculation.
     ///
-    /// There are several security concerns about adding foregin UTXOs that application
+    /// There are several security concerns about adding foreign UTXOs that application
     /// developers should consider. First, how do you know the value of the input is correct? If a
     /// `non_witness_utxo` is provided in the `psbt_input` then this method implicitly verifies the
     /// value by checking it against the transaction. If only a `witness_utxo` is provided then this
@@ -337,8 +336,9 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     /// 1. The `psbt_input` does not contain a `witness_utxo` or `non_witness_utxo`.
     /// 2. The data in `non_witness_utxo` does not match what is in `outpoint`.
     ///
-    /// Note unless you set [`only_witness_utxo`] any `psbt_input` you pass to this method must
-    /// have `non_witness_utxo` set otherwise you will get an error when [`finish`] is called.
+    /// Note unless you set [`only_witness_utxo`] any non-taproot `psbt_input` you pass to this
+    /// method must have `non_witness_utxo` set otherwise you will get an error when [`finish`]
+    /// is called.
     ///
     /// [`only_witness_utxo`]: Self::only_witness_utxo
     /// [`finish`]: Self::finish
@@ -412,7 +412,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     /// Sign with a specific sig hash
     ///
     /// **Use this option very carefully**
-    pub fn sighash(&mut self, sighash: SigHashType) -> &mut Self {
+    pub fn sighash(&mut self, sighash: psbt::PsbtSighashType) -> &mut Self {
         self.params.sighash = Some(sighash);
         self
     }
@@ -508,7 +508,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
     pub fn coin_selection<P: CoinSelectionAlgorithm<D>>(
         self,
         coin_selection: P,
-    ) -> TxBuilder<'a, B, D, P, Ctx> {
+    ) -> TxBuilder<'a, D, P, Ctx> {
         TxBuilder {
             wallet: self.wallet,
             params: self.params,
@@ -517,7 +517,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
         }
     }
 
-    /// Finish the building the transaction.
+    /// Finish building the transaction.
     ///
     /// Returns the [`BIP174`] "PSBT" and summary details about the transaction.
     ///
@@ -545,9 +545,33 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderConte
         self.params.rbf = Some(RbfValue::Value(nsequence));
         self
     }
+
+    /// Set the current blockchain height.
+    ///
+    /// This will be used to:
+    /// 1. Set the nLockTime for preventing fee sniping.
+    /// **Note**: This will be ignored if you manually specify a nlocktime using [`TxBuilder::nlocktime`].
+    /// 2. Decide whether coinbase outputs are mature or not. If the coinbase outputs are not
+    ///    mature at `current_height`, we ignore them in the coin selection.
+    ///    If you want to create a transaction that spends immature coinbase inputs, manually
+    ///    add them using [`TxBuilder::add_utxos`].
+    ///
+    /// In both cases, if you don't provide a current height, we use the last sync height.
+    pub fn current_height(&mut self, height: u32) -> &mut Self {
+        self.params.current_height = Some(height);
+        self
+    }
+
+    /// Set whether or not the dust limit is checked.
+    ///
+    /// **Note**: by avoiding a dust limit check you may end up with a transaction that is non-standard.
+    pub fn allow_dust(&mut self, allow_dust: bool) -> &mut Self {
+        self.params.allow_dust = allow_dust;
+        self
+    }
 }
 
-impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, B, D, Cs, CreateTx> {
+impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, D, Cs, CreateTx> {
     /// Replace the recipients already added with a new list
     pub fn set_recipients(&mut self, recipients: Vec<(Script, u64)>) -> &mut Self {
         self.params.recipients = recipients;
@@ -576,6 +600,9 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, B, D,
     /// difference is that it is valid to use `drain_to` without setting any ordinary recipients
     /// with [`add_recipient`] (but it is perfectly fine to add recipients as well).
     ///
+    /// If you choose not to set any recipients, you should either provide the utxos that the
+    /// transaction should spend via [`add_utxos`], or set [`drain_wallet`] to spend all of them.
+    ///
     /// When bumping the fees of a transaction made with this option, you probably want to
     /// use [`allow_shrinking`] to allow this output to be reduced to pay for the extra fees.
     ///
@@ -606,6 +633,7 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, B, D,
     ///
     /// [`allow_shrinking`]: Self::allow_shrinking
     /// [`add_recipient`]: Self::add_recipient
+    /// [`add_utxos`]: Self::add_utxos
     /// [`drain_wallet`]: Self::drain_wallet
     pub fn drain_to(&mut self, script_pubkey: Script) -> &mut Self {
         self.params.drain_to = Some(script_pubkey);
@@ -614,8 +642,8 @@ impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, B, D,
 }
 
 // methods supported only by bump_fee
-impl<'a, B, D: BatchDatabase> TxBuilder<'a, B, D, DefaultCoinSelectionAlgorithm, BumpFee> {
-    /// Explicitly tells the wallet that it is allowed to reduce the fee of the output matching this
+impl<'a, D: BatchDatabase> TxBuilder<'a, D, DefaultCoinSelectionAlgorithm, BumpFee> {
+    /// Explicitly tells the wallet that it is allowed to reduce the amount of the output matching this
     /// `script_pubkey` in order to bump the transaction fee. Without specifying this the wallet
     /// will attempt to find a change output to shrink instead.
     ///
@@ -838,6 +866,7 @@ mod test {
                 },
                 txout: Default::default(),
                 keychain: KeychainKind::External,
+                is_spent: false,
             },
             LocalUtxo {
                 outpoint: OutPoint {
@@ -846,6 +875,7 @@ mod test {
                 },
                 txout: Default::default(),
                 keychain: KeychainKind::Internal,
+                is_spent: false,
             },
         ]
     }

src/wallet/utils.rs

@@ -9,13 +9,11 @@
 // You may not use this file except in accordance with one or both of these
 // licenses.
 
+use bitcoin::blockdata::script::Script;
 use bitcoin::secp256k1::{All, Secp256k1};
 
 use miniscript::{MiniscriptKey, Satisfier, ToPublicKey};
 
-// De-facto standard "dust limit" (even though it should change based on the output type)
-pub const DUST_LIMIT_SATOSHI: u64 = 546;
-
 // MSB of the nSequence. If set there's no consensus-constraint, so it must be disabled when
 // spending using CSV in order to enforce CSV rules
 pub(crate) const SEQUENCE_LOCKTIME_DISABLE_FLAG: u32 = 1 << 31;
@@ -28,18 +26,19 @@ pub(crate) const SEQUENCE_LOCKTIME_MASK: u32 = 0x0000FFFF;
 // Threshold for nLockTime to be considered a block-height-based timelock rather than time-based
 pub(crate) const BLOCKS_TIMELOCK_THRESHOLD: u32 = 500000000;
 
-/// Trait to check if a value is below the dust limit
+/// Trait to check if a value is below the dust limit.
+/// We are performing dust value calculation for a given script public key using rust-bitcoin to
+/// keep it compatible with network dust rate
 // we implement this trait to make sure we don't mess up the comparison with off-by-one like a <
-// instead of a <= etc. The constant value for the dust limit is not public on purpose, to
-// encourage the usage of this trait.
+// instead of a <= etc.
 pub trait IsDust {
     /// Check whether or not a value is below dust limit
-    fn is_dust(&self) -> bool;
+    fn is_dust(&self, script: &Script) -> bool;
 }
 
 impl IsDust for u64 {
-    fn is_dust(&self) -> bool {
-        *self <= DUST_LIMIT_SATOSHI
+    fn is_dust(&self, script: &Script) -> bool {
+        *self < script.dust_value().as_sat()
     }
 }
 
@@ -138,64 +137,30 @@ impl<Pk: MiniscriptKey + ToPublicKey> Satisfier<Pk> for Older {
 
 pub(crate) type SecpCtx = Secp256k1<All>;
 
-pub struct ChunksIterator<I: Iterator> {
-    iter: I,
-    size: usize,
-}
-
-#[cfg(any(feature = "electrum", feature = "esplora"))]
-impl<I: Iterator> ChunksIterator<I> {
-    pub fn new(iter: I, size: usize) -> Self {
-        ChunksIterator { iter, size }
-    }
-}
-
-impl<I: Iterator> Iterator for ChunksIterator<I> {
-    type Item = Vec<<I as std::iter::Iterator>::Item>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        let mut v = Vec::new();
-        for _ in 0..self.size {
-            let e = self.iter.next();
-
-            match e {
-                None => break,
-                Some(val) => v.push(val),
-            }
-        }
-
-        if v.is_empty() {
-            return None;
-        }
-
-        Some(v)
-    }
-}
-
 #[cfg(test)]
 mod test {
     use super::{
-        check_nlocktime, check_nsequence_rbf, BLOCKS_TIMELOCK_THRESHOLD,
+        check_nlocktime, check_nsequence_rbf, IsDust, BLOCKS_TIMELOCK_THRESHOLD,
         SEQUENCE_LOCKTIME_TYPE_FLAG,
     };
-    use crate::types::FeeRate;
+    use crate::bitcoin::Address;
+    use std::str::FromStr;
 
     #[test]
-    fn test_fee_from_btc_per_kb() {
-        let fee = FeeRate::from_btc_per_kvb(1e-5);
-        assert!((fee.as_sat_vb() - 1.0).abs() < 0.0001);
-    }
+    fn test_is_dust() {
+        let script_p2pkh = Address::from_str("1GNgwA8JfG7Kc8akJ8opdNWJUihqUztfPe")
+            .unwrap()
+            .script_pubkey();
+        assert!(script_p2pkh.is_p2pkh());
+        assert!(545.is_dust(&script_p2pkh));
+        assert!(!546.is_dust(&script_p2pkh));
 
-    #[test]
-    fn test_fee_from_sats_vbyte() {
-        let fee = FeeRate::from_sat_per_vb(1.0);
-        assert!((fee.as_sat_vb() - 1.0).abs() < 0.0001);
-    }
-
-    #[test]
-    fn test_fee_default_min_relay_fee() {
-        let fee = FeeRate::default_min_relay_fee();
-        assert!((fee.as_sat_vb() - 1.0).abs() < 0.0001);
+        let script_p2wpkh = Address::from_str("bc1qxlh2mnc0yqwas76gqq665qkggee5m98t8yskd8")
+            .unwrap()
+            .script_pubkey();
+        assert!(script_p2wpkh.is_v0_p2wpkh());
+        assert!(293.is_dust(&script_p2wpkh));
+        assert!(!294.is_dust(&script_p2wpkh));
     }
 
     #[test]

src/wallet/verify.rs

@@ -17,7 +17,7 @@ use std::fmt;
 use bitcoin::consensus::serialize;
 use bitcoin::{OutPoint, Transaction, Txid};
 
-use crate::blockchain::Blockchain;
+use crate::blockchain::GetTx;
 use crate::database::Database;
 use crate::error::Error;
 
@@ -29,7 +29,7 @@ use crate::error::Error;
 /// Depending on the [capabilities](crate::blockchain::Blockchain::get_capabilities) of the
 /// [`Blockchain`] backend, the method could fail when called with old "historical" transactions or
 /// with unconfirmed transactions that have been evicted from the backend's memory.
-pub fn verify_tx<D: Database, B: Blockchain>(
+pub fn verify_tx<D: Database, B: GetTx>(
     tx: &Transaction,
     database: &D,
     blockchain: &B,
@@ -104,43 +104,18 @@ impl_error!(bitcoinconsensus::Error, Consensus, VerifyError);
 
 #[cfg(test)]
 mod test {
-    use std::collections::HashSet;
-
+    use super::*;
+    use crate::database::{BatchOperations, MemoryDatabase};
     use bitcoin::consensus::encode::deserialize;
     use bitcoin::hashes::hex::FromHex;
     use bitcoin::{Transaction, Txid};
 
-    use crate::blockchain::{Blockchain, Capability, Progress};
-    use crate::database::{BatchDatabase, BatchOperations, MemoryDatabase};
-    use crate::FeeRate;
-
-    use super::*;
-
     struct DummyBlockchain;
 
-    impl Blockchain for DummyBlockchain {
-        fn get_capabilities(&self) -> HashSet<Capability> {
-            Default::default()
-        }
-        fn setup<D: BatchDatabase, P: 'static + Progress>(
-            &self,
-            _database: &mut D,
-            _progress_update: P,
-        ) -> Result<(), Error> {
-            Ok(())
-        }
+    impl GetTx for DummyBlockchain {
         fn get_tx(&self, _txid: &Txid) -> Result<Option<Transaction>, Error> {
             Ok(None)
         }
-        fn broadcast(&self, _tx: &Transaction) -> Result<(), Error> {
-            Ok(())
-        }
-        fn get_height(&self) -> Result<u32, Error> {
-            Ok(42)
-        }
-        fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
-            Ok(FeeRate::default_min_relay_fee())
-        }
     }
 
     #[test]