ci: update MSRV version pinning in workflow and README

530ba36b07 ci: fix msrv dependency versions for reqest and h2 (Daniela Brozzoni)
7a359d5eef fix(electrum): Don't ignore multiple coinbase txs (Daniela Brozzoni)

Pull request description:

  We would previously insert just one coinbase transaction in the database if we caught multiple in the same sync.
  When we sync with electrum, before committing to the database, we remove from the update conflicting transactions, using the `make_txs_consistent` function. This function considers two txs to be conflicting if they spend from the same outpoint - but every coinbase transaction spends from the same outpoint!
  Here we make sure to avoid filtering out coinbase transactions, by adding a check on the txid just before we do the filtering.

  Fixes #1051

  ### Changelog notice

  - Fix a bug when syncing coinbase utxos on electrum

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### Bugfixes:

  * [x] I've added tests to reproduce the issue which are now passing
  * [x] I'm linking the issue being fixed by this PR

ACKs for top commit:
  notmandatory:
    tACK 530ba36b07

Tree-SHA512: ebbc6af86d4433ac4083212033a23f183d109641db345cc06ab4d506995ab71657761351c03772462ab4ff0d081226ecc97f1042490194aaf8661914cbeb72cb

.cargo/audit.toml

@@ -0,0 +1,2 @@
+[advisories]
+ignore = ["RUSTSEC-2022-0046"]

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,26 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: 'bug'
+assignees: ''
+
+---
+
+**Describe the bug**  
+<!-- A clear and concise description of what the bug is. -->
+
+**To Reproduce**  
+<!-- Steps or code to reproduce the behavior. -->
+
+**Expected behavior**  
+<!-- A clear and concise description of what you expected to happen. -->
+
+**Build environment**  
+ - BDK tag/commit: <!-- e.g. v0.13.0, 3a07614 --> 
+ - OS+version: <!-- e.g. ubuntu 20.04.01, macOS 12.0.1, windows -->  
+ - Rust/Cargo version: <!-- e.g. 1.56.0 --> 
+ - Rust/Cargo target: <!-- e.g. x86_64-apple-darwin, x86_64-unknown-linux-gnu, etc. -->  
+
+**Additional context**  
+<!-- Add any other context about the problem here. --> 

.github/ISSUE_TEMPLATE/enhancement_request.md

@@ -0,0 +1,17 @@
+---
+name: Enhancement request
+about: Request a new feature or change to an existing feature
+title: ''
+labels: 'enhancement'
+assignees: ''
+
+---
+
+**Describe the enhancement**  
+<!-- A clear and concise description of what you would like added or changed. -->
+
+**Use case**  
+<!-- Tell us how you or others will use this new feature or change to an existing feature. --> 
+
+**Additional context**
+<!-- Add any other context about the enhancement here. --> 

.github/ISSUE_TEMPLATE/minor_release.md

@@ -0,0 +1,99 @@
+---
+name: Minor Release
+about: Create a new minor release [for release managers only]
+title: 'Release MAJOR.MINOR+1.0'
+labels: 'release'
+assignees: ''
+
+---
+
+## Create a new minor release
+
+### Summary
+
+<--release summary to be used in announcements-->
+
+### Commit
+
+<--latest commit ID to include in this release-->
+
+### Changelog
+
+<--add notices from PRs merged since the prior release, see ["keep a changelog"]-->
+
+### Checklist
+
+Release numbering must follow [Semantic Versioning]. These steps assume the current `master`
+branch **development** version is *MAJOR.MINOR.0*.
+
+#### On the day of the feature freeze
+
+Change the `master` branch to the next MINOR+1 version:
+
+- [ ] Switch to the `master` branch.
+- [ ] Create a new PR branch called `bump_dev_MAJOR_MINOR+1`, eg. `bump_dev_0_22`.
+- [ ] Bump the `bump_dev_MAJOR_MINOR+1` branch to the next development MINOR+1 version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR+1.0`.
+  - Update the `CHANGELOG.md` file.
+  - The commit message should be "Bump version to MAJOR.MINOR+1.0".
+- [ ] Create PR and merge the `bump_dev_MAJOR_MINOR+1` branch to `master`.
+  - Title PR "Bump version to MAJOR.MINOR+1.0".
+
+Create a new release branch and release candidate tag:
+
+- [ ] Double check that your local `master` is up-to-date with the upstream repo.
+- [ ] Create a new branch called `release/MAJOR.MINOR+1` from `master`.
+- [ ] Bump the `release/MAJOR.MINOR+1` branch to `MAJOR.MINOR+1.0-rc.1` version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR+1.0-rc.1`.
+  - The commit message should be "Bump version to MAJOR.MINOR+1.0-rc.1".
+- [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR+1` branch.
+  - The tag name should be `vMAJOR.MINOR+1.0-rc.1`
+  - Use message "Release MAJOR.MINOR+1.0 rc.1".
+  - Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
+- [ ] Push the `release/MAJOR.MINOR` branch and new tag to the `bitcoindevkit/bdk` repo.
+  - Use `git push --tags` option to push the new `vMAJOR.MINOR+1.0-rc.1` tag.
+
+If any issues need to be fixed before the *MAJOR.MINOR+1.0* version is released:
+
+- [ ] Merge fix PRs to the `master` branch.
+- [ ] Git cherry-pick fix commits to the `release/MAJOR.MINOR+1` branch.
+- [ ] Verify fixes in `release/MAJOR.MINOR+1` branch.
+- [ ] Bump the `release/MAJOR.MINOR+1` branch to `MAJOR.MINOR+1.0-rc.x+1` version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR+1.0-rc.x+1`.
+  - The commit message should be "Bump version to MAJOR.MINOR+1.0-rc.x+1".
+- [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR+1` branch.
+  - The tag name should be `vMAJOR.MINOR+1.0-rc.x+1`, where x is the current release candidate number.
+  - Use tag message "Release MAJOR.MINOR+1.0 rc.x+1".
+  - Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
+- [ ] Push the new tag to the `bitcoindevkit/bdk` repo.
+  - Use `git push --tags` option to push the new `vMAJOR.MINOR+1.0-rc.x+1` tag.
+
+#### On the day of the release
+
+Tag and publish new release:
+
+- [ ] Bump the `release/MAJOR.MINOR+1` branch to `MAJOR.MINOR+1.0` version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR+1.0`.
+  - The commit message should be "Bump version to MAJOR.MINOR+1.0".
+- [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR+1` branch.
+  - The tag name should be `vMAJOR.MINOR+1.0`
+  - The first line of the tag message should be "Release MAJOR.MINOR+1.0".
+  - In the body of the tag message put a copy of the **Summary** and **Changelog** for the release.
+  - Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
+- [ ] Wait for the CI to finish one last time.
+- [ ] Push the new tag to the `bitcoindevkit/bdk` repo.
+- [ ] Publish **all** the updated crates to crates.io.
+- [ ] Create the release on GitHub.
+  - Go to "tags", click on the dots on the right and select "Create Release".
+  - Set the title to `Release MAJOR.MINOR+1.0`.
+  - In the release notes body put the **Summary** and **Changelog**.
+  - Use the "+ Auto-generate release notes" button to add details from included PRs.
+  - Until we reach a `1.0.0` release check the "Pre-release" box.
+- [ ] Make sure the new release shows up on [crates.io] and that the docs are built correctly on [docs.rs].
+- [ ] Announce the release, using the **Summary**, on Discord, Twitter and Mastodon.
+- [ ] Celebrate 🎉
+
+[Semantic Versioning]: https://semver.org/
+[crates.io]: https://crates.io/crates/bdk
+[docs.rs]: https://docs.rs/bdk/latest/bdk
+["keep a changelog"]: https://keepachangelog.com/en/1.0.0/

.github/ISSUE_TEMPLATE/patch_release.md

@@ -0,0 +1,71 @@
+---
+name: Patch Release
+about: Create a new patch release [for release managers only]
+title: 'Release MAJOR.MINOR.PATCH+1'
+labels: 'release'
+assignees: ''
+
+---
+
+## Create a new patch release
+
+### Summary
+
+<--release summary to be used in announcements-->
+
+### Commit
+
+<--latest commit ID to include in this release-->
+
+### Changelog
+
+<--add notices from PRs merged since the prior release, see ["keep a changelog"]-->
+
+### Checklist
+
+Release numbering must follow [Semantic Versioning]. These steps assume the current `master`
+branch **development** version is *MAJOR.MINOR.PATCH*.
+
+### On the day of the patch release
+
+Change the `master` branch to the new PATCH+1 version:
+
+- [ ] Switch to the `master` branch.
+- [ ] Create a new PR branch called `bump_dev_MAJOR_MINOR_PATCH+1`, eg. `bump_dev_0_22_1`.
+- [ ] Bump the `bump_dev_MAJOR_MINOR` branch to the next development PATCH+1 version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR.PATCH+1`.
+  - Update the `CHANGELOG.md` file.
+  - The commit message should be "Bump version to MAJOR.MINOR.PATCH+1".
+- [ ] Create PR and merge the `bump_dev_MAJOR_MINOR_PATCH+1` branch to `master`.
+  - Title PR "Bump version to MAJOR.MINOR.PATCH+1".
+
+Cherry-pick, tag and publish new PATCH+1 release:
+
+- [ ] Merge fix PRs to the `master` branch.
+- [ ] Git cherry-pick fix commits to the `release/MAJOR.MINOR` branch to be patched.
+- [ ] Verify fixes in `release/MAJOR.MINOR` branch.
+- [ ] Bump the `release/MAJOR.MINOR.PATCH+1` branch to `MAJOR.MINOR.PATCH+1` version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR.MINOR.PATCH+1`.
+  - The commit message should be "Bump version to MAJOR.MINOR.PATCH+1".
+- [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR` branch.
+  - The tag name should be `vMAJOR.MINOR.PATCH+1`
+  - The first line of the tag message should be "Release MAJOR.MINOR.PATCH+1".
+  - In the body of the tag message put a copy of the **Summary** and **Changelog** for the release.
+  - Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
+- [ ] Wait for the CI to finish one last time.
+- [ ] Push the new tag to the `bitcoindevkit/bdk` repo.
+- [ ] Publish **all** the updated crates to crates.io.
+- [ ] Create the release on GitHub.
+  - Go to "tags", click on the dots on the right and select "Create Release".
+  - Set the title to `Release MAJOR.MINOR.PATCH+1`.
+  - In the release notes body put the **Summary** and **Changelog**.
+  - Use the "+ Auto-generate release notes" button to add details from included PRs.
+  - Until we reach a `1.0.0` release check the "Pre-release" box.
+- [ ] Make sure the new release shows up on [crates.io] and that the docs are built correctly on [docs.rs].
+- [ ] Announce the release, using the **Summary**, on Discord, Twitter and Mastodon.
+- [ ] Celebrate 🎉
+
+[Semantic Versioning]: https://semver.org/
+[crates.io]: https://crates.io/crates/bdk
+[docs.rs]: https://docs.rs/bdk/latest/bdk
+["keep a changelog"]: https://keepachangelog.com/en/1.0.0/

.github/ISSUE_TEMPLATE/summer_project.md

@@ -0,0 +1,77 @@
+---
+name: Summer of Bitcoin Project
+about: Template to suggest a new https://www.summerofbitcoin.org/ project.
+title: ''
+labels: 'summer-of-bitcoin'
+assignees: ''
+
+---
+
+<!--
+## Overview  
+
+Project ideas are scoped for a university-level student with a basic background in CS and bitcoin 
+fundamentals - achievable over 12-weeks. Below are just a few types of ideas: 
+
+ - Low-hanging fruit: Relatively short projects with clear goals; requires basic technical knowledge 
+   and minimal familiarity with the codebase.
+ - Core development: These projects derive from the ongoing work from the core of your development 
+   team. The list of features and bugs is never-ending, and help is always welcome.
+ - Risky/Exploratory: These projects push the scope boundaries of your development effort. They 
+   might require expertise in an area not covered by your current development team. They might take 
+   advantage of a new technology. There is a reasonable chance that the project might be less 
+   successful, but the potential rewards make it worth the attempt.
+ - Infrastructure/Automation: These projects are the code that your organization uses to get its 
+   development work done; for example, projects that improve the automation of releases, regression 
+   tests and automated builds. This is a category where a Summer of Bitcoin student can be really 
+   helpful, doing work that the development team has been putting off while they focus on core 
+   development.
+ - Quality Assurance/Testing: Projects that work on and test your project's software development 
+   process. Additionally, projects that involve a thorough test and review of individual PRs.
+ - Fun/Peripheral: These projects might not be related to the current core development focus, but 
+   create new innovations and new perspectives for your project.
+-->
+
+**Description**
+<!-- Description: 3-7 sentences describing the project background and tasks to be done. -->  
+
+**Expected Outcomes**  
+<!-- Short bullet list describing what is to be accomplished -->   
+
+**Resources**  
+<!-- 2-3 reading materials for candidate to learn about the repo, project, scope etc -->  
+<!-- Recommended reading such as a developer/contributor guide -->  
+<!-- [Another example a paper citation](https://arxiv.org/pdf/1802.08091.pdf) -->  
+<!-- [Another example an existing issue](https://github.com/opencv/opencv/issues/11013) -->  
+<!-- [An existing related module](https://github.com/opencv/opencv_contrib/tree/master/modules/optflow) -->  
+
+**Skills Required**  
+<!-- 3-4 technical skills that the candidate should know -->  
+<!-- hands on experience with git -->  
+<!-- mastery plus experience coding in C++ -->  
+<!-- basic knowledge in matrix and tensor computations, college course work in cryptography -->  
+<!-- strong mathematical background -->
+<!-- Bonus - has experience with React Native. Best if you have also worked with OSSFuzz -->
+
+**Mentor(s)**  
+<!-- names of mentor(s) for this project go here -->
+
+**Difficulty**  
+<!-- Easy, Medium, Hard -->
+
+**Competency Test (optional)**  
+<!-- 2-3 technical tasks related to the project idea or repository you’d like a candidate to 
+    perform in order to demonstrate competency, good first bugs, warm-up exercises -->
+<!-- ex. Read the instructions here to get Bitcoin core running on your machine -->
+<!-- ex. pick an issue labeled as “newcomer” in the repository, and send a merge request to the 
+    repository. You can also suggest some other improvement that we did not think of yet, or 
+    something that you find interesting or useful -->
+<!-- ex. fixes for coding style are usually easy to do, and are good issues for first time 
+    contributions for those learning how to interact with the project. After you are done with the 
+    coding style issue, try making a different contribution. -->
+<!-- ex. setup a full Debian packaging development environment and learn the basics of Debian 
+    packaging. Then identify and package the missing dependencies to package Specter Desktop -->
+<!-- ex. write a pull parser for CSV files. You'll be judged by the decisions to store the parser 
+    state and how flexible it is to wrap this parser in other scenarios. -->
+<!-- ex. Stretch Goal: Implement some basic metaprogram/app to prove you're very familiar with BDK. 
+    Be prepared to make adjustments as we judge your solution. -->

.github/pull_request_template.md

@@ -0,0 +1,34 @@
+<!-- You can erase any parts of this template not applicable to your Pull Request. -->
+
+### Description
+
+<!-- Describe the purpose of this PR, what's being adding and/or fixed -->
+
+### Notes to the reviewers
+
+<!-- In this section you can include notes directed to the reviewers, like explaining why some parts
+of the PR were done in a specific way -->
+
+### Changelog notice
+
+<!-- Notice the release manager should include in the release tag message changelog -->
+<!-- See https://keepachangelog.com/en/1.0.0/ for examples -->
+
+### Checklists
+
+#### All Submissions:
+
+* [ ] I've signed all my commits
+* [ ] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
+* [ ] I ran `cargo fmt` and `cargo clippy` before committing
+
+#### New Features:
+
+* [ ] I've added tests for the new feature
+* [ ] I've added docs for the new feature
+
+#### Bugfixes:
+
+* [ ] This pull request breaks the existing API
+* [ ] I've added tests to reproduce the issue which are now passing
+* [ ] I'm linking the issue being fixed by this PR

.github/workflows/audit.yml

@@ -0,0 +1,22 @@
+name: Audit
+
+on:
+  push:
+    branches:
+      - 'master'
+      - 'release/*'
+    paths:
+      - '**/Cargo.toml'
+      - '**/Cargo.lock'
+  schedule:
+    - cron: '0 0 * * 0' # Once per week
+
+jobs:
+
+  security_audit:
+    runs-on: ubuntu-20.04
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions-rs/audit-check@v1
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/code_coverage.yml

@@ -0,0 +1,69 @@
+on:
+  push:
+    branches:
+      - 'master'
+      - 'release/*'
+  pull_request:
+    branches:
+      - 'master'
+      - 'release/*'
+
+name: Code Coverage
+
+jobs:
+  Codecov:
+    name: Code Coverage
+    runs-on: ubuntu-latest
+    env:
+      RUSTFLAGS: "-Cinstrument-coverage"
+      RUSTDOCFLAGS: "-Cinstrument-coverage"
+      LLVM_PROFILE_FILE: "report-%p-%m.profraw"
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Install lcov tools
+        run: sudo apt-get install lcov -y
+      - name: Install rustup
+        run: curl https://sh.rustup.rs -sSf | sh -s -- -y
+      - name: Set default toolchain
+        run: rustup default nightly
+      - name: Set profile
+        run: rustup set profile minimal
+      - name: Add llvm tools
+        run: rustup component add llvm-tools-preview
+      - name: Update toolchain
+        run: rustup update
+      - name: Cache cargo
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cargo/bin/
+            ~/.cargo/registry/index/
+            ~/.cargo/registry/cache/
+            ~/.cargo/git/db/
+          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
+      - name: Install grcov
+        run: if [[ ! -e ~/.cargo/bin/grcov ]]; then cargo install grcov; fi
+      - name: Test
+        # WARNING: this is not testing the following features: test-esplora, test-hardware-signer, async-interface
+        # This is because some of our features are mutually exclusive, and generating various reports and
+        # merging them doesn't seem to be working very well.
+        # For more info, see:
+        # - https://github.com/bitcoindevkit/bdk/issues/696
+        # - https://github.com/bitcoindevkit/bdk/pull/748#issuecomment-1242721040
+        run: cargo test --features all-keys,compact_filters,compiler,key-value-db,sqlite,sqlite-bundled,test-electrum,test-rpc,verify
+      - name: Run grcov
+        run: mkdir coverage; grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore '/*' -o ./coverage/lcov.info
+      - name: Generate HTML coverage report
+        run: genhtml -o coverage-report.html ./coverage/lcov.info
+
+      - name: Coveralls upload
+        uses: coverallsapp/github-action@master
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+      - name: Upload artifact
+        uses: actions/upload-artifact@v2
+        with:
+          name: coverage-report
+          path: coverage-report.html

.github/workflows/cont_integration.yml

@@ -0,0 +1,249 @@
+on:
+  push:
+    branches:
+      - 'master'
+      - 'release/*'
+  pull_request:
+    branches:
+      - 'master'
+      - 'release/*'
+
+name: CI
+
+jobs:
+
+  build-test:
+    name: Build and test
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        rust:
+          - version: 1.65.0 # STABLE
+            clippy: true
+          - version: 1.57.0 # MSRV
+        features:
+          - default
+          - minimal
+          - all-keys
+          - minimal,use-esplora-blocking
+          - key-value-db
+          - electrum
+          - compact_filters
+          - use-esplora-blocking,key-value-db,electrum
+          - compiler
+          - rpc
+          - verify
+          - async-interface
+          - use-esplora-async
+          - sqlite
+          - sqlite-bundled
+    steps:
+      - name: checkout
+        uses: actions/checkout@v2
+      - name: Generate cache key
+        run: echo "${{ matrix.rust.version }} ${{ matrix.features }}" | tee .cache_key
+      - name: cache
+        uses: actions/cache@v2
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-cargo-${{ hashFiles('.cache_key') }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
+      - name: Set default toolchain
+        run: rustup default ${{ matrix.rust.version }}
+      - name: Set profile
+        run: rustup set profile minimal
+      - name: Add clippy
+        if: ${{ matrix.rust.clippy }}
+        run: rustup component add clippy
+      - name: Update toolchain
+        run: rustup update
+      - name: Pin dependencies for MSRV
+        if: matrix.rust.version == '1.57.0'
+        run: |
+          cargo update -p log --precise "0.4.18"
+          cargo update -p tempfile --precise "3.6.0"
+          cargo update -p hashlink --precise "0.8.1"
+          cargo update -p regex --precise "1.7.3"
+          cargo update -p zip:0.6.6 --precise "0.6.3"
+          cargo update -p rustix --precise "0.37.23"
+          cargo update -p tokio --precise "1.29.1"
+          cargo update -p tokio-util --precise "0.7.8"
+          cargo update -p cc --precise "1.0.81"
+          cargo update -p rustls:0.20.9 --precise "0.20.8"
+          cargo update -p rustls:0.21.7 --precise "0.21.1"
+          cargo update -p flate2:1.0.27 --precise "1.0.26"
+          cargo update -p reqwest --precise "0.11.18"
+          cargo update -p h2 --precise "0.3.20"
+          cargo update -p rustls-webpki:0.100.3 --precise "0.100.1"
+          cargo update -p rustls-webpki:0.101.6 --precise "0.101.1"
+      - name: Build
+        run: cargo build --features ${{ matrix.features }} --no-default-features
+      - name: Clippy
+        if: ${{ matrix.rust.clippy }}
+        run: cargo clippy --all-targets --features ${{ matrix.features }} --no-default-features -- -D warnings
+      - name: Test
+        run: cargo test --features ${{ matrix.features }} --no-default-features
+
+  test-readme-examples:
+    name: Test README.md examples
+    runs-on: ubuntu-latest
+    steps:
+      - name: checkout
+        uses: actions/checkout@v2
+      - name: cache
+        uses: actions/cache@v2
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-cargo-test-md-docs-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
+      - name: Set default toolchain
+        run: rustup default nightly
+      - name: Set profile
+        run: rustup set profile minimal
+      - name: Update toolchain
+        run: rustup update
+      - name: Test
+        run: cargo test --features test-md-docs --no-default-features -- doctest::ReadmeDoctests
+
+  test-blockchains:
+    name: Blockchain ${{ matrix.blockchain.features }}
+    runs-on: ubuntu-20.04
+    strategy:
+      fail-fast: false
+      matrix:
+        blockchain:
+          - name: electrum
+            testprefix: blockchain::electrum::test
+            features: test-electrum,verify
+          - name: rpc
+            testprefix: blockchain::rpc::test
+            features: test-rpc
+          - name: rpc-legacy
+            testprefix: blockchain::rpc::test
+            features: test-rpc-legacy
+          - name: esplora
+            testprefix: esplora
+            features: test-esplora,use-esplora-async,verify
+          - name: esplora
+            testprefix: esplora
+            features: test-esplora,use-esplora-blocking,verify
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Cache
+        uses: actions/cache@v2
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-cargo-${{ github.job }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
+      - name: Setup rust toolchain
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
+      - name: Test
+        run: cargo test --no-default-features --features ${{ matrix.blockchain.features }} ${{ matrix.blockchain.testprefix }}::bdk_blockchain_tests
+
+  check-wasm:
+    name: Check WASM
+    runs-on: ubuntu-20.04
+    env:
+      CC: clang-10
+      CFLAGS: -I/usr/include
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Cache
+        uses: actions/cache@v2
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-cargo-${{ github.job }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
+        # Install a recent version of clang that supports wasm32
+      - run: wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - || exit 1
+      - run: sudo apt-add-repository "deb http://apt.llvm.org/focal/ llvm-toolchain-focal-10 main" || exit 1
+      - run: sudo apt-get update || exit 1
+      - run: sudo apt-get install -y libclang-common-10-dev clang-10 libc6-dev-i386 || exit 1
+      - name: Set default toolchain
+        run: rustup default 1.65.0 # STABLE
+      - name: Set profile
+        run: rustup set profile minimal
+      - name: Add target wasm32
+        run: rustup target add wasm32-unknown-unknown
+      - name: Update toolchain
+        run: rustup update
+      - name: Check
+        run: cargo check --target wasm32-unknown-unknown --features async-interface,use-esplora-async,dev-getrandom-wasm --no-default-features
+
+  fmt:
+    name: Rust fmt
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Set default toolchain
+        run: rustup default nightly
+      - name: Set profile
+        run: rustup set profile minimal
+      - name: Add rustfmt
+        run: rustup component add rustfmt
+      - name: Update toolchain
+        run: rustup update
+      - name: Check fmt
+        run: cargo fmt --all -- --config format_code_in_doc_comments=true --check
+
+  test_hardware_wallet:
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        rust:
+          - version: 1.65.0 # STABLE
+          - version: 1.57.0 # MSRV
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v3
+    - name: Build simulator image
+      run: docker build -t hwi/ledger_emulator ./ci -f ci/Dockerfile.ledger
+    - name: Run simulator image
+      run: docker run --name simulator --network=host hwi/ledger_emulator &
+    - name: Install Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.9'
+    - name: Install python dependencies
+      run: pip install hwi==2.1.1 protobuf==3.20.1
+    - name: Set default toolchain
+      run: rustup default ${{ matrix.rust.version }}
+    - name: Set profile
+      run: rustup set profile minimal
+    - name: Update toolchain
+      run: rustup update
+    - name: Pin dependencies for MSRV
+      if: matrix.rust.version == '1.57.0'
+      run: |
+        cargo update -p log --precise "0.4.18"
+          cargo update -p tempfile --precise "3.6.0"
+          cargo update -p hashlink --precise "0.8.1"
+          cargo update -p regex --precise "1.7.3"
+          cargo update -p zip:0.6.6 --precise "0.6.3"
+          cargo update -p rustix --precise "0.37.23"
+          cargo update -p tokio --precise "1.29.1"
+          cargo update -p tokio-util --precise "0.7.8"
+          cargo update -p cc --precise "1.0.81"
+          cargo update -p rustls:0.20.9 --precise "0.20.8"
+          cargo update -p rustls:0.21.7 --precise "0.21.1"
+          cargo update -p flate2:1.0.27 --precise "1.0.26"
+          cargo update -p reqwest --precise "0.11.18"
+          cargo update -p h2 --precise "0.3.20"
+          cargo update -p rustls-webpki:0.100.3 --precise "0.100.1"
+          cargo update -p rustls-webpki:0.101.6 --precise "0.101.1"
+    - name: Test
+      run: cargo test --features test-hardware-signer

.github/workflows/nightly_docs.yml

@@ -0,0 +1,69 @@
+name: Publish Nightly Docs
+
+on:
+  push:
+    branches:
+      - 'master'
+      - 'release/*'
+  pull_request:
+    branches:
+      - 'master'
+      - 'release/*'
+
+jobs:
+  build_docs:
+    name: Build docs
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v2
+      - name: Setup cache
+        uses: actions/cache@v2
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: nightly-docs-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
+      - name: Set default toolchain
+        run: rustup default nightly-2022-12-14
+      - name: Set profile
+        run: rustup set profile minimal
+      - name: Update toolchain
+        run: rustup update
+      - name: Build docs
+        run: cargo rustdoc --verbose --features=compiler,electrum,esplora,use-esplora-blocking,compact_filters,rpc,key-value-db,sqlite,all-keys,verify,hardware-signer -- --cfg docsrs -Dwarnings
+      - name: Upload artifact
+        uses: actions/upload-artifact@v2
+        with:
+          name: built-docs
+          path: ./target/doc/*
+
+  publish_docs:
+    name: 'Publish docs'
+    if: github.ref == 'refs/heads/master'
+    needs: [build_docs]
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout `bitcoindevkit.org`
+        uses: actions/checkout@v2
+        with:
+          ssh-key: ${{ secrets.DOCS_PUSH_SSH_KEY }}
+          repository: bitcoindevkit/bitcoindevkit.org
+          ref: master
+      - name: Create directories
+        run: mkdir -p ./docs/.vuepress/public/docs-rs/bdk/nightly
+      - name: Remove old latest
+        run: rm -rf ./docs/.vuepress/public/docs-rs/bdk/nightly/latest
+      - name: Download built docs
+        uses: actions/download-artifact@v1
+        with:
+          name: built-docs
+          path: ./docs/.vuepress/public/docs-rs/bdk/nightly/latest
+      - name: Configure git
+        run: git config user.email "github-actions@github.com" && git config user.name "github-actions"
+      - name: Commit
+        continue-on-error: true         # If there's nothing to commit this step fails, but it's fine
+        run: git add ./docs/.vuepress/public/docs-rs && git commit -m "Publish autogenerated nightly docs"
+      - name: Push
+        run: git push origin master

.gitignore

@@ -1,4 +1,6 @@
 /target
 Cargo.lock
+/.vscode
 
 *.swp
+.idea

.travis.yml

@@ -1,22 +0,0 @@
-language: rust
-rust:
-  - stable
-#  - 1.31.0
-#  - 1.22.0
-before_script:
-  - rustup component add rustfmt
-script:
-  - cargo fmt -- --check --verbose
-  - cargo test --verbose --all
-  - cargo build --verbose --all
-  - cargo build --verbose --no-default-features --features=minimal
-  - cargo build --verbose --no-default-features --features=minimal,esplora
-  - cargo build --verbose --no-default-features --features=key-value-db
-  - cargo build --verbose --no-default-features --features=electrum
-
-notifications:
-  email: false
-
-before_cache:
-  - rm -rf "$TRAVIS_HOME/.cargo/registry/src"
-cache: cargo

CHANGELOG.md

@@ -0,0 +1,696 @@
+# Changelog
+
+All notable changes to this project can be found here and in each release's git tag and can be viewed with `git tag -ln100 "v*"`. See also [DEVELOPMENT_CYCLE.md](DEVELOPMENT_CYCLE.md) for more details.
+
+Contributors do not need to change this file but do need to add changelog details in their PR descriptions. The person making the next release will collect changelog details from included PRs and edit this file prior to each release.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [v0.29.0]
+
+### Summary
+
+This maintenance release updates our `rust-bitcoin` dependency to 0.30.x and fixes a wallet balance bug when a wallet has more than one coinbase transaction.
+
+### Changed
+
+- Update rust-bitcoin to 0.30 #1071
+
+### Fixed
+
+- Fix a bug when syncing coinbase utxos on electrum #1090
+
+## [v0.28.2]
+
+### Summary
+
+Reverts the 0.28.1 esplora-client version update from 0.5.0 back to 0.4.0.
+
+## [v0.28.1]
+
+### Summary
+
+This patch release backports (from the BDK 1.0 dev branch) a fix for a bug in the policy condition calculation and adds a new taproot single key descriptor template (BIP-86). The policy condition calculation bug can cause issues when a policy subtree fails due to missing info even if it's not selected when creating a new transaction, errors on unused policy paths are now ignored.
+
+### Fixed
+
+- Backported #932 fix for policy condition calculation #1008
+
+### Added
+
+-  Backported #840 taproot descriptor template (BIP-86) #1033
+
+## [v0.28.0]
+
+### Summary
+
+Disable default-features for rust-bitcoin and rust-miniscript dependencies, and for rust-esplora-client optional dependency. New default `std` feature must be enabled unless building for wasm.
+
+### Changed
+
+- Bump bip39 crate to v2.0.0 #875
+- Set default-features = false for rust-bitcoin and rust-miniscript #882
+- Update esplora client dependency to version 0.4 #884
+- Added new `std` feature as part of default features #930
+
+## [v0.27.1]
+
+### Summary
+
+Fixes [RUSTSEC-2022-0090], this issue is only applicable if you are using the optional sqlite database feature.
+
+[RUSTSEC-2022-0090]: https://rustsec.org/advisories/RUSTSEC-2022-0090
+
+### Changed
+
+- Update optional sqlite dependency from 0.27.0 to 0.28.0. #867
+
+## [v0.27.0]
+
+### Summary
+
+A maintenance release with a bump in project MSRV to 1.57.0, updated dependence and a few developer oriented improvements. Improvements include  better error formatting, don't default to async/await for wasm32 and adding derived PartialEq and Eq on SyncTime.
+
+### Changed
+
+- Improve display error formatting #814
+- Don't default to use async/await on wasm32 #831
+- Project MSRV changed from 1.56.1 to 1.57.0 #842
+- Update rust-miniscript dependency to latest bug fix release 9.0 #844
+
+### Added
+
+- Derive PartialEq, Eq on SyncTime #837
+
+## [v0.26.0]
+
+### Summary
+
+This release improves Fulcrum electrum server compatibility and fixes public descriptor template key origin paths. We also snuck in small enhancements to configure the electrum client to validate the domain using SSL and sort TransactionDetails by block height and timestamp.
+  
+### Fixed
+  
+- Make electrum blockchain client `save_tx` function order independent to work with Fulcrum servers. #808
+- Fix wrong testnet key origin path in public descriptor templates. #818
+- Make README.md code examples compile without errors. #820
+
+### Changed
+
+- Bump `hwi` dependency to `0.4.0`. #825
+- Bump `esplora-client` dependency to `0.3` #830
+
+### Added
+  
+- For electrum blockchain client, allow user to configure whether to validate the domain using SSL. #805
+- Implement ordering for `TransactionDetails`. #812
+
+## [v0.25.0]
+
+### Summary
+
+This release fixes slow sync time and big script_pubkeys table with SQLite, the wallet rescan height for the FullyNodedExport and setting the network for keys in the KeyMap when using descriptor templates. Also added are new blockchain and mnemonic examples.
+  
+### Fixed
+  
+- Slow sync time and big script_pubkeys table with SQLite.
+- Wallet rescan height for the FullyNodedExport.
+- Setting the network for keys in the KeyMap when using descriptor templates.
+  
+### Added
+  
+- Examples for connecting to Esplora, Electrum Server, Neutrino and Bitcoin Core.
+- Example for using a mnemonic in a descriptors.
+
+## [v0.24.0]
+
+### Summary
+
+This release contains important dependency updates for `rust-bitcoin` to `0.29` and `rust-miniscript` to `8.0`, plus related crates that also depend on the latest version of `rust-bitcoin`. The release also includes a breaking change to the BDK signer which now produces low-R signatures by default, saving one byte. A bug was found in the `get_checksum` and `get_checksum_bytes` functions, which are now deprecated in favor of fixed versions called `calc_checksum` and `calc_checksum_bytes`. And finally a new `hardware-signer` features was added that re-exports the `hwi` crate, along with a new `hardware_signers.rs` example file.
+   
+### Changed
+
+- Updated dependency versions for `rust-bitcoin` to `0.29` and `rust-miniscript` to `8.0`, plus all related crates. @afilini #770
+- BDK Signer now produces low-R signatures by default, saving one byte. If you want to preserve the original behavior, set allow_grinding in the SignOptions to false. @vladimirfomene #779
+- Deprecated `get_checksum`and `get_checksum_bytes` due to bug where they calculates the checksum of a descriptor that already has a checksum.  Use `calc_checksum` and `calc_checksum_bytes` instead. @evanlinjin #765
+- Remove deprecated "address validators". @afilini #770
+  
+### Added
+
+- New `calc_checksum` and `calc_checksum_bytes`, replace deprecated `get_checksum` and `get_checksum_bytes`. @evanlinjin #765
+- Re-export the hwi crate when the feature hardware-signer is on.  @danielabrozzoni #758
+- New examples/hardware_signer.rs. @danielabrozzoni #758
+- Make psbt module public to expose PsbtUtils trait to downstream projects. @notmandatory #782
+
+## [v0.23.0]
+
+### Summary
+
+This release brings new utilities functions on PSBTs like `fee_amount()` and `fee_rate()` and migrates BDK to use our new external esplora client library.
+As always many bug fixes, docs and tests improvement are also included.
+
+### Changed
+
+- Update electrum-client to 0.11.0 by @afilini in https://github.com/bitcoindevkit/bdk/pull/737
+- Change configs for source-base code coverage by @wszdexdrf in https://github.com/bitcoindevkit/bdk/pull/708
+- Improve docs regarding PSBT finalization by @tnull in https://github.com/bitcoindevkit/bdk/pull/753
+- Update compiler example to a Policy example by @rajarshimaitra in https://github.com/bitcoindevkit/bdk/pull/730
+- Fix the release process by @afilini in https://github.com/bitcoindevkit/bdk/pull/754
+- Remove redundant duplicated keys check by @afilini in https://github.com/bitcoindevkit/bdk/pull/761
+- Remove genesis_block lazy initialization by @shobitb in https://github.com/bitcoindevkit/bdk/pull/756
+- Fix `Wallet::descriptor_checksum` to actually return the checksum by @evanlinjin in https://github.com/bitcoindevkit/bdk/pull/763
+- Use the esplora client crate by @afilini in https://github.com/bitcoindevkit/bdk/pull/764
+
+### Added
+
+- Run code coverage on every PR by @danielabrozzoni in https://github.com/bitcoindevkit/bdk/pull/747
+- Add psbt_signer.rs example by @notmandatory in https://github.com/bitcoindevkit/bdk/pull/744
+- Add fee_amount() and fee_rate() functions to PsbtUtils trait by @notmandatory in https://github.com/bitcoindevkit/bdk/pull/728
+- Add tests to improve coverage by @vladimirfomene in https://github.com/bitcoindevkit/bdk/pull/745
+- Enable signing taproot transactions with only `non_witness_utxos` by @afilini in https://github.com/bitcoindevkit/bdk/pull/757
+- Add datatype for is_spent sqlite column by @vladimirfomene in https://github.com/bitcoindevkit/bdk/pull/713
+- Add vscode filter to gitignore by @evanlinjin in https://github.com/bitcoindevkit/bdk/pull/762
+
+## [v0.22.0]
+
+### Summary
+
+This release brings support for hardware signers on desktop through the HWI library.
+It also includes fixes and improvements which are part of our ongoing effort of integrating
+BDK and LDK together.
+
+### Changed
+
+- FeeRate function name as_sat_vb to as_sat_per_vb. #678
+- Verify signatures after signing. #718
+- Dependency electrum-client to 0.11.0. #737
+
+### Added
+  
+- Functions to create FeeRate from sats/kvbytes and sats/kwu. #678
+- Custom hardware wallet signer HwiSigner in wallet::hardwaresigner module. #682
+- Function allow_dust on TxBuilder. #689
+- Implementation of Deref<Target=UrlClient> for EsploraBlockchain. #722
+- Implementation of Deref<Target=Client> for ElectrumBlockchain #705
+- Implementation of Deref<Target=Client> for RpcBlockchain. #731
+
+## [v0.21.0]
+
+- Add `descriptor::checksum::get_checksum_bytes` method.
+- Add `Excess` enum to handle remaining amount after coin selection.
+- Move change creation from `Wallet::create_tx` to `CoinSelectionAlgorithm::coin_select`.
+- Change the interface of `SqliteDatabase::new` to accept any type that implement AsRef<Path>
+- Add the ability to specify which leaves to sign in a taproot transaction through `TapLeavesOptions` in `SignOptions`
+- Add the ability to specify whether a taproot transaction should be signed using the internal key or not, using `sign_with_tap_internal_key` in `SignOptions`
+- Consolidate params `fee_amount` and `amount_needed` in `target_amount` in `CoinSelectionAlgorithm::coin_select` signature.
+- Change the meaning of the `fee_amount` field inside `CoinSelectionResult`: from now on the `fee_amount` will represent only the fees asociated with the utxos in the `selected` field of `CoinSelectionResult`.
+- New `RpcBlockchain` implementation with various fixes.
+- Return balance in separate categories, namely `confirmed`, `trusted_pending`, `untrusted_pending` & `immature`.
+
+## [v0.20.0]
+
+- New MSRV set to `1.56.1`
+- Fee sniping discouraging through nLockTime - if the user specifies a `current_height`, we use that as a nlocktime, otherwise we use the last sync height (or 0 if we never synced)
+- Fix hang when `ElectrumBlockchainConfig::stop_gap` is zero.
+- Set coin type in BIP44, BIP49, and BIP84 templates
+- Get block hash given a block height - A `get_block_hash` method is now defined on the `GetBlockHash` trait and implemented on every blockchain backend. This method expects a block height and returns the corresponding block hash.
+- Add `remove_partial_sigs` and `try_finalize` to `SignOptions`
+- Deprecate `AddressValidator`
+- Fix Electrum wallet sync potentially causing address index decrement - compare proposed index and current index before applying batch operations during sync.
+
+## [v0.19.0]
+
+- added `OldestFirstCoinSelection` impl to `CoinSelectionAlgorithm`
+- New MSRV set to `1.56`
+- Unpinned tokio to `1`
+- Add traits to reuse `Blockchain`s across multiple wallets (`BlockchainFactory` and `StatelessBlockchain`).
+- Upgrade to rust-bitcoin `0.28`
+- If using the `sqlite-db` feature all cached wallet data is deleted due to a possible UTXO inconsistency, a wallet.sync will recreate it
+- Update `PkOrF` in the policy module to become an enum
+- Add experimental support for Taproot, including:
+  - Support for `tr()` descriptors with complex tapscript trees
+  - Creation of Taproot PSBTs (BIP-371)
+  - Signing Taproot PSBTs (key spend and script spend)
+  - Support for `tr()` descriptors in the `descriptor!()` macro
+- Add support for Bitcoin Core 23.0 when using the `rpc` blockchain
+
+## [v0.18.0]
+
+- Add `sqlite-bundled` feature for deployments that need a bundled version of sqlite, i.e. for mobile platforms.
+- Added `Wallet::get_signers()`, `Wallet::descriptor_checksum()` and `Wallet::get_address_validators()`, exposed the `AsDerived` trait.
+- Deprecate `database::Database::flush()`, the function is only needed for the sled database on mobile, instead for mobile use the sqlite database.
+- Add `keychain: KeychainKind` to `wallet::AddressInfo`.
+- Improve key generation traits
+- Rename `WalletExport` to `FullyNodedExport`, deprecate the former.
+- Bump `miniscript` dependency version to `^6.1`.
+
+## [v0.17.0]
+
+- Removed default verification from `wallet::sync`. sync-time verification is added in `script_sync` and is activated by `verify` feature flag.
+- `verify` flag removed from `TransactionDetails`.
+- Add `get_internal_address` to allow you to get internal addresses just as you get external addresses.
+- added `ensure_addresses_cached` to `Wallet` to let offline wallets load and cache addresses in their database
+- Add `is_spent` field to `LocalUtxo`; when we notice that a utxo has been spent we set `is_spent` field to true instead of deleting it from the db.
+
+### Sync API change
+
+To decouple the `Wallet` from the `Blockchain` we've made major changes:
+
+- Removed `Blockchain` from Wallet.
+- Removed `Wallet::broadcast` (just use `Blockchain::broadcast`)
+- Deprecated `Wallet::new_offline` (all wallets are offline now)
+- Changed `Wallet::sync` to take a `Blockchain`.
+- Stop making a request for the block height when calling `Wallet:new`.
+- Added `SyncOptions` to capture extra (future) arguments to `Wallet::sync`.
+- Removed `max_addresses` sync parameter which determined how many addresses to cache before syncing since this can just be done with `ensure_addresses_cached`.
+- remove `flush` method from the `Database` trait.
+
+## [v0.16.1]
+
+- Pin tokio dependency version to ~1.14 to prevent errors due to their new MSRV 1.49.0
+
+## [v0.16.0]
+
+- Disable `reqwest` default features.
+- Added `reqwest-default-tls` feature: Use this to restore the TLS defaults of reqwest if you don't want to add a dependency to it in your own manifest.
+- Use dust_value from rust-bitcoin
+- Fixed generating WIF in the correct network format.
+
+## [v0.15.0]
+
+- Overhauled sync logic for electrum and esplora.
+- Unify ureq and reqwest esplora backends to have the same configuration parameters. This means reqwest now has a timeout parameter and ureq has a concurrency parameter.
+- Fixed esplora fee estimation.
+
+## [v0.14.0]
+
+- BIP39 implementation dependency, in `keys::bip39` changed from tiny-bip39 to rust-bip39.
+- Add new method on the `TxBuilder` to embed data in the transaction via `OP_RETURN`. To allow that a fix to check the dust only on spendable output has been introduced.
+- Update the `Database` trait to store the last sync timestamp and block height
+- Rename `ConfirmationTime` to `BlockTime`
+
+## [v0.13.0]
+
+- Exposed `get_tx()` method from `Database` to `Wallet`.
+
+## [v0.12.0]
+
+- Activate `miniscript/use-serde` feature to allow consumers of the library to access it via the re-exported `miniscript` crate.
+- Add support for proxies in `EsploraBlockchain`
+- Added `SqliteDatabase` that implements `Database` backed by a sqlite database using `rusqlite` crate.
+
+## [v0.11.0]
+
+- Added `flush` method to the `Database` trait to explicitly flush to disk latest changes on the db.
+
+## [v0.10.0]
+
+- Added `RpcBlockchain` in the `AnyBlockchain` struct to allow using Rpc backend where `AnyBlockchain` is used (eg `bdk-cli`)
+- Removed hard dependency on `tokio`.
+
+### Wallet
+
+- Removed and replaced `set_single_recipient` with more general `drain_to` and replaced `maintain_single_recipient` with `allow_shrinking`.
+
+### Blockchain
+
+- Removed `stop_gap` from `Blockchain` trait and added it to only `ElectrumBlockchain` and `EsploraBlockchain` structs.
+- Added a `ureq` backend for use when not using feature `async-interface` or target WASM. `ureq` is a blocking HTTP client.
+
+## [v0.9.0]
+
+### Wallet
+
+- Added Bitcoin core RPC added as blockchain backend
+- Added a `verify` feature that can be enable to verify the unconfirmed txs we download against the consensus rules
+
+## [v0.8.0]
+
+### Wallet
+- Added an option that must be explicitly enabled to allow signing using non-`SIGHASH_ALL` sighashes (#350)
+#### Changed
+`get_address` now returns an `AddressInfo` struct that includes the index and derefs to `Address`.
+
+## [v0.7.0]
+
+### Policy
+#### Changed
+Removed `fill_satisfaction` method in favor of enum parameter in `extract_policy` method
+
+#### Added
+Timelocks are considered (optionally) in building the `satisfaction` field
+
+### Wallet
+
+- Changed `Wallet::{sign, finalize_psbt}` now take a `&mut psbt` rather than consuming it.
+- Require and validate `non_witness_utxo` for SegWit signatures by default, can be adjusted with `SignOptions`
+- Replace the opt-in builder option `force_non_witness_utxo` with the opposite `only_witness_utxo`. From now on we will provide the `non_witness_utxo`, unless explicitly asked not to.
+
+## [v0.6.0]
+
+### Misc
+#### Changed
+- New minimum supported rust version is 1.46.0
+- Changed `AnyBlockchainConfig` to use serde tagged representation.
+
+### Descriptor
+#### Added
+- Added ability to analyze a `PSBT` to check which and how many signatures are already available
+
+### Wallet
+#### Changed
+- `get_new_address()` refactored to `get_address(AddressIndex::New)` to support different `get_address()` index selection strategies
+
+#### Added
+- Added `get_address(AddressIndex::LastUnused)` which returns the last derived address if it has not been used or if used in a received transaction returns a new address
+- Added `get_address(AddressIndex::Peek(u32))` which returns a derived address for a specified descriptor index but does not change the current index
+- Added `get_address(AddressIndex::Reset(u32))` which returns a derived address for a specified descriptor index and resets current index to the given value
+- Added `get_psbt_input` to create the corresponding psbt input for a local utxo.
+
+#### Fixed
+- Fixed `coin_select` calculation for UTXOs where `value < fee` that caused over-/underflow errors.
+
+## [v0.5.1]
+
+### Misc
+#### Changed
+- Pin `hyper` to `=0.14.4` to make it compile on Rust 1.45
+
+## [v0.5.0]
+
+### Misc
+#### Changed
+- Updated `electrum-client` to version `0.7`
+
+### Wallet
+#### Changed
+- `FeeRate` constructors `from_sat_per_vb` and `default_min_relay_fee` are now `const` functions
+
+## [v0.4.0]
+
+### Keys
+#### Changed
+- Renamed `DerivableKey::add_metadata()` to `DerivableKey::into_descriptor_key()`
+- Renamed `ToDescriptorKey::to_descriptor_key()` to `IntoDescriptorKey::into_descriptor_key()`
+#### Added
+- Added an `ExtendedKey` type that is an enum of `bip32::ExtendedPubKey` and `bip32::ExtendedPrivKey`
+- Added `DerivableKey::into_extended_key()` as the only method that needs to be implemented
+
+### Misc
+#### Removed
+- Removed the `parse_descriptor` example, since it wasn't demonstrating any bdk-specific API anymore.
+#### Changed
+- Updated `bitcoin` to `0.26`, `miniscript` to `5.1` and `electrum-client` to `0.6`
+#### Added
+- Added support for the `signet` network (issue #62)
+- Added a function to get the version of BDK at runtime
+
+### Wallet
+#### Changed
+- Removed the explicit `id` argument from `Wallet::add_signer()` since that's now part of `Signer` itself
+- Renamed `ToWalletDescriptor::to_wallet_descriptor()` to `IntoWalletDescriptor::into_wallet_descriptor()`
+
+### Policy
+#### Changed
+- Removed unneeded `Result<(), PolicyError>` return type for `Satisfaction::finalize()`
+- Removed the `TooManyItemsSelected` policy error (see commit message for more details)
+
+## [v0.3.0]
+
+### Descriptor
+#### Changed
+- Added an alias `DescriptorError` for `descriptor::error::Error`
+- Changed the error returned by `descriptor!()` and `fragment!()` to `DescriptorError`
+- Changed the error type in `ToWalletDescriptor` to `DescriptorError`
+- Improved checks on descriptors built using the macros
+
+### Blockchain
+#### Changed
+- Remove `BlockchainMarker`, `OfflineClient` and `OfflineWallet` in favor of just using the unit
+  type to mark for a missing client.
+- Upgrade `tokio` to `1.0`.
+
+### Transaction Creation Overhaul
+
+The `TxBuilder` is now created from the `build_tx` or `build_fee_bump` functions on wallet and the
+final transaction is created by calling `finish` on the builder.
+
+- Removed `TxBuilder::utxos` in favor of `TxBuilder::add_utxos`
+- Added `Wallet::build_tx` to replace `Wallet::create_tx`
+- Added `Wallet::build_fee_bump` to replace `Wallet::bump_fee`
+- Added `Wallet::get_utxo`
+- Added `Wallet::get_descriptor_for_keychain`
+
+### `add_foreign_utxo`
+
+- Renamed `UTXO` to `LocalUtxo`
+- Added `WeightedUtxo` to replace floating `(UTXO, usize)`.
+- Added `Utxo` enum to incorporate both local utxos and foreign utxos
+- Added `TxBuilder::add_foreign_utxo` which allows adding a utxo external to the wallet.
+
+### CLI
+#### Changed
+- Remove `cli.rs` module, `cli-utils` feature and `repl.rs` example; moved to new [`bdk-cli`](https://github.com/bitcoindevkit/bdk-cli) repository
+
+## [v0.2.0]
+
+### Project
+#### Added
+- Add CONTRIBUTING.md
+- Add a Discord badge to the README
+- Add code coverage github actions workflow
+- Add scheduled audit check in CI
+- Add CHANGELOG.md
+
+#### Changed
+- Rename the library to `bdk`
+- Rename `ScriptType` to `KeychainKind`
+- Prettify README examples on github
+- Change CI to github actions
+- Bump rust-bitcoin to 0.25, fix Cargo dependencies
+- Enable clippy for stable and tests by default
+- Switch to "mainline" rust-miniscript
+- Generate a different cache key for every CI job
+- Fix to at least bitcoin ^0.25.2
+
+#### Fixed
+- Fix or ignore clippy warnings for all optional features except compact_filters
+- Pin cc version because last breaks rocksdb build
+
+### Blockchain
+#### Added
+- Add a trait to create `Blockchain`s from a configuration
+- Add an `AnyBlockchain` enum to allow switching at runtime
+- Document `AnyBlockchain` and `ConfigurableBlockchain`
+- Use our Instant struct to be compatible with wasm
+- Make esplora call in parallel
+- Allow to set concurrency in Esplora config and optionally pass it in repl
+
+#### Fixed
+- Fix receiving a coinbase using Electrum/Esplora
+- Use proper type for EsploraHeader, make conversion to BlockHeader infallible
+- Eagerly unwrap height option, save one collect
+
+#### Changed
+- Simplify the architecture of blockchain traits
+- Improve sync
+- Remove unused varaint HeaderParseFail
+
+### CLI
+#### Added
+- Conditionally remove cli args according to enabled feature
+
+#### Changed
+- Add max_addresses param in sync
+- Split the internal and external policy paths
+
+### Database
+#### Added
+- Add `AnyDatabase` and `ConfigurableDatabase` traits
+
+### Descriptor
+#### Added
+- Add a macro to write descriptors from code
+- Add descriptor templates, add `DerivableKey`
+- Add ToWalletDescriptor trait tests
+- Add support for `sortedmulti` in `descriptor!`
+- Add ExtractPolicy trait tests
+- Add get_checksum tests, cleanup tests
+- Add descriptor macro tests
+
+#### Changes
+- Improve the descriptor macro, add traits for key and descriptor types
+
+#### Fixes
+- Fix the recovery of a descriptor given a PSBT
+
+### Keys
+#### Added
+- Add BIP39 support
+- Take `ScriptContext` into account when converting keys
+- Add a way to restrict the networks in which keys are valid
+- Add a trait for keys that can be generated
+- Fix entropy generation
+- Less convoluted entropy generation
+- Re-export tiny-bip39
+- Implement `GeneratableKey` trait for `bitcoin::PrivateKey`
+- Implement `ToDescriptorKey` trait for `GeneratedKey`
+- Add a shortcut to generate keys with the default options
+
+#### Fixed
+- Fix all-keys and cli-utils tests
+
+### Wallet
+#### Added
+- Allow to define static fees for transactions Fixes #137
+- Merging two match expressions for fee calculation
+- Incorporate RBF rules into utxo selection function
+- Add Branch and Bound coin selection
+- Add tests for BranchAndBoundCoinSelection::coin_select
+- Add tests for BranchAndBoundCoinSelection::bnb
+- Add tests for BranchAndBoundCoinSelection::single_random_draw
+- Add test that shwpkh populates witness_utxo
+- Add witness and redeem scripts to PSBT outputs
+- Add an option to include `PSBT_GLOBAL_XPUB`s in PSBTs
+- Eagerly finalize inputs
+
+#### Changed
+- Use collect to avoid iter unwrapping Options
+- Make coin_select take may/must use utxo lists
+- Improve `CoinSelectionAlgorithm`
+- Refactor `Wallet::bump_fee()`
+- Default to SIGHASH_ALL if not specified
+- Replace ChangeSpendPolicy::filter_utxos with a predicate
+- Make 'unspendable' into a HashSet
+- Stop implicitly enforcing manaul selection by .add_utxo
+- Rename DumbCS to LargestFirstCoinSelection
+- Rename must_use_utxos to required_utxos
+- Rename may_use_utxos to optional_uxtos
+- Rename get_must_may_use_utxos to preselect_utxos
+- Remove redundant Box around address validators
+- Remove redundant Box around signers
+- Make Signer and AddressValidator Send and Sync
+- Split `send_all` into `set_single_recipient` and `drain_wallet`
+- Use TXIN_DEFAULT_WEIGHT constant in coin selection
+- Replace `must_use` with `required` in coin selection
+- Take both spending policies into account in create_tx
+- Check last derivation in cache to avoid recomputation
+- Use the branch-and-bound cs by default
+- Make coin_select return UTXOs instead of TxIns
+- Build output lookup inside complete transaction
+- Don't wrap SignersContainer arguments in Arc
+- More consistent references with 'signers' variables
+
+#### Fixed
+- Fix signing for `ShWpkh` inputs
+- Fix the recovery of a descriptor given a PSBT
+
+### Examples
+#### Added
+- Support esplora blockchain source in repl
+
+#### Changed
+- Revert back the REPL example to use Electrum
+- Remove the `magic` alias for `repl`
+- Require esplora feature for repl example
+
+#### Security
+- Use dirs-next instead of dirs since the latter is unmantained
+
+## [0.1.0-beta.1] - 2020-09-08
+
+### Blockchain
+#### Added
+- Lightweight Electrum client with SSL/SOCKS5 support
+- Add a generalized "Blockchain" interface
+- Add Error::OfflineClient
+- Add the Esplora backend
+- Use async I/O in the various blockchain impls
+- Compact Filters blockchain implementation
+- Add support for Tor
+- Impl OnlineBlockchain for types wrapped in Arc
+
+### Database
+#### Added
+- Add a generalized database trait and a Sled-based implementation
+- Add an in-memory database
+
+### Descriptor
+#### Added
+- Wrap Miniscript descriptors to support xpubs
+- Policy and contribution
+- Transform a descriptor into its "public" version
+- Use `miniscript::DescriptorPublicKey`
+
+### Macros
+#### Added
+- Add a feature to enable the async interface on non-wasm32 platforms
+
+### Wallet
+#### Added
+- Wallet logic
+- Add `assume_height_reached` in PSBTSatisfier
+- Add an option to change the assumed current height
+- Specify the policy branch with a map
+- Add a few commands to handle psbts
+- Add hd_keypaths to outputs
+- Add a `TxBuilder` struct to simplify `create_tx()`'s interface
+- Abstract coin selection in a separate trait
+- Refill the address pool whenever necessary
+- Implement the wallet import/export format from FullyNoded
+- Add a type convert fee units, add `Wallet::estimate_fee()`
+- TxOrdering, shuffle/bip69 support
+- Add RBF and custom versions in TxBuilder
+- Allow limiting the use of internal utxos in TxBuilder
+- Add `force_non_witness_utxo()` to TxBuilder
+- RBF and add a few tests
+- Add AddressValidators
+- Add explicit ordering for the signers
+- Support signing the whole tx instead of individual inputs
+- Create a PSBT signer from an ExtendedDescriptor
+
+### Examples
+#### Added
+- Add REPL broadcast command
+- Add a miniscript compiler CLI
+- Expose list_transactions() in the REPL
+- Use `MemoryDatabase` in the compiler example
+- Make the REPL return JSON
+
+[0.1.0-beta.1]: https://github.com/bitcoindevkit/bdk/compare/96c87ea5...0.1.0-beta.1
+[v0.2.0]: https://github.com/bitcoindevkit/bdk/compare/0.1.0-beta.1...v0.2.0
+[v0.3.0]: https://github.com/bitcoindevkit/bdk/compare/v0.2.0...v0.3.0
+[v0.4.0]: https://github.com/bitcoindevkit/bdk/compare/v0.3.0...v0.4.0
+[v0.5.0]: https://github.com/bitcoindevkit/bdk/compare/v0.4.0...v0.5.0
+[v0.5.1]: https://github.com/bitcoindevkit/bdk/compare/v0.5.0...v0.5.1
+[v0.6.0]: https://github.com/bitcoindevkit/bdk/compare/v0.5.1...v0.6.0
+[v0.7.0]: https://github.com/bitcoindevkit/bdk/compare/v0.6.0...v0.7.0
+[v0.8.0]: https://github.com/bitcoindevkit/bdk/compare/v0.7.0...v0.8.0
+[v0.9.0]: https://github.com/bitcoindevkit/bdk/compare/v0.8.0...v0.9.0
+[v0.10.0]: https://github.com/bitcoindevkit/bdk/compare/v0.9.0...v0.10.0
+[v0.11.0]: https://github.com/bitcoindevkit/bdk/compare/v0.10.0...v0.11.0
+[v0.12.0]: https://github.com/bitcoindevkit/bdk/compare/v0.11.0...v0.12.0
+[v0.13.0]: https://github.com/bitcoindevkit/bdk/compare/v0.12.0...v0.13.0
+[v0.14.0]: https://github.com/bitcoindevkit/bdk/compare/v0.13.0...v0.14.0
+[v0.15.0]: https://github.com/bitcoindevkit/bdk/compare/v0.14.0...v0.15.0
+[v0.16.0]: https://github.com/bitcoindevkit/bdk/compare/v0.15.0...v0.16.0
+[v0.16.1]: https://github.com/bitcoindevkit/bdk/compare/v0.16.0...v0.16.1
+[v0.17.0]: https://github.com/bitcoindevkit/bdk/compare/v0.16.1...v0.17.0
+[v0.18.0]: https://github.com/bitcoindevkit/bdk/compare/v0.17.0...v0.18.0
+[v0.19.0]: https://github.com/bitcoindevkit/bdk/compare/v0.18.0...v0.19.0
+[v0.20.0]: https://github.com/bitcoindevkit/bdk/compare/v0.19.0...v0.20.0
+[v0.21.0]: https://github.com/bitcoindevkit/bdk/compare/v0.20.0...v0.21.0
+[v0.22.0]: https://github.com/bitcoindevkit/bdk/compare/v0.21.0...v0.22.0
+[v0.23.0]: https://github.com/bitcoindevkit/bdk/compare/v0.22.0...v0.23.0
+[v0.24.0]: https://github.com/bitcoindevkit/bdk/compare/v0.23.0...v0.24.0
+[v0.25.0]: https://github.com/bitcoindevkit/bdk/compare/v0.24.0...v0.25.0
+[v0.26.0]: https://github.com/bitcoindevkit/bdk/compare/v0.25.0...v0.26.0
+[v0.27.0]: https://github.com/bitcoindevkit/bdk/compare/v0.26.0...v0.27.0
+[v0.27.1]: https://github.com/bitcoindevkit/bdk/compare/v0.27.0...v0.27.1
+[v0.28.0]: https://github.com/bitcoindevkit/bdk/compare/v0.27.1...v0.28.0
+[v0.28.1]: https://github.com/bitcoindevkit/bdk/compare/v0.28.0...v0.28.1
+[v0.28.2]: https://github.com/bitcoindevkit/bdk/compare/v0.28.1...v0.28.2
+[v0.29.0]: https://github.com/bitcoindevkit/bdk/compare/v0.28.2...v0.29.0
+[Unreleased]: https://github.com/bitcoindevkit/bdk/compare/v0.29.0...HEAD

CONTRIBUTING.md

@@ -0,0 +1,118 @@
+Contributing to BDK
+==============================
+
+The BDK project operates an open contributor model where anyone is welcome to
+contribute towards development in the form of peer review, documentation,
+testing and patches.
+
+Anyone is invited to contribute without regard to technical experience,
+"expertise", OSS experience, age, or other concern. However, the development of
+cryptocurrencies demands a high-level of rigor, adversarial thinking, thorough
+testing and risk-minimization.
+Any bug may cost users real money. That being said, we deeply welcome people
+contributing for the first time to an open source project or picking up Rust while
+contributing. Don't be shy, you'll learn.
+
+Communications Channels
+-----------------------
+
+Communication about BDK happens primarily on the [BDK Discord](https://discord.gg/dstn4dQ).
+
+Discussion about code base improvements happens in GitHub [issues](https://github.com/bitcoindevkit/bdk/issues) and
+on [pull requests](https://github.com/bitcoindevkit/bdk/pulls).
+
+Contribution Workflow
+---------------------
+
+The codebase is maintained using the "contributor workflow" where everyone
+without exception contributes patch proposals using "pull requests". This
+facilitates social contribution, easy testing and peer review.
+
+To contribute a patch, the worflow is a as follows:
+
+  1. Fork Repository
+  2. Create topic branch
+  3. Commit patches
+
+In general commits should be atomic and diffs should be easy to read.
+For this reason do not mix any formatting fixes or code moves with actual code
+changes. Further, each commit, individually, should compile and pass tests, in
+order to ensure git bisect and other automated tools function properly.
+
+When adding a new feature, thought must be given to the long term technical
+debt.
+Every new feature should be covered by functional tests where possible.
+
+When refactoring, structure your PR to make it easy to review and don't
+hesitate to split it into multiple small, focused PRs.
+
+The Minimal Supported Rust Version is 1.46 (enforced by our CI).
+
+Commits should cover both the issue fixed and the solution's rationale.
+These [guidelines](https://chris.beams.io/posts/git-commit/) should be kept in mind.
+
+To facilitate communication with other contributors, the project is making use
+of GitHub's "assignee" field. First check that no one is assigned and then
+comment suggesting that you're working on it. If someone is already assigned,
+don't hesitate to ask if the assigned party or previous commenters are still
+working on it if it has been awhile.
+
+Deprecation policy
+------------------
+
+Where possible, breaking existing APIs should be avoided. Instead, add new APIs and
+use [`#[deprecated]`](https://github.com/rust-lang/rfcs/blob/master/text/1270-deprecation.md)
+to discourage use of the old one.
+
+Deprecated APIs are typically maintained for one release cycle. In other words, an
+API that has been deprecated with the 0.10 release can be expected to be removed in the
+0.11 release. This allows for smoother upgrades without incurring too much technical
+debt inside this library.
+
+If you deprecated an API as part of a contribution, we encourage you to "own" that API
+and send a follow-up to remove it as part of the next release cycle.
+
+Peer review
+-----------
+
+Anyone may participate in peer review which is expressed by comments in the
+pull request. Typically reviewers will review the code for obvious errors, as
+well as test out the patch set and opine on the technical merits of the patch.
+PR should be reviewed first on the conceptual level before focusing on code
+style or grammar fixes.
+
+Coding Conventions
+------------------
+
+This codebase uses spaces, not tabs.
+Use `cargo fmt` with the default settings to format code before committing.
+This is also enforced by the CI.
+
+Security
+--------
+
+Security is a high priority of BDK; disclosure of security vulnerabilites helps
+prevent user loss of funds.
+
+Note that BDK is currently considered "pre-production" during this time, there
+is no special handling of security issues. Please simply open an issue on
+Github.
+
+Testing
+-------
+
+Related to the security aspect, BDK developers take testing very seriously.
+Due to the modular nature of the project, writing new functional tests is easy
+and good test coverage of the codebase is an important goal.
+Refactoring the project to enable fine-grained unit testing is also an ongoing
+effort.
+
+Going further
+-------------
+
+You may be interested by Jon Atacks guide on [How to review Bitcoin Core PRs](https://github.com/jonatack/bitcoin-development/blob/master/how-to-review-bitcoin-core-prs.md)
+and [How to make Bitcoin Core PRs](https://github.com/jonatack/bitcoin-development/blob/master/how-to-make-bitcoin-core-prs.md).
+While there are differences between the projects in terms of context and
+maturity, many of the suggestions offered apply to this project.
+
+Overall, have fun :)

Cargo.toml

@@ -1,50 +1,122 @@
 [package]
-name = "magical-bitcoin-wallet"
-version = "0.1.0"
+name = "bdk"
+version = "0.29.0"
 edition = "2018"
-authors = ["Riccardo Casatta <riccardo@casatta.it>", "Alekos Filini <alekos.filini@gmail.com>"]
+authors = ["Alekos Filini <alekos.filini@gmail.com>", "Riccardo Casatta <riccardo@casatta.it>"]
+homepage = "https://bitcoindevkit.org"
+repository = "https://github.com/bitcoindevkit/bdk"
+documentation = "https://docs.rs/bdk"
+description = "A modern, lightweight, descriptor-based wallet library"
+keywords = ["bitcoin", "wallet", "descriptor", "psbt"]
+readme = "README.md"
+license = "MIT OR Apache-2.0"
 
 [dependencies]
-log = "^0.4"
-bitcoin = { version = "0.23", features = ["use-serde"] }
-miniscript = { version = "0.12" }
+bdk-macros = "^0.6"
+log = "0.4"
+miniscript = { version = "10.0", default-features = false, features = ["serde"] }
+bitcoin = { version = "0.30", default-features = false, features = ["serde", "base64", "rand-std"] }
 serde = { version = "^1.0", features = ["derive"] }
 serde_json = { version = "^1.0" }
-base64 = "^0.11"
-async-trait = "0.1"
+rand = "^0.8"
 
 # Optional dependencies
-sled = { version = "0.31.0", optional = true }
-electrum-client = { git = "https://github.com/MagicalBitcoin/rust-electrum-client.git", optional = true }
-reqwest = { version = "0.10", optional = true, features = ["json"] }
+sled = { version = "0.34", optional = true }
+electrum-client = { version = "0.18", optional = true }
+esplora-client = { version = "0.6", default-features = false, optional = true }
+rusqlite = { version = "0.28.0", optional = true }
+ahash = { version = "0.7.6", optional = true }
 futures = { version = "0.3", optional = true }
-clap = { version = "2.33", optional = true }
+async-trait = { version = "0.1", optional = true }
+rocksdb = { version = "0.14", default-features = false, features = ["snappy"], optional = true }
+cc = { version = ">=1.0.64", optional = true }
+socks = { version = "0.3", optional = true }
+hwi = { version = "0.7", optional = true, features = ["miniscript"] }
+
+bip39 = { version = "2.0.0", optional = true }
+bitcoinconsensus = { version = "0.19.0-3", optional = true }
+
+# Needed by bdk_blockchain_tests macro and the `rpc` feature
+bitcoincore-rpc = { package="core-rpc", version = "0.17", optional = true }
+
+# Platform-specific dependencies
+[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
+tokio = { version = "1", features = ["rt", "macros"] }
+
+[target.'cfg(target_arch = "wasm32")'.dependencies]
+getrandom = "0.2"
+async-trait = "0.1"
+js-sys = "0.3"
 
 [features]
 minimal = []
 compiler = ["miniscript/compiler"]
-default = ["key-value-db", "electrum"]
-electrum = ["electrum-client"]
-esplora = ["reqwest", "futures"]
+verify = ["bitcoinconsensus"]
+default = ["std", "key-value-db", "electrum"]
+# std feature is always required unless building for wasm32-unknown-unknown target
+# if building for wasm user must add dependencies bitcoin/no-std,miniscript/no-std
+std = ["bitcoin/std", "miniscript/std"]
+sqlite = ["rusqlite", "ahash"]
+sqlite-bundled = ["sqlite", "rusqlite/bundled"]
+compact_filters = ["rocksdb", "socks", "cc"]
 key-value-db = ["sled"]
-cli-utils = ["clap"]
-multiparty = []
+all-keys = ["keys-bip39"]
+keys-bip39 = ["bip39"]
+rpc = ["bitcoincore-rpc"]
+hardware-signer = ["hwi"]
+
+# We currently provide mulitple implementations of `Blockchain`, all are
+# blocking except for the `EsploraBlockchain` which can be either async or
+# blocking, depending on the HTTP client in use.
+#
+# - Users wanting asynchronous HTTP calls should enable `async-interface` to get
+#   access to the asynchronous method implementations. Then, if Esplora is wanted,
+#   enable the `use-esplora-async` feature.
+# - Users wanting blocking HTTP calls can use any of the other blockchain
+#   implementations (`compact_filters`, `electrum`, or `esplora`). Users wanting to
+#   use Esplora should enable the `use-esplora-blocking` feature.
+#
+# WARNING: Please take care with the features below, various combinations will
+# fail to build. We cannot currently build `bdk` with `--all-features`.
+async-interface = ["async-trait"]
+electrum = ["electrum-client"]
+# MUST ALSO USE `--no-default-features`.
+use-esplora-async = ["esplora", "esplora-client/async", "futures"]
+use-esplora-blocking = ["esplora", "esplora-client/blocking"]
+# Deprecated aliases
+use-esplora-reqwest = ["use-esplora-async"]
+use-esplora-ureq = ["use-esplora-blocking"]
+# Typical configurations will not need to use `esplora` feature directly.
+esplora = []
+
+# Use below feature with `use-esplora-async` to enable reqwest default TLS support
+reqwest-default-tls = ["esplora-client/async-https"]
+
+# Debug/Test features
+test-blockchains = ["bitcoincore-rpc", "electrum-client"]
+test-electrum = ["electrum", "electrsd/electrs_0_8_10", "electrsd/bitcoind_22_0", "test-blockchains"]
+test-rpc = ["rpc", "electrsd/electrs_0_8_10", "electrsd/bitcoind_22_0", "test-blockchains"]
+test-rpc-legacy = ["rpc", "electrsd/electrs_0_8_10", "electrsd/bitcoind_0_20_0", "test-blockchains"]
+test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "electrsd/bitcoind_22_0", "test-blockchains"]
+test-md-docs = ["electrum"]
+test-hardware-signer = ["hardware-signer"]
+
+# This feature is used to run `cargo check` in our CI targeting wasm. It's not recommended
+# for libraries to explicitly include the "getrandom/js" feature, so we only do it when
+# necessary for running our CI. See: https://docs.rs/getrandom/0.2.8/getrandom/#webassembly-support
+dev-getrandom-wasm = ["getrandom/js"]
 
 [dev-dependencies]
-tokio = { version = "0.2", features = ["macros"] }
+miniscript = { version = "10.0", features = ["std"] }
+bitcoin = { version = "0.30", features = ["std"] }
 lazy_static = "1.4"
-rustyline = "6.0"
-dirs = "2.0"
-env_logger = "0.7"
-rand = "0.7"
+env_logger = { version = "0.7", default-features = false }
+electrsd = "0.24"
+assert_matches = "1.5.0"
 
 [[example]]
-name = "repl"
-required-features = ["cli-utils"]
-[[example]]
-name = "psbt"
-[[example]]
-name = "parse_descriptor"
+name = "compact_filters_balance"
+required-features = ["compact_filters"]
 
 [[example]]
 name = "miniscriptc"
@@ -52,12 +124,47 @@ path = "examples/compiler.rs"
 required-features = ["compiler"]
 
 [[example]]
-name = "multiparty"
-required-features = ["multiparty","compiler"]
+name = "policy"
+path = "examples/policy.rs"
 
-# Provide a more user-friendly alias for the REPL
 [[example]]
-name = "magic"
-path = "examples/repl.rs"
-required-features = ["cli-utils"]
+name = "rpcwallet"
+path = "examples/rpcwallet.rs"
+required-features = ["keys-bip39", "key-value-db", "rpc", "electrsd/bitcoind_22_0"]
 
+[[example]]
+name = "psbt_signer"
+path = "examples/psbt_signer.rs"
+required-features = ["electrum"]
+
+[[example]]
+name = "hardware_signer"
+path = "examples/hardware_signer.rs"
+required-features = ["electrum", "hardware-signer"]
+
+[[example]]
+name = "electrum_backend"
+path = "examples/electrum_backend.rs"
+required-features = ["electrum"]
+
+[[example]]
+name = "esplora_backend_synchronous"
+path = "examples/esplora_backend_synchronous.rs"
+required-features = ["use-esplora-ureq"]
+
+[[example]]
+name = "esplora_backend_asynchronous"
+path = "examples/esplora_backend_asynchronous.rs"
+required-features = ["use-esplora-reqwest", "reqwest-default-tls", "async-interface"]
+
+[[example]]
+name = "mnemonic_to_descriptors"
+path = "examples/mnemonic_to_descriptors.rs"
+required-features = ["all-keys"]
+
+[workspace]
+members = ["macros"]
+[package.metadata.docs.rs]
+features = ["compiler", "electrum", "esplora", "use-esplora-blocking", "compact_filters", "rpc", "key-value-db", "sqlite", "all-keys", "verify", "hardware-signer"]
+# defines the configuration attribute `docsrs`
+rustdoc-args = ["--cfg", "docsrs"]

DEVELOPMENT_CYCLE.md

@@ -0,0 +1,16 @@
+# Development Cycle
+
+This project follows a regular releasing schedule similar to the one [used by the Rust language]. In short, this means that a new release is made at a regular cadence, with all the feature/bugfixes that made it to `master` in time. This ensures that we don't keep delaying releases waiting for "just one more little thing".
+
+This project uses [Semantic Versioning], but is currently at MAJOR version zero (0.y.z) meaning it is still in initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable. Until we reach version `1.0.0` we will do our best to document any breaking API changes in the changelog info attached to each release tag.
+
+We decided to maintain a faster release cycle while the library is still in "beta", i.e. before release `1.0.0`: since we are constantly adding new features and, even more importantly, fixing issues, we want developers to have access to those updates as fast as possible. For this reason we will make a release **every 4 weeks**.
+
+Once the project reaches a more mature state (>= `1.0.0`), we will very likely switch to longer release cycles of **6 weeks**.
+
+The "feature freeze" will happen **one week before the release date**. This means a new branch will be created originating from the `master` tip at that time, and in that branch we will stop adding new features and only focus on ensuring the ones we've added are working properly.
+
+To create a new release a release manager will create a new issue using the `Release` template and follow the template instructions.
+
+[used by the Rust language]: https://doc.rust-lang.org/book/appendix-07-nightly-rust.html
+[Semantic Versioning]: https://semver.org/

LICENSE

@@ -1,21 +1,14 @@
-MIT License
+This software is licensed under [Apache 2.0](LICENSE-APACHE) or
+[MIT](LICENSE-MIT), at your option.
 
-Copyright (c) 2020 Magical Bitcoin
+Some files retain their own copyright notice, however, for full authorship
+information, see version control history.
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+Except as otherwise noted in individual files, all files in this repository are
+licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+http://www.apache.org/licenses/LICENSE-2.0> or the MIT license <LICENSE-MIT or
+http://opensource.org/licenses/MIT>, at your option.
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+You may not use, copy, modify, merge, publish, distribute, sublicense, and/or
+sell copies of this software or any files in this repository except in
+accordance with one or both of these licenses.

LICENSE-APACHE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

LICENSE-MIT

@@ -0,0 +1,16 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -1,7 +1,244 @@
-# Magical Bitcoin Wallet
+<div align="center">
+  <h1>BDK</h1>
 
-A modern, lightweight, descriptor-based wallet written in Rust!
+  <img src="./static/bdk.png" width="220" />
 
-## Getting Started
+  <p>
+    <strong>A modern, lightweight, descriptor-based wallet library written in Rust!</strong>
+  </p>
 
-See the documentation at [magicalbitcoin.org](https://magicalbitcoin.org)
+  <p>
+    <a href="https://crates.io/crates/bdk"><img alt="Crate Info" src="https://img.shields.io/crates/v/bdk.svg"/></a>
+    <a href="https://github.com/bitcoindevkit/bdk/blob/master/LICENSE"><img alt="MIT or Apache-2.0 Licensed" src="https://img.shields.io/badge/license-MIT%2FApache--2.0-blue.svg"/></a>
+    <a href="https://github.com/bitcoindevkit/bdk/actions?query=workflow%3ACI"><img alt="CI Status" src="https://github.com/bitcoindevkit/bdk/workflows/CI/badge.svg"></a>
+    <a href="https://coveralls.io/github/bitcoindevkit/bdk?branch=master"><img src="https://coveralls.io/repos/github/bitcoindevkit/bdk/badge.svg?branch=master"/></a>
+    <a href="https://docs.rs/bdk"><img alt="API Docs" src="https://img.shields.io/badge/docs.rs-bdk-green"/></a>
+    <a href="https://blog.rust-lang.org/2021/12/02/Rust-1.57.0.html"><img alt="Rustc Version 1.57.0+" src="https://img.shields.io/badge/rustc-1.57.0%2B-lightgrey.svg"/></a>
+    <a href="https://discord.gg/d7NkDKm"><img alt="Chat on Discord" src="https://img.shields.io/discord/753336465005608961?logo=discord"></a>
+  </p>
+
+  <h4>
+    <a href="https://bitcoindevkit.org">Project Homepage</a>
+    <span> | </span>
+    <a href="https://docs.rs/bdk">Documentation</a>
+  </h4>
+</div>
+
+## About
+
+The `bdk` library aims to be the core building block for Bitcoin wallets of any kind.
+
+* It uses [Miniscript](https://github.com/rust-bitcoin/rust-miniscript) to support descriptors with generalized conditions. This exact same library can be used to build
+  single-sig wallets, multisigs, timelocked contracts and more.
+* It supports multiple blockchain backends and databases, allowing developers to choose exactly what's right for their projects.
+* It's built to be cross-platform: the core logic works on desktop, mobile, and even WebAssembly.
+* It's very easy to extend: developers can implement customized logic for blockchain backends, databases, signers, coin selection, and more, without having to fork and modify this library.
+
+## Examples
+
+### Sync the balance of a descriptor
+
+```rust,no_run
+use bdk::Wallet;
+use bdk::database::MemoryDatabase;
+use bdk::blockchain::ElectrumBlockchain;
+use bdk::SyncOptions;
+use bdk::electrum_client::Client;
+use bdk::bitcoin::Network;
+
+fn main() -> Result<(), bdk::Error> {
+    let blockchain = ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
+    let wallet = Wallet::new(
+        "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
+        Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
+        Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!("Descriptor balance: {} SAT", wallet.get_balance()?);
+
+    Ok(())
+}
+```
+
+### Generate a few addresses
+
+```rust
+use bdk::{Wallet, database::MemoryDatabase};
+use bdk::wallet::AddressIndex::New;
+use bdk::bitcoin::Network;
+
+fn main() -> Result<(), bdk::Error> {
+    let wallet = Wallet::new(
+        "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
+        Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
+        Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    println!("Address #0: {}", wallet.get_address(New)?);
+    println!("Address #1: {}", wallet.get_address(New)?);
+    println!("Address #2: {}", wallet.get_address(New)?);
+
+    Ok(())
+}
+```
+
+### Create a transaction
+
+```rust,no_run
+use bdk::{FeeRate, Wallet, SyncOptions};
+use bdk::database::MemoryDatabase;
+use bdk::blockchain::ElectrumBlockchain;
+
+use bdk::electrum_client::Client;
+use bdk::wallet::AddressIndex::New;
+
+use bitcoin::base64;
+use bdk::bitcoin::consensus::serialize;
+use bdk::bitcoin::Network;
+
+fn main() -> Result<(), bdk::Error> {
+    let blockchain = ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
+    let wallet = Wallet::new(
+        "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
+        Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
+        Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    let send_to = wallet.get_address(New)?;
+    let (psbt, details) = {
+        let mut builder = wallet.build_tx();
+        builder
+            .add_recipient(send_to.script_pubkey(), 50_000)
+            .enable_rbf()
+            .do_not_spend_change()
+            .fee_rate(FeeRate::from_sat_per_vb(5.0));
+        builder.finish()?
+    };
+
+    println!("Transaction details: {:#?}", details);
+    println!("Unsigned PSBT: {}", base64::encode(psbt.serialize()));
+
+    Ok(())
+}
+```
+
+### Sign a transaction
+
+```rust,no_run
+use bdk::{Wallet, SignOptions, database::MemoryDatabase};
+
+use bitcoin::base64;
+use bdk::bitcoin::consensus::deserialize;
+use bdk::bitcoin::{psbt::Psbt, Network};
+
+fn main() -> Result<(), bdk::Error> {
+    let wallet = Wallet::new(
+        "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
+        Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
+        Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    let psbt = "...";
+    let mut psbt = Psbt::deserialize(&base64::decode(psbt).unwrap())?;
+
+    let _finalized = wallet.sign(&mut psbt, SignOptions::default())?;
+
+    Ok(())
+}
+```
+
+## Testing
+
+### Unit testing
+
+```bash
+cargo test
+```
+
+### Integration testing
+
+Integration testing require testing features, for example:
+
+```bash
+cargo test --features test-electrum
+```
+
+The other options are `test-esplora`, `test-rpc` or `test-rpc-legacy` which runs against an older version of Bitcoin Core.
+Note that `electrs` and `bitcoind` binaries are automatically downloaded (on mac and linux), to specify you already have installed binaries you must use `--no-default-features` and provide `BITCOIND_EXE` and `ELECTRS_EXE` as environment variables.
+
+## Running under WASM
+
+If you want to run this library under WASM you will probably have to add the following lines to you `Cargo.toml`:
+
+```toml
+[dependencies]
+getrandom = { version = "0.2", features = ["js"] }
+```
+
+This enables the `rand` crate to work in environments where JavaScript is available. See [this link](https://docs.rs/getrandom/0.2.8/getrandom/#webassembly-support) to learn more.
+
+## License
+
+Licensed under either of
+
+ * Apache License, Version 2.0
+   ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
+ * MIT license
+   ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
+
+at your option.
+
+## Contribution
+
+Unless you explicitly state otherwise, any contribution intentionally submitted
+for inclusion in the work by you, as defined in the Apache-2.0 license, shall be
+dual licensed as above, without any additional terms or conditions.
+
+## Minimum Supported Rust Version (MSRV)
+
+This library should compile with any combination of features with Rust 1.57.0.
+
+To build with the MSRV you will need to pin dependencies as follows:
+
+```shell
+# log 0.4.19 has MSRV 1.60.0
+cargo update -p log --precise "0.4.18"
+# tempfile 3.7.0 has MSRV 1.63.0
+cargo update -p tempfile --precise "3.6.0"
+# required for sqlite feature, hashlink 0.8.2 has MSRV 1.61.0
+cargo update -p hashlink --precise "0.8.1"
+# required for compact_filters feature, regex after 1.7.3 has MSRV 1.60.0
+cargo update -p regex --precise "1.7.3"
+# zip 0.6.3 has MSRV 1.59.0 but still works
+cargo update -p zip:0.6.6 --precise "0.6.3"
+# rustix 0.38.0 has MSRV 1.65.0
+cargo update -p rustix --precise "0.37.23"
+# tokio 1.30 has MSRV 1.63.0+
+cargo update -p tokio --precise "1.29.1"
+# tokio-util 0.7.9 doesn't build with MSRV 1.57.0
+cargo update -p tokio-util --precise "0.7.8"
+# cc 1.0.82 is throwing error with rust 1.57.0, "error[E0599]: no method named `retain_mut`..."
+cargo update -p cc --precise "1.0.81"
+# rustls 0.20.9 has MSRV 1.60.0+
+cargo update -p rustls:0.20.9 --precise "0.20.8"
+# rustls 0.21.2 has MSRV 1.60.0+
+cargo update -p rustls:0.21.7 --precise "0.21.1"
+# flate2 1.0.27 has MSRV 1.63.0+
+cargo update -p flate2:1.0.27 --precise "1.0.26"
+# reqwest 0.11.19 has MSRV 1.63.0+
+cargo update -p reqwest --precise "0.11.18"
+# h2 0.3.21 has MSRV 1.63.0+
+cargo update -p h2 --precise "0.3.20"
+# rustls-webpki 0.100.2 has MSRV 1.60+
+cargo update -p rustls-webpki:0.100.3 --precise "0.100.1"
+# rustls-webpki 0.101.6 has MSRV 1.60+
+cargo update -p rustls-webpki:0.101.6 --precise "0.101.1"
+```

ci/Dockerfile.ledger

@@ -0,0 +1,9 @@
+# Taken from bitcoindevkit/rust-hwi
+FROM ghcr.io/ledgerhq/speculos
+
+RUN apt-get update
+RUN apt-get install wget -y
+RUN wget "https://github.com/LedgerHQ/speculos/blob/master/apps/nanos%23btc%232.1%231c8db8da.elf?raw=true" -O /speculos/btc.elf
+ADD automation.json /speculos/automation.json
+
+ENTRYPOINT ["python", "./speculos.py", "--automation", "file:automation.json", "--model", "nanos", "--display", "headless", "--vnc-port", "41000", "btc.elf"]

ci/automation.json

@@ -0,0 +1,30 @@
+{
+    "version": 1,
+    "rules": [
+        {
+            "regexp": "Address \\(\\d/\\d\\)|Message hash \\(\\d/\\d\\)|Confirm|Fees|Review|Amount",
+            "actions": [
+                [ "button", 2, true ],
+                [ "button", 2, false ]
+            ]
+        },
+        {
+            "text": "Sign",
+            "conditions": [
+                [ "seen", false ]
+            ],
+            "actions": [
+                [ "button", 2, true ],
+                [ "button", 2, false ],
+                [ "setbool", "seen", true ]
+            ]
+        },
+        {
+            "regexp": "Approve|Sign|Accept",
+            "actions": [
+                [ "button", 3, true ],
+                [ "button", 3, false ]
+            ]
+        }
+    ]
+}

ci/start-core.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env sh
+
+echo "Starting bitcoin node."
+mkdir $GITHUB_WORKSPACE/.bitcoin
+/root/bitcoind -regtest -server -daemon -datadir=$GITHUB_WORKSPACE/.bitcoin -fallbackfee=0.0002 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0 -blockfilterindex=1 -peerblockfilters=1
+
+echo "Waiting for bitcoin node."
+until /root/bitcoin-cli -regtest -datadir=$GITHUB_WORKSPACE/.bitcoin getblockchaininfo; do
+    sleep 1
+done
+/root/bitcoin-cli -regtest -datadir=$GITHUB_WORKSPACE/.bitcoin createwallet $BDK_RPC_WALLET
+echo "Generating 150 bitcoin blocks."
+ADDR=$(/root/bitcoin-cli -regtest -datadir=$GITHUB_WORKSPACE/.bitcoin -rpcwallet=$BDK_RPC_WALLET getnewaddress)
+/root/bitcoin-cli -regtest -datadir=$GITHUB_WORKSPACE/.bitcoin generatetoaddress 150 $ADDR

examples/compact_filters_balance.rs

@@ -0,0 +1,41 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use bdk::blockchain::compact_filters::*;
+use bdk::database::MemoryDatabase;
+use bdk::*;
+use bitcoin::*;
+use blockchain::compact_filters::CompactFiltersBlockchain;
+use blockchain::compact_filters::CompactFiltersError;
+use log::info;
+use std::sync::Arc;
+
+/// This will return wallet balance using compact filters
+/// Requires a synced local bitcoin node 0.21 running on testnet with blockfilterindex=1 and peerblockfilters=1
+fn main() -> Result<(), CompactFiltersError> {
+    env_logger::init();
+    info!("start");
+
+    let num_threads = 4;
+    let mempool = Arc::new(Mempool::default());
+    let peers = (0..num_threads)
+        .map(|_| Peer::connect("localhost:18333", Arc::clone(&mempool), Network::Testnet))
+        .collect::<Result<_, _>>()?;
+    let blockchain = CompactFiltersBlockchain::new(peers, "./wallet-filters", Some(500_000))?;
+    info!("done {:?}", blockchain);
+    let descriptor = "wpkh(tpubD6NzVbkrYhZ4X2yy78HWrr1M9NT8dKeWfzNiQqDdMqqa9UmmGztGGz6TaLFGsLfdft5iu32gxq1T4eMNxExNNWzVCpf9Y6JZi5TnqoC9wJq/*)";
+
+    let database = MemoryDatabase::default();
+    let wallet = Arc::new(Wallet::new(descriptor, None, Network::Testnet, database).unwrap());
+    wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+    info!("balance: {}", wallet.get_balance()?);
+    Ok(())
+}

examples/compiler.rs

@@ -1,110 +1,76 @@
-extern crate bitcoin;
-extern crate clap;
-extern crate log;
-extern crate magical_bitcoin_wallet;
-extern crate miniscript;
-extern crate rand;
-extern crate serde_json;
-extern crate sled;
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
+extern crate bdk;
+extern crate bitcoin;
+extern crate log;
+extern crate miniscript;
+extern crate serde_json;
+
+use std::error::Error;
 use std::str::FromStr;
 
 use log::info;
 
-use rand::distributions::Alphanumeric;
-use rand::{thread_rng, Rng};
-
-use clap::{App, Arg};
-
 use bitcoin::Network;
 use miniscript::policy::Concrete;
 use miniscript::Descriptor;
 
-use magical_bitcoin_wallet::types::ScriptType;
-use magical_bitcoin_wallet::{OfflineWallet, Wallet};
+use bdk::database::memory::MemoryDatabase;
+use bdk::wallet::AddressIndex::New;
+use bdk::{KeychainKind, Wallet};
 
-fn main() {
+/// Miniscript policy is a high level abstraction of spending conditions. Defined in the
+/// rust-miniscript library here  https://docs.rs/miniscript/7.0.0/miniscript/policy/index.html
+/// rust-miniscript provides a `compile()` function that can be used to compile any miniscript policy
+/// into a descriptor. This descriptor then in turn can be used in bdk a fully functioning wallet
+/// can be derived from the policy.
+///
+/// This example demonstrates the interaction between a bdk wallet and miniscript policy.
+
+fn main() -> Result<(), Box<dyn Error>> {
     env_logger::init_from_env(
         env_logger::Env::default().filter_or(env_logger::DEFAULT_FILTER_ENV, "info"),
     );
 
-    let matches = App::new("Miniscript Compiler")
-        .arg(
-            Arg::with_name("POLICY")
-                .help("Sets the spending policy to compile")
-                .required(true)
-                .index(1),
-        )
-        .arg(
-            Arg::with_name("TYPE")
-                .help("Sets the script type used to embed the compiled policy")
-                .required(true)
-                .index(2)
-                .possible_values(&["sh", "wsh", "sh-wsh"]),
-        )
-        .arg(
-            Arg::with_name("parsed_policy")
-                .long("parsed_policy")
-                .short("p")
-                .help("Also return the parsed spending policy in JSON format"),
-        )
-        .arg(
-            Arg::with_name("network")
-                .short("n")
-                .long("network")
-                .help("Sets the network")
-                .takes_value(true)
-                .default_value("testnet")
-                .possible_values(&["testnet", "regtest"]),
-        )
-        .get_matches();
+    // We start with a generic miniscript policy string
+    let policy_str = "or(10@thresh(4,pk(029ffbe722b147f3035c87cb1c60b9a5947dd49c774cc31e94773478711a929ac0),pk(025f05815e3a1a8a83bfbb03ce016c9a2ee31066b98f567f6227df1d76ec4bd143),pk(025625f41e4a065efc06d5019cbbd56fe8c07595af1231e7cbc03fafb87ebb71ec),pk(02a27c8b850a00f67da3499b60562673dcf5fdfb82b7e17652a7ac54416812aefd),pk(03e618ec5f384d6e19ca9ebdb8e2119e5bef978285076828ce054e55c4daf473e2)),1@and(older(4209713),thresh(2,pk(03deae92101c790b12653231439f27b8897264125ecb2f46f48278603102573165),pk(033841045a531e1adf9910a6ec279589a90b3b8a904ee64ffd692bd08a8996c1aa),pk(02aebf2d10b040eb936a6f02f44ee82f8b34f5c1ccb20ff3949c2b28206b7c1068))))";
+    info!("Compiling policy: \n{}", policy_str);
 
-    let policy_str = matches.value_of("POLICY").unwrap();
-    info!("Compiling policy: {}", policy_str);
+    // Parse the string as a [`Concrete`] type miniscript policy.
+    let policy = Concrete::<String>::from_str(policy_str)?;
 
-    let policy = Concrete::<String>::from_str(&policy_str).unwrap();
-    let compiled = policy.compile().unwrap();
+    // Create a `wsh` type descriptor from the policy.
+    // `policy.compile()` returns the resulting miniscript from the policy.
+    let descriptor = Descriptor::new_wsh(policy.compile()?)?;
 
-    let descriptor = match matches.value_of("TYPE").unwrap() {
-        "sh" => Descriptor::Sh(compiled),
-        "wsh" => Descriptor::Wsh(compiled),
-        "sh-wsh" => Descriptor::ShWsh(compiled),
-        _ => panic!("Invalid type"),
-    };
+    info!("Compiled into following Descriptor: \n{}", descriptor);
 
-    info!("... Descriptor: {}", descriptor);
+    let database = MemoryDatabase::new();
 
-    let temp_db = {
-        let mut temp_db = std::env::temp_dir();
-        let rand_string: String = thread_rng().sample_iter(&Alphanumeric).take(15).collect();
-        temp_db.push(rand_string);
+    // Create a new wallet from this descriptor
+    let wallet = Wallet::new(&format!("{}", descriptor), None, Network::Regtest, database)?;
 
-        let database = sled::open(&temp_db).unwrap();
+    info!(
+        "First derived address from the descriptor: \n{}",
+        wallet.get_address(New)?
+    );
 
-        let network = match matches.value_of("network") {
-            Some("regtest") => Network::Regtest,
-            Some("testnet") | _ => Network::Testnet,
-        };
-        let wallet: OfflineWallet<_> = Wallet::new_offline(
-            &format!("{}", descriptor),
-            None,
-            network,
-            database.open_tree("").unwrap(),
-        )
-        .unwrap();
+    // BDK also has it's own `Policy` structure to represent the spending condition in a more
+    // human readable json format.
+    let spending_policy = wallet.policies(KeychainKind::External)?;
+    info!(
+        "The BDK spending policy: \n{}",
+        serde_json::to_string_pretty(&spending_policy)?
+    );
 
-        info!("... First address: {}", wallet.get_new_address().unwrap());
-
-        if matches.is_present("parsed_policy") {
-            let spending_policy = wallet.policies(ScriptType::External).unwrap();
-            info!(
-                "... Spending policy:\n{}",
-                serde_json::to_string_pretty(&spending_policy).unwrap()
-            );
-        }
-
-        temp_db
-    };
-
-    std::fs::remove_dir_all(temp_db).unwrap();
+    Ok(())
 }

examples/electrum_backend.rs

@@ -0,0 +1,87 @@
+use std::str::FromStr;
+
+use bdk::bitcoin::bip32::ExtendedPrivKey;
+use bdk::bitcoin::Network;
+use bdk::blockchain::{Blockchain, ElectrumBlockchain};
+use bdk::database::MemoryDatabase;
+use bdk::template::Bip84;
+use bdk::wallet::export::FullyNodedExport;
+use bdk::{KeychainKind, SyncOptions, Wallet};
+
+use bdk::electrum_client::Client;
+use bdk::wallet::AddressIndex;
+use bitcoin::bip32;
+
+pub mod utils;
+
+use crate::utils::tx::build_signed_tx;
+
+/// This will create a wallet from an xpriv and get the balance by connecting to an Electrum server.
+/// If enough amount is available, this will send a transaction to an address.
+/// Otherwise, this will display a wallet address to receive funds.
+///
+/// This can be run with `cargo run --example electrum_backend` in the root folder.
+fn main() {
+    let network = Network::Testnet;
+
+    let xpriv = "tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy";
+
+    let electrum_url = "ssl://electrum.blockstream.info:60002";
+
+    run(&network, electrum_url, xpriv);
+}
+
+fn create_wallet(network: &Network, xpriv: &ExtendedPrivKey) -> Wallet<MemoryDatabase> {
+    Wallet::new(
+        Bip84(*xpriv, KeychainKind::External),
+        Some(Bip84(*xpriv, KeychainKind::Internal)),
+        *network,
+        MemoryDatabase::default(),
+    )
+    .unwrap()
+}
+
+fn run(network: &Network, electrum_url: &str, xpriv: &str) {
+    let xpriv = bip32::ExtendedPrivKey::from_str(xpriv).unwrap();
+
+    // Apparently it works only with Electrs (not EletrumX)
+    let blockchain = ElectrumBlockchain::from(Client::new(electrum_url).unwrap());
+
+    let wallet = create_wallet(network, &xpriv);
+
+    wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+
+    let address = wallet.get_address(AddressIndex::New).unwrap().address;
+
+    println!("address: {}", address);
+
+    let balance = wallet.get_balance().unwrap();
+
+    println!("Available coins in BDK wallet : {} sats", balance);
+
+    if balance.confirmed > 6500 {
+        // the wallet sends the amount to itself.
+        let recipient_address = wallet
+            .get_address(AddressIndex::New)
+            .unwrap()
+            .address
+            .to_string();
+
+        let amount = 5359;
+
+        let tx = build_signed_tx(&wallet, &recipient_address, amount);
+
+        blockchain.broadcast(&tx).unwrap();
+
+        println!("tx id: {}", tx.txid());
+    } else {
+        println!("Insufficient Funds. Fund the wallet with the address above");
+    }
+
+    let export = FullyNodedExport::export_wallet(&wallet, "exported wallet", true)
+        .map_err(ToString::to_string)
+        .map_err(bdk::Error::Generic)
+        .unwrap();
+
+    println!("------\nWallet Backup: {}", export.to_string());
+}

examples/esplora_backend_asynchronous.rs

@@ -0,0 +1,93 @@
+use std::str::FromStr;
+
+use bdk::blockchain::Blockchain;
+use bdk::{
+    blockchain::esplora::EsploraBlockchain,
+    database::MemoryDatabase,
+    template::Bip84,
+    wallet::{export::FullyNodedExport, AddressIndex},
+    KeychainKind, SyncOptions, Wallet,
+};
+use bitcoin::{
+    bip32::{self, ExtendedPrivKey},
+    Network,
+};
+
+pub mod utils;
+
+use crate::utils::tx::build_signed_tx;
+
+/// This will create a wallet from an xpriv and get the balance by connecting to an Esplora server,
+/// using non blocking asynchronous calls with `reqwest`.
+/// If enough amount is available, this will send a transaction to an address.
+/// Otherwise, this will display a wallet address to receive funds.
+///
+/// This can be run with `cargo run --no-default-features --features="use-esplora-reqwest, reqwest-default-tls, async-interface" --example esplora_backend_asynchronous`
+/// in the root folder.
+#[tokio::main(flavor = "current_thread")]
+async fn main() {
+    let network = Network::Signet;
+
+    let xpriv = "tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy";
+
+    let esplora_url = "https://explorer.bc-2.jp/api";
+
+    run(&network, esplora_url, xpriv).await;
+}
+
+fn create_wallet(network: &Network, xpriv: &ExtendedPrivKey) -> Wallet<MemoryDatabase> {
+    Wallet::new(
+        Bip84(*xpriv, KeychainKind::External),
+        Some(Bip84(*xpriv, KeychainKind::Internal)),
+        *network,
+        MemoryDatabase::default(),
+    )
+    .unwrap()
+}
+
+async fn run(network: &Network, esplora_url: &str, xpriv: &str) {
+    let xpriv = bip32::ExtendedPrivKey::from_str(xpriv).unwrap();
+
+    let blockchain = EsploraBlockchain::new(esplora_url, 20);
+
+    let wallet = create_wallet(network, &xpriv);
+
+    wallet
+        .sync(&blockchain, SyncOptions::default())
+        .await
+        .unwrap();
+
+    let address = wallet.get_address(AddressIndex::New).unwrap().address;
+
+    println!("address: {}", address);
+
+    let balance = wallet.get_balance().unwrap();
+
+    println!("Available coins in BDK wallet : {} sats", balance);
+
+    if balance.confirmed > 10500 {
+        // the wallet sends the amount to itself.
+        let recipient_address = wallet
+            .get_address(AddressIndex::New)
+            .unwrap()
+            .address
+            .to_string();
+
+        let amount = 9359;
+
+        let tx = build_signed_tx(&wallet, &recipient_address, amount);
+
+        let _ = blockchain.broadcast(&tx);
+
+        println!("tx id: {}", tx.txid());
+    } else {
+        println!("Insufficient Funds. Fund the wallet with the address above");
+    }
+
+    let export = FullyNodedExport::export_wallet(&wallet, "exported wallet", true)
+        .map_err(ToString::to_string)
+        .map_err(bdk::Error::Generic)
+        .unwrap();
+
+    println!("------\nWallet Backup: {}", export.to_string());
+}

examples/esplora_backend_synchronous.rs

@@ -0,0 +1,89 @@
+use std::str::FromStr;
+
+use bdk::blockchain::Blockchain;
+use bdk::{
+    blockchain::esplora::EsploraBlockchain,
+    database::MemoryDatabase,
+    template::Bip84,
+    wallet::{export::FullyNodedExport, AddressIndex},
+    KeychainKind, SyncOptions, Wallet,
+};
+use bitcoin::{
+    bip32::{self, ExtendedPrivKey},
+    Network,
+};
+
+pub mod utils;
+
+use crate::utils::tx::build_signed_tx;
+
+/// This will create a wallet from an xpriv and get the balance by connecting to an Esplora server,
+/// using blocking calls with `ureq`.
+/// If enough amount is available, this will send a transaction to an address.
+/// Otherwise, this will display a wallet address to receive funds.
+///
+/// This can be run with `cargo run --features=use-esplora-ureq --example esplora_backend_synchronous`
+/// in the root folder.
+fn main() {
+    let network = Network::Signet;
+
+    let xpriv = "tprv8ZgxMBicQKsPcx5nBGsR63Pe8KnRUqmbJNENAfGftF3yuXoMMoVJJcYeUw5eVkm9WBPjWYt6HMWYJNesB5HaNVBaFc1M6dRjWSYnmewUMYy";
+
+    let esplora_url = "https://explorer.bc-2.jp/api";
+
+    run(&network, esplora_url, xpriv);
+}
+
+fn create_wallet(network: &Network, xpriv: &ExtendedPrivKey) -> Wallet<MemoryDatabase> {
+    Wallet::new(
+        Bip84(*xpriv, KeychainKind::External),
+        Some(Bip84(*xpriv, KeychainKind::Internal)),
+        *network,
+        MemoryDatabase::default(),
+    )
+    .unwrap()
+}
+
+fn run(network: &Network, esplora_url: &str, xpriv: &str) {
+    let xpriv = bip32::ExtendedPrivKey::from_str(xpriv).unwrap();
+
+    let blockchain = EsploraBlockchain::new(esplora_url, 20);
+
+    let wallet = create_wallet(network, &xpriv);
+
+    wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+
+    let address = wallet.get_address(AddressIndex::New).unwrap().address;
+
+    println!("address: {}", address);
+
+    let balance = wallet.get_balance().unwrap();
+
+    println!("Available coins in BDK wallet : {} sats", balance);
+
+    if balance.confirmed > 10500 {
+        // the wallet sends the amount to itself.
+        let recipient_address = wallet
+            .get_address(AddressIndex::New)
+            .unwrap()
+            .address
+            .to_string();
+
+        let amount = 9359;
+
+        let tx = build_signed_tx(&wallet, &recipient_address, amount);
+
+        blockchain.broadcast(&tx).unwrap();
+
+        println!("tx id: {}", tx.txid());
+    } else {
+        println!("Insufficient Funds. Fund the wallet with the address above");
+    }
+
+    let export = FullyNodedExport::export_wallet(&wallet, "exported wallet", true)
+        .map_err(ToString::to_string)
+        .map_err(bdk::Error::Generic)
+        .unwrap();
+
+    println!("------\nWallet Backup: {}", export.to_string());
+}

examples/hardware_signer.rs

@@ -0,0 +1,106 @@
+use bdk::bitcoin::{Address, Network};
+use bdk::blockchain::{Blockchain, ElectrumBlockchain};
+use bdk::database::MemoryDatabase;
+use bdk::hwi::HWIClient;
+use bdk::miniscript::{Descriptor, DescriptorPublicKey};
+use bdk::signer::SignerOrdering;
+use bdk::wallet::{hardwaresigner::HWISigner, AddressIndex};
+use bdk::{FeeRate, KeychainKind, SignOptions, SyncOptions, Wallet};
+use electrum_client::Client;
+use std::str::FromStr;
+use std::sync::Arc;
+
+// This example shows how to sync a wallet, create a transaction, sign it
+// and broadcast it using an external hardware wallet.
+// The hardware wallet must be connected to the computer and unlocked before
+// running the example. Also, the `hwi` python package should be installed
+// and available in the environment.
+//
+// To avoid loss of funds, consider using an hardware wallet simulator:
+// * Coldcard: https://github.com/Coldcard/firmware
+// * Ledger: https://github.com/LedgerHQ/speculos
+// * Trezor: https://docs.trezor.io/trezor-firmware/core/emulator/index.html
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    println!("Hold tight, I'm connecting to your hardware wallet...");
+
+    // Listing all the available hardware wallet devices...
+    let mut devices = HWIClient::enumerate()?;
+    if devices.is_empty() {
+        panic!("No devices found. Either plug in a hardware wallet, or start a simulator.");
+    }
+    let first_device = devices.remove(0)?;
+    // ...and creating a client out of the first one
+    let client = HWIClient::get_client(&first_device, true, Network::Testnet.into())?;
+    println!("Look what I found, a {}!", first_device.model);
+
+    // Getting the HW's public descriptors
+    let descriptors = client.get_descriptors::<Descriptor<DescriptorPublicKey>>(None)?;
+    println!(
+        "The hardware wallet's descriptor is: {}",
+        descriptors.receive[0]
+    );
+
+    // Creating a custom signer from the device
+    let custom_signer = HWISigner::from_device(&first_device, Network::Testnet.into())?;
+    let mut wallet = Wallet::new(
+        descriptors.receive[0].clone(),
+        Some(descriptors.internal[0].clone()),
+        Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    // Adding the hardware signer to the BDK wallet
+    wallet.add_signer(
+        KeychainKind::External,
+        SignerOrdering(200),
+        Arc::new(custom_signer),
+    );
+
+    // create client for Blockstream's testnet electrum server
+    let blockchain =
+        ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
+
+    println!("Syncing the wallet...");
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    // get deposit address
+    let deposit_address = wallet.get_address(AddressIndex::New)?;
+
+    let balance = wallet.get_balance()?;
+    println!("Wallet balances in SATs: {}", balance);
+
+    if balance.get_total() < 10000 {
+        println!(
+            "Send some sats from the u01.net testnet faucet to address '{addr}'.\nFaucet URL: https://bitcoinfaucet.uo1.net/?to={addr}",
+            addr = deposit_address.address
+        );
+        return Ok(());
+    }
+
+    let return_address = Address::from_str("tb1ql7w62elx9ucw4pj5lgw4l028hmuw80sndtntxt")?
+        .require_network(Network::Testnet)?;
+    let (mut psbt, _details) = {
+        let mut builder = wallet.build_tx();
+        builder
+            .drain_wallet()
+            .drain_to(return_address.script_pubkey())
+            .enable_rbf()
+            .fee_rate(FeeRate::from_sat_per_vb(5.0));
+        builder.finish()?
+    };
+
+    // `sign` will call the hardware wallet asking for a signature
+    assert!(
+        wallet.sign(&mut psbt, SignOptions::default())?,
+        "The hardware wallet couldn't finalize the transaction :("
+    );
+
+    println!("Let's broadcast your tx...");
+    let raw_transaction = psbt.extract_tx();
+    let txid = raw_transaction.txid();
+
+    blockchain.broadcast(&raw_transaction)?;
+    println!("Transaction broadcasted! TXID: {txid}.\nExplorer URL: https://mempool.space/testnet/tx/{txid}", txid = txid);
+
+    Ok(())
+}

examples/mnemonic_to_descriptors.rs

@@ -0,0 +1,60 @@
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use bdk::bitcoin::bip32::DerivationPath;
+use bdk::bitcoin::secp256k1::Secp256k1;
+use bdk::bitcoin::Network;
+use bdk::descriptor;
+use bdk::descriptor::IntoWalletDescriptor;
+use bdk::keys::bip39::{Language, Mnemonic, WordCount};
+use bdk::keys::{GeneratableKey, GeneratedKey};
+use bdk::miniscript::Tap;
+use bdk::Error as BDK_Error;
+use std::error::Error;
+use std::str::FromStr;
+
+/// This example demonstrates how to generate a mnemonic phrase
+/// using BDK and use that to generate a descriptor string.
+fn main() -> Result<(), Box<dyn Error>> {
+    let secp = Secp256k1::new();
+
+    // In this example we are generating a 12 words mnemonic phrase
+    // but it is also possible generate 15, 18, 21 and 24 words
+    // using their respective `WordCount` variant.
+    let mnemonic: GeneratedKey<_, Tap> =
+        Mnemonic::generate((WordCount::Words12, Language::English))
+            .map_err(|_| BDK_Error::Generic("Mnemonic generation error".to_string()))?;
+
+    println!("Mnemonic phrase: {}", *mnemonic);
+    let mnemonic_with_passphrase = (mnemonic, None);
+
+    // define external and internal derivation key path
+    let external_path = DerivationPath::from_str("m/86h/0h/0h/0").unwrap();
+    let internal_path = DerivationPath::from_str("m/86h/0h/0h/1").unwrap();
+
+    // generate external and internal descriptor from mnemonic
+    let (external_descriptor, ext_keymap) =
+        descriptor!(tr((mnemonic_with_passphrase.clone(), external_path)))?
+            .into_wallet_descriptor(&secp, Network::Testnet)?;
+    let (internal_descriptor, int_keymap) =
+        descriptor!(tr((mnemonic_with_passphrase, internal_path)))?
+            .into_wallet_descriptor(&secp, Network::Testnet)?;
+
+    println!("tpub external descriptor: {}", external_descriptor);
+    println!("tpub internal descriptor: {}", internal_descriptor);
+    println!(
+        "tprv external descriptor: {}",
+        external_descriptor.to_string_with_secret(&ext_keymap)
+    );
+    println!(
+        "tprv internal descriptor: {}",
+        internal_descriptor.to_string_with_secret(&int_keymap)
+    );
+
+    Ok(())
+}

examples/multiparty.rs

@@ -1,96 +0,0 @@
-extern crate bitcoin;
-extern crate clap;
-extern crate log;
-extern crate magical_bitcoin_wallet;
-extern crate miniscript;
-extern crate rand;
-extern crate serde_json;
-extern crate sled;
-
-use std::str::FromStr;
-
-use log::info;
-
-use clap::{App, Arg};
-
-use bitcoin::PublicKey;
-
-use miniscript::policy::Concrete;
-use miniscript::Descriptor;
-
-use magical_bitcoin_wallet::multiparty::{Coordinator, Participant, Peer};
-
-fn main() {
-    env_logger::init_from_env(
-        env_logger::Env::default().filter_or(env_logger::DEFAULT_FILTER_ENV, "info"),
-    );
-
-    let matches = App::new("Multiparty Tools")
-        .arg(
-            Arg::with_name("POLICY")
-                .help("Sets the spending policy to compile")
-                .required(true)
-                .index(1),
-        )
-        .arg(
-            Arg::with_name("TYPE")
-                .help("Sets the script type used to embed the compiled policy")
-                .required(true)
-                .index(2)
-                .possible_values(&["sh", "wsh", "sh-wsh"]),
-        )
-        .get_matches();
-
-    let policy_str = matches.value_of("POLICY").unwrap();
-    info!("Compiling policy: {}", policy_str);
-
-    let policy = Concrete::<String>::from_str(&policy_str).unwrap();
-    let compiled = policy.compile().unwrap();
-
-    let descriptor = match matches.value_of("TYPE").unwrap() {
-        "sh" => Descriptor::Sh(compiled),
-        "wsh" => Descriptor::Wsh(compiled),
-        "sh-wsh" => Descriptor::ShWsh(compiled),
-        _ => panic!("Invalid type"),
-    };
-
-    info!("Descriptor: {}", descriptor);
-
-    let mut coordinator: Participant<Coordinator> = Participant::new(descriptor).unwrap();
-    /*let policy = coordinator.policy_for(vec![]).unwrap();
-    info!(
-        "Policy:\n{}",
-        serde_json::to_string_pretty(&policy).unwrap()
-    );*/
-
-    let missing_keys = coordinator.missing_keys();
-    info!("Missing keys: {:?}", missing_keys);
-
-    let pk =
-        PublicKey::from_str("02c65413e56b343a0a31c18d506f1502a17fc64dfbcef6bfb00d1c0d6229bb6f61")
-            .unwrap();
-    coordinator.add_key("Alice", pk.into()).unwrap();
-    coordinator.add_key("Carol", pk.into()).unwrap();
-
-    let for_bob = coordinator.descriptor_for("Bob").unwrap();
-    info!("Descriptor for Bob: {}", for_bob);
-
-    let mut bob_peer: Participant<Peer> = Participant::new(for_bob).unwrap();
-    info!(
-        "Bob's policy: {}",
-        serde_json::to_string(&bob_peer.policy().unwrap().unwrap()).unwrap()
-    );
-    bob_peer.use_key(pk.into()).unwrap();
-    info!("Bob's my_key: {}", bob_peer.my_key().unwrap());
-
-    coordinator.add_key("Bob", pk.into()).unwrap();
-    info!("Coordinator completed: {}", coordinator.completed());
-
-    let coord_map = coordinator.get_map().unwrap();
-
-    let finalized = coordinator.finalize().unwrap();
-    info!("Coordinator final: {}", finalized);
-
-    let bob_finalized = bob_peer.apply_map(coord_map).unwrap();
-    info!("Bob final: {}", bob_finalized);
-}

examples/parse_descriptor.rs

@@ -1,31 +0,0 @@
-extern crate magical_bitcoin_wallet;
-extern crate serde_json;
-
-use std::str::FromStr;
-
-use magical_bitcoin_wallet::bitcoin::*;
-use magical_bitcoin_wallet::descriptor::*;
-
-fn main() {
-    let desc = "wsh(or_d(\
-                    thresh_m(\
-                      2,[d34db33f/44'/0'/0']xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/1/*,tprv8ZgxMBicQKsPduL5QnGihpprdHyypMGi4DhimjtzYemu7se5YQNcZfAPLqXRuGHb5ZX2eTQj62oNqMnyxJ7B7wz54Uzswqw8fFqMVdcmVF7/1/*\
-                    ),\
-                    and_v(vc:pk_h(cVt4o7BGAig1UXywgGSmARhxMdzP5qvQsxKkSsc1XEkw3tDTQFpy),older(1000))\
-                   ))";
-
-    let extended_desc = ExtendedDescriptor::from_str(desc).unwrap();
-    println!("{:?}", extended_desc);
-
-    let policy = extended_desc.extract_policy().unwrap();
-    println!("policy: {}", serde_json::to_string(&policy).unwrap());
-
-    let derived_desc = extended_desc.derive(42).unwrap();
-    println!("{:?}", derived_desc);
-
-    let addr = derived_desc.address(Network::Testnet).unwrap();
-    println!("{}", addr);
-
-    let script = derived_desc.witness_script();
-    println!("{:?}", script);
-}

examples/policy.rs

@@ -0,0 +1,66 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+extern crate bdk;
+extern crate env_logger;
+extern crate log;
+use std::error::Error;
+
+use bdk::bitcoin::Network;
+use bdk::descriptor::{policy::BuildSatisfaction, ExtractPolicy, IntoWalletDescriptor};
+use bdk::wallet::signer::SignersContainer;
+
+/// This example describes the use of the BDK's [`bdk::descriptor::policy`] module.
+///
+/// Policy is higher abstraction representation of the wallet descriptor spending condition.
+/// This is useful to express complex miniscript spending conditions into more human readable form.
+/// The resulting `Policy` structure  can be used to derive spending conditions the wallet is capable
+/// to spend from.
+///
+/// This example demos a Policy output for a 2of2 multisig between between 2 parties, where the wallet holds
+/// one of the Extend Private key.
+
+fn main() -> Result<(), Box<dyn Error>> {
+    env_logger::init_from_env(
+        env_logger::Env::default().filter_or(env_logger::DEFAULT_FILTER_ENV, "info"),
+    );
+
+    let secp = bitcoin::secp256k1::Secp256k1::new();
+
+    // The descriptor used in the example
+    // The form is "wsh(multi(2, <privkey>, <pubkey>))"
+    let desc = "wsh(multi(2,tprv8ZgxMBicQKsPdpkqS7Eair4YxjcuuvDPNYmKX3sCniCf16tHEVrjjiSXEkFRnUH77yXc6ZcwHHcLNfjdi5qUvw3VDfgYiH5mNsj5izuiu2N/1/*,tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/1/*))";
+
+    // Use the descriptor string to derive the full descriptor and a keymap.
+    // The wallet descriptor can be used to create a new bdk::wallet.
+    // While the `keymap` can be used to create a `SignerContainer`.
+    //
+    // The `SignerContainer` can sign for `PSBT`s.
+    // a bdk::wallet internally uses these to handle transaction signing.
+    // But they can be used as independent tools also.
+    let (wallet_desc, keymap) = desc.into_wallet_descriptor(&secp, Network::Testnet)?;
+
+    log::info!("Example Descriptor for policy analysis : {}", wallet_desc);
+
+    // Create the signer with the keymap and descriptor.
+    let signers_container = SignersContainer::build(keymap, &wallet_desc, &secp);
+
+    // Extract the Policy from the given descriptor and signer.
+    // Note that Policy is a wallet specific structure. It depends on the the descriptor, and
+    // what the concerned wallet with a given signer can sign for.
+    let policy = wallet_desc
+        .extract_policy(&signers_container, BuildSatisfaction::None, &secp)?
+        .expect("We expect a policy");
+
+    log::info!("Derived Policy for the descriptor {:#?}", policy);
+
+    Ok(())
+}

examples/psbt.rs

@@ -1,50 +0,0 @@
-extern crate base64;
-extern crate magical_bitcoin_wallet;
-
-use std::str::FromStr;
-
-use magical_bitcoin_wallet::bitcoin;
-use magical_bitcoin_wallet::descriptor::*;
-use magical_bitcoin_wallet::psbt::*;
-use magical_bitcoin_wallet::signer::Signer;
-
-use bitcoin::consensus::encode::{deserialize, serialize};
-use bitcoin::util::psbt::PartiallySignedTransaction;
-use bitcoin::SigHashType;
-
-fn main() {
-    let desc = "pkh(tprv8ZgxMBicQKsPd7Uf69XL1XwhmjHopUGep8GuEiJDZmbQz6o58LninorQAfcKZWARbtRtfnLcJ5MQ2AtHcQJCCRUcMRvmDUjyEmNUWwx8UbK/*)";
-
-    let extended_desc = ExtendedDescriptor::from_str(desc).unwrap();
-
-    let psbt_str = "cHNidP8BAFMCAAAAAd9SiQfxXZ+CKjgjRNonWXsnlA84aLvjxtwCmMfRc0ZbAQAAAAD+////ASjS9QUAAAAAF6kUYJR3oB0lS1M0W1RRMMiENSX45IuHAAAAAAABAPUCAAAAA9I7/OqeFeOFdr5VTLnj3UI/CNRw2eWmMPf7qDv6uIF6AAAAABcWABTG+kgr0g44V0sK9/9FN9oG/CxMK/7///+d0ffphPcV6FE9J/3ZPKWu17YxBnWWTJQyRJs3HUo1gwEAAAAA/v///835mYd9DmnjVnUKd2421MDoZmIxvB4XyJluN3SPUV9hAAAAABcWABRfvwFGp+x/yWdXeNgFs9v0duyeS/7///8CFbH+AAAAAAAXqRSEnTOAjJN/X6ZgR9ftKmwisNSZx4cA4fUFAAAAABl2qRTs6pS4x17MSQ4yNs/1GPsfdlv2NIisAAAAACIGApVE9PPtkcqp8Da43yrXGv4nLOotZdyxwJoTWQxuLxIuCAxfmh4JAAAAAAA=";
-    let psbt_buf = base64::decode(psbt_str).unwrap();
-    let mut psbt: PartiallySignedTransaction = deserialize(&psbt_buf).unwrap();
-
-    let signer = PSBTSigner::from_descriptor(&psbt.global.unsigned_tx, &extended_desc).unwrap();
-
-    for (index, input) in psbt.inputs.iter_mut().enumerate() {
-        for (pubkey, (fing, path)) in &input.hd_keypaths {
-            let sighash = input.sighash_type.unwrap_or(SigHashType::All);
-
-            // Ignore the "witness_utxo" case because we know this psbt is a legacy tx
-            if let Some(non_wit_utxo) = &input.non_witness_utxo {
-                let prev_script = &non_wit_utxo.output
-                    [psbt.global.unsigned_tx.input[index].previous_output.vout as usize]
-                    .script_pubkey;
-                let (signature, sighash) = signer
-                    .sig_legacy_from_fingerprint(index, sighash, fing, path, prev_script)
-                    .unwrap()
-                    .unwrap();
-
-                let mut concat_sig = Vec::new();
-                concat_sig.extend_from_slice(&signature.serialize_der());
-                concat_sig.extend_from_slice(&[sighash as u8]);
-
-                input.partial_sigs.insert(*pubkey, concat_sig);
-            }
-        }
-    }
-
-    println!("signed: {}", base64::encode(&serialize(&psbt)));
-}

examples/psbt_signer.rs

@@ -0,0 +1,121 @@
+// Copyright (c) 2020-2022 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use bdk::blockchain::{Blockchain, ElectrumBlockchain};
+use bdk::database::MemoryDatabase;
+use bdk::wallet::AddressIndex;
+use bdk::{descriptor, SyncOptions};
+use bdk::{FeeRate, SignOptions, Wallet};
+use bitcoin::secp256k1::Secp256k1;
+use bitcoin::{Address, Network};
+use electrum_client::Client;
+use miniscript::descriptor::DescriptorSecretKey;
+use std::error::Error;
+use std::str::FromStr;
+
+/// This example shows how to sign and broadcast the transaction for a PSBT (Partially Signed
+/// Bitcoin Transaction) for a single key, witness public key hash (WPKH) based descriptor wallet.
+/// The electrum protocol is used to sync blockchain data from the testnet bitcoin network and
+/// wallet data is stored in an ephemeral in-memory database. The process steps are:
+/// 1. Create a "signing" wallet and a "watch-only" wallet based on the same private keys.
+/// 2. Deposit testnet funds into the watch only wallet.
+/// 3. Sync the watch only wallet and create a spending transaction to return all funds to the testnet faucet.
+/// 4. Sync the signing wallet and sign and finalize the PSBT created by the watch only wallet.
+/// 5. Broadcast the transactions from the finalized PSBT.
+fn main() -> Result<(), Box<dyn Error>> {
+    // test key created with `bdk-cli key generate` and `bdk-cli key derive` commands
+    let external_secret_xkey = DescriptorSecretKey::from_str("[e9824965/84'/1'/0']tprv8fvem7qWxY3SGCQczQpRpqTKg455wf1zgixn6MZ4ze8gRfHjov5gXBQTadNfDgqs9ERbZZ3Bi1PNYrCCusFLucT39K525MWLpeURjHwUsfX/0/*").unwrap();
+    let internal_secret_xkey = DescriptorSecretKey::from_str("[e9824965/84'/1'/0']tprv8fvem7qWxY3SGCQczQpRpqTKg455wf1zgixn6MZ4ze8gRfHjov5gXBQTadNfDgqs9ERbZZ3Bi1PNYrCCusFLucT39K525MWLpeURjHwUsfX/1/*").unwrap();
+
+    let secp = Secp256k1::new();
+    let external_public_xkey = external_secret_xkey.to_public(&secp).unwrap();
+    let internal_public_xkey = internal_secret_xkey.to_public(&secp).unwrap();
+
+    let signing_external_descriptor = descriptor!(wpkh(external_secret_xkey)).unwrap();
+    let signing_internal_descriptor = descriptor!(wpkh(internal_secret_xkey)).unwrap();
+
+    let watch_only_external_descriptor = descriptor!(wpkh(external_public_xkey)).unwrap();
+    let watch_only_internal_descriptor = descriptor!(wpkh(internal_public_xkey)).unwrap();
+
+    // create client for Blockstream's testnet electrum server
+    let blockchain =
+        ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
+
+    // create watch only wallet
+    let watch_only_wallet: Wallet<MemoryDatabase> = Wallet::new(
+        watch_only_external_descriptor,
+        Some(watch_only_internal_descriptor),
+        Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    // create signing wallet
+    let signing_wallet: Wallet<MemoryDatabase> = Wallet::new(
+        signing_external_descriptor,
+        Some(signing_internal_descriptor),
+        Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    println!("Syncing watch only wallet.");
+    watch_only_wallet.sync(&blockchain, SyncOptions::default())?;
+
+    // get deposit address
+    let deposit_address = watch_only_wallet.get_address(AddressIndex::New)?;
+
+    let balance = watch_only_wallet.get_balance()?;
+    println!("Watch only wallet balances in SATs: {}", balance);
+
+    if balance.get_total() < 10000 {
+        println!(
+            "Send at least 10000 SATs (0.0001 BTC) from the u01.net testnet faucet to address '{addr}'.\nFaucet URL: https://bitcoinfaucet.uo1.net/?to={addr}",
+            addr = deposit_address.address
+        );
+    } else if balance.get_spendable() < 10000 {
+        println!(
+            "Wait for at least 10000 SATs of your wallet transactions to be confirmed...\nBe patient, this could take 10 mins or longer depending on how testnet is behaving."
+        );
+        for tx_details in watch_only_wallet
+            .list_transactions(false)?
+            .iter()
+            .filter(|txd| txd.received > 0 && txd.confirmation_time.is_none())
+        {
+            println!(
+                "See unconfirmed tx for {} SATs: https://mempool.space/testnet/tx/{}",
+                tx_details.received, tx_details.txid
+            );
+        }
+    } else {
+        println!("Creating a PSBT sending 9800 SATs plus fee to the u01.net testnet faucet return address 'tb1ql7w62elx9ucw4pj5lgw4l028hmuw80sndtntxt'.");
+        let return_address = Address::from_str("tb1ql7w62elx9ucw4pj5lgw4l028hmuw80sndtntxt")?
+            .require_network(Network::Testnet)?;
+        let mut builder = watch_only_wallet.build_tx();
+        builder
+            .add_recipient(return_address.script_pubkey(), 9_800)
+            .enable_rbf()
+            .fee_rate(FeeRate::from_sat_per_vb(1.0));
+
+        let (mut psbt, details) = builder.finish()?;
+        println!("Transaction details: {:#?}", details);
+        println!("Unsigned PSBT: {}", psbt);
+
+        // Sign and finalize the PSBT with the signing wallet
+        let finalized = signing_wallet.sign(&mut psbt, SignOptions::default())?;
+        assert!(finalized, "The PSBT was not finalized!");
+        println!("The PSBT has been signed and finalized.");
+
+        // Broadcast the transaction
+        let raw_transaction = psbt.extract_tx();
+        let txid = raw_transaction.txid();
+
+        blockchain.broadcast(&raw_transaction)?;
+        println!("Transaction broadcast! TXID: {txid}.\nExplorer URL: https://mempool.space/testnet/tx/{txid}", txid = txid);
+    }
+
+    Ok(())
+}

examples/repl.rs

@@ -1,126 +0,0 @@
-use std::fs;
-use std::path::PathBuf;
-use std::sync::Arc;
-
-use rustyline::error::ReadlineError;
-use rustyline::Editor;
-
-use clap::AppSettings;
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace, LevelFilter};
-
-use bitcoin::Network;
-
-use magical_bitcoin_wallet::bitcoin;
-use magical_bitcoin_wallet::blockchain::ElectrumBlockchain;
-use magical_bitcoin_wallet::cli;
-use magical_bitcoin_wallet::sled;
-use magical_bitcoin_wallet::{Client, Wallet};
-
-fn prepare_home_dir() -> PathBuf {
-    let mut dir = PathBuf::new();
-    dir.push(&dirs::home_dir().unwrap());
-    dir.push(".magical-bitcoin");
-
-    if !dir.exists() {
-        info!("Creating home directory {}", dir.as_path().display());
-        fs::create_dir(&dir).unwrap();
-    }
-
-    dir.push("database.sled");
-    dir
-}
-
-#[tokio::main]
-async fn main() {
-    env_logger::init();
-
-    let app = cli::make_cli_subcommands();
-    let mut repl_app = app.clone().setting(AppSettings::NoBinaryName);
-
-    let app = cli::add_global_flags(app);
-
-    let matches = app.get_matches();
-
-    // TODO
-    // let level = match matches.occurrences_of("v") {
-    //     0 => LevelFilter::Info,
-    //     1 => LevelFilter::Debug,
-    //     _ => LevelFilter::Trace,
-    // };
-
-    let network = match matches.value_of("network") {
-        Some("regtest") => Network::Regtest,
-        Some("testnet") | _ => Network::Testnet,
-    };
-
-    let descriptor = matches.value_of("descriptor").unwrap();
-    let change_descriptor = matches.value_of("change_descriptor");
-    debug!("descriptors: {:?} {:?}", descriptor, change_descriptor);
-
-    let database = sled::open(prepare_home_dir().to_str().unwrap()).unwrap();
-    let tree = database
-        .open_tree(matches.value_of("wallet").unwrap())
-        .unwrap();
-    debug!("database opened successfully");
-
-    let client = Client::new(matches.value_of("server").unwrap())
-        .await
-        .unwrap();
-    let wallet = Wallet::new(
-        descriptor,
-        change_descriptor,
-        network,
-        tree,
-        ElectrumBlockchain::from(client),
-    )
-    .await
-    .unwrap();
-    let wallet = Arc::new(wallet);
-
-    if let Some(_sub_matches) = matches.subcommand_matches("repl") {
-        let mut rl = Editor::<()>::new();
-
-        // if rl.load_history("history.txt").is_err() {
-        //     println!("No previous history.");
-        // }
-
-        loop {
-            let readline = rl.readline(">> ");
-            match readline {
-                Ok(line) => {
-                    if line.trim() == "" {
-                        continue;
-                    }
-
-                    rl.add_history_entry(line.as_str());
-                    let matches = repl_app.get_matches_from_safe_borrow(line.split(" "));
-                    if let Err(err) = matches {
-                        println!("{}", err.message);
-                        continue;
-                    }
-
-                    if let Some(s) = cli::handle_matches(&Arc::clone(&wallet), matches.unwrap())
-                        .await
-                        .unwrap()
-                    {
-                        println!("{}", s);
-                    }
-                }
-                Err(ReadlineError::Interrupted) => continue,
-                Err(ReadlineError::Eof) => break,
-                Err(err) => {
-                    println!("{:?}", err);
-                    break;
-                }
-            }
-        }
-
-    // rl.save_history("history.txt").unwrap();
-    } else {
-        if let Some(s) = cli::handle_matches(&wallet, matches).await.unwrap() {
-            println!("{}", s);
-        }
-    }
-}

examples/rpcwallet.rs

@@ -0,0 +1,232 @@
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use bdk::bitcoin::secp256k1::Secp256k1;
+use bdk::bitcoin::Amount;
+use bdk::bitcoin::Network;
+use bdk::bitcoincore_rpc::RpcApi;
+
+use bdk::blockchain::rpc::{Auth, RpcBlockchain, RpcConfig};
+use bdk::blockchain::ConfigurableBlockchain;
+
+use bdk::keys::bip39::{Language, Mnemonic, WordCount};
+use bdk::keys::{DerivableKey, GeneratableKey, GeneratedKey};
+
+use bdk::miniscript::miniscript::Segwitv0;
+
+use bdk::sled;
+use bdk::template::Bip84;
+use bdk::wallet::{signer::SignOptions, wallet_name_from_descriptor, AddressIndex, SyncOptions};
+use bdk::KeychainKind;
+use bdk::Wallet;
+
+use bdk::blockchain::Blockchain;
+
+use electrsd;
+
+use std::error::Error;
+use std::path::PathBuf;
+use std::str::FromStr;
+
+/// This example demonstrates a typical way to create a wallet and work with bdk.
+///
+/// This example bdk wallet is connected to a bitcoin core rpc regtest node,
+/// and will attempt to receive, create and broadcast transactions.
+///
+/// To start a bitcoind regtest node programmatically, this example uses
+/// `electrsd` library, which is also a bdk dev-dependency.
+///
+/// But you can start your own bitcoind backend, and the rest of the example should work fine.
+
+fn main() -> Result<(), Box<dyn Error>> {
+    // -- Setting up background bitcoind process
+
+    println!(">> Setting up bitcoind");
+
+    // Start the bitcoind process
+    let bitcoind_conf = electrsd::bitcoind::Conf::default();
+
+    // electrsd will automatically download the bitcoin core binaries
+    let bitcoind_exe =
+        electrsd::bitcoind::downloaded_exe_path().expect("We should always have downloaded path");
+
+    // Launch bitcoind and gather authentication access
+    let bitcoind = electrsd::bitcoind::BitcoinD::with_conf(bitcoind_exe, &bitcoind_conf).unwrap();
+    let bitcoind_auth = Auth::Cookie {
+        file: bitcoind.params.cookie_file.clone(),
+    };
+
+    // Get a new core address
+    let core_address = bitcoind
+        .client
+        .get_new_address(None, None)?
+        .require_network(Network::Regtest)?;
+
+    // Generate 101 blocks and use the above address as coinbase
+    bitcoind.client.generate_to_address(101, &core_address)?;
+
+    println!(">> bitcoind setup complete");
+    println!(
+        "Available coins in Core wallet : {}",
+        bitcoind.client.get_balance(None, None)?
+    );
+
+    // -- Setting up the Wallet
+
+    println!("\n>> Setting up BDK wallet");
+
+    // Get a random private key
+    let xprv = generate_random_ext_privkey()?;
+
+    // Use the derived descriptors from the privatekey to
+    // create unique wallet name.
+    // This is a special utility function exposed via `bdk::wallet_name_from_descriptor()`
+    let wallet_name = wallet_name_from_descriptor(
+        Bip84(xprv.clone(), KeychainKind::External),
+        Some(Bip84(xprv.clone(), KeychainKind::Internal)),
+        Network::Regtest,
+        &Secp256k1::new(),
+    )?;
+
+    // Create a database (using default sled type) to store wallet data
+    let mut datadir = PathBuf::from_str("/tmp/")?;
+    datadir.push(".bdk-example");
+    let database = sled::open(datadir)?;
+    let database = database.open_tree(wallet_name.clone())?;
+
+    // Create a RPC configuration of the running bitcoind backend we created in last step
+    // Note: If you are using custom regtest node, use the appropriate url and auth
+    let rpc_config = RpcConfig {
+        url: bitcoind.params.rpc_socket.to_string(),
+        auth: bitcoind_auth,
+        network: Network::Regtest,
+        wallet_name,
+        sync_params: None,
+    };
+
+    // Use the above configuration to create a RPC blockchain backend
+    let blockchain = RpcBlockchain::from_config(&rpc_config)?;
+
+    // Combine Database + Descriptor to create the final wallet
+    let wallet = Wallet::new(
+        Bip84(xprv.clone(), KeychainKind::External),
+        Some(Bip84(xprv.clone(), KeychainKind::Internal)),
+        Network::Regtest,
+        database,
+    )?;
+
+    // The `wallet` and the `blockchain` are independent structs.
+    // The wallet will be used to do all wallet level actions
+    // The blockchain can be used to do all blockchain level actions.
+    // For certain actions (like sync) the wallet will ask for a blockchain.
+
+    // Sync the wallet
+    // The first sync is important as this will instantiate the
+    // wallet files.
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> BDK wallet setup complete.");
+    println!(
+        "Available initial coins in BDK wallet : {} sats",
+        wallet.get_balance()?
+    );
+
+    // -- Wallet transaction demonstration
+
+    println!("\n>> Sending coins: Core --> BDK, 10 BTC");
+    // Get a new address to receive coins
+    let bdk_new_addr = wallet.get_address(AddressIndex::New)?.address;
+
+    // Send 10 BTC from core wallet to bdk wallet
+    bitcoind.client.send_to_address(
+        &bdk_new_addr,
+        Amount::from_btc(10.0)?,
+        None,
+        None,
+        None,
+        None,
+        None,
+        None,
+    )?;
+
+    // Confirm transaction by generating 1 block
+    bitcoind.client.generate_to_address(1, &core_address)?;
+
+    // Sync the BDK wallet
+    // This time the sync will fetch the new transaction and update it in
+    // wallet database
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> Received coins in BDK wallet");
+    println!(
+        "Available balance in BDK wallet: {} sats",
+        wallet.get_balance()?
+    );
+
+    println!("\n>> Sending coins: BDK --> Core, 5 BTC");
+    // Attempt to send back 5.0 BTC to core address by creating a transaction
+    //
+    // Transactions are created using a `TxBuilder`.
+    // This helps us to systematically build a transaction with all
+    // required customization.
+    // A full list of APIs offered by `TxBuilder` can be found at
+    // https://docs.rs/bdk/latest/bdk/wallet/tx_builder/struct.TxBuilder.html
+    let mut tx_builder = wallet.build_tx();
+
+    // For a regular transaction, just set the recipient and amount
+    tx_builder.set_recipients(vec![(core_address.script_pubkey(), 500000000)]);
+
+    // Finalize the transaction and extract the PSBT
+    let (mut psbt, _) = tx_builder.finish()?;
+
+    // Set signing option
+    let signopt = SignOptions {
+        assume_height: None,
+        ..Default::default()
+    };
+
+    // Sign the psbt
+    wallet.sign(&mut psbt, signopt)?;
+
+    // Extract the signed transaction
+    let tx = psbt.extract_tx();
+
+    // Broadcast the transaction
+    blockchain.broadcast(&tx)?;
+
+    // Confirm transaction by generating some blocks
+    bitcoind.client.generate_to_address(1, &core_address)?;
+
+    // Sync the BDK wallet
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!(">> Coins sent to Core wallet");
+    println!(
+        "Remaining BDK wallet balance: {} sats",
+        wallet.get_balance()?
+    );
+    println!("\nCongrats!! you made your first test transaction with bdk and bitcoin core.");
+
+    Ok(())
+}
+
+// Helper function demonstrating privatekey extraction using bip39 mnemonic
+// The mnemonic can be shown to user to safekeeping and the same wallet
+// private descriptors can be recreated from it.
+fn generate_random_ext_privkey() -> Result<impl DerivableKey<Segwitv0> + Clone, Box<dyn Error>> {
+    // a Bip39 passphrase can be set optionally
+    let password = Some("random password".to_string());
+
+    // Generate a random mnemonic, and use that to create a "DerivableKey"
+    let mnemonic: GeneratedKey<_, _> = Mnemonic::generate((WordCount::Words12, Language::English))
+        .map_err(|e| e.expect("Unknown Error"))?;
+
+    // `Ok(mnemonic)` would also work if there's no passphrase and it would
+    // yield the same result as this construct with `password` = `None`.
+    Ok((mnemonic, password))
+}

examples/utils/mod.rs

@@ -0,0 +1,33 @@
+pub(crate) mod tx {
+
+    use std::str::FromStr;
+
+    use bdk::{database::BatchDatabase, SignOptions, Wallet};
+    use bitcoin::{Address, Transaction};
+
+    pub fn build_signed_tx<D: BatchDatabase>(
+        wallet: &Wallet<D>,
+        recipient_address: &str,
+        amount: u64,
+    ) -> Transaction {
+        // Create a transaction builder
+        let mut tx_builder = wallet.build_tx();
+
+        let to_address = Address::from_str(recipient_address)
+            .unwrap()
+            .require_network(wallet.network())
+            .unwrap();
+
+        // Set recipient of the transaction
+        tx_builder.set_recipients(vec![(to_address.script_pubkey(), amount)]);
+
+        // Finalise the transaction and extract PSBT
+        let (mut psbt, _) = tx_builder.finish().unwrap();
+
+        // Sign the above psbt with signing option
+        wallet.sign(&mut psbt, SignOptions::default()).unwrap();
+
+        // Extract the final transaction
+        psbt.extract_tx()
+    }
+}

macros/Cargo.toml

@@ -0,0 +1,24 @@
+[package]
+name = "bdk-macros"
+version = "0.6.0"
+authors = ["Alekos Filini <alekos.filini@gmail.com>"]
+edition = "2018"
+homepage = "https://bitcoindevkit.org"
+repository = "https://github.com/bitcoindevkit/bdk"
+documentation = "https://docs.rs/bdk-macros"
+description = "Supporting macros for `bdk`"
+keywords = ["bdk"]
+license = "MIT OR Apache-2.0"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+syn = { version = "1.0", features = ["parsing", "full"] }
+proc-macro2 = "1.0"
+quote = "1.0"
+
+[features]
+debug = ["syn/extra-traits"]
+
+[lib]
+proc-macro = true

macros/src/lib.rs

@@ -0,0 +1,146 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+#[macro_use]
+extern crate quote;
+
+use proc_macro::TokenStream;
+
+use syn::spanned::Spanned;
+use syn::{parse, ImplItemMethod, ItemImpl, ItemTrait, Token};
+
+fn add_async_trait(mut parsed: ItemTrait) -> TokenStream {
+    let output = quote! {
+        #[cfg(not(feature = "async-interface"))]
+        #parsed
+    };
+
+    for mut item in &mut parsed.items {
+        if let syn::TraitItem::Method(m) = &mut item {
+            m.sig.asyncness = Some(Token![async](m.span()));
+        }
+    }
+
+    let output = quote! {
+        #output
+
+        #[cfg(feature = "async-interface")]
+        #[async_trait(?Send)]
+        #parsed
+    };
+
+    output.into()
+}
+
+fn add_async_method(mut parsed: ImplItemMethod) -> TokenStream {
+    let output = quote! {
+        #[cfg(not(feature = "async-interface"))]
+        #parsed
+    };
+
+    parsed.sig.asyncness = Some(Token![async](parsed.span()));
+
+    let output = quote! {
+        #output
+
+        #[cfg(feature = "async-interface")]
+        #parsed
+    };
+
+    output.into()
+}
+
+fn add_async_impl_trait(mut parsed: ItemImpl) -> TokenStream {
+    let output = quote! {
+        #[cfg(not(feature = "async-interface"))]
+        #parsed
+    };
+
+    for mut item in &mut parsed.items {
+        if let syn::ImplItem::Method(m) = &mut item {
+            m.sig.asyncness = Some(Token![async](m.span()));
+        }
+    }
+
+    let output = quote! {
+        #output
+
+        #[cfg(feature = "async-interface")]
+        #[async_trait(?Send)]
+        #parsed
+    };
+
+    output.into()
+}
+
+/// Makes a method or every method of a trait `async`, if the `async-interface` feature is enabled.
+///
+/// Requires the `async-trait` crate as a dependency whenever this attribute is used on a trait
+/// definition or trait implementation.
+#[proc_macro_attribute]
+pub fn maybe_async(_attr: TokenStream, item: TokenStream) -> TokenStream {
+    if let Ok(parsed) = parse(item.clone()) {
+        add_async_trait(parsed)
+    } else if let Ok(parsed) = parse(item.clone()) {
+        add_async_method(parsed)
+    } else if let Ok(parsed) = parse(item) {
+        add_async_impl_trait(parsed)
+    } else {
+        (quote! {
+            compile_error!("#[maybe_async] can only be used on methods, trait or trait impl blocks")
+        })
+        .into()
+    }
+}
+
+/// Awaits, if the `async-interface` feature is enabled.
+#[proc_macro]
+pub fn maybe_await(expr: TokenStream) -> TokenStream {
+    let expr: proc_macro2::TokenStream = expr.into();
+    let quoted = quote! {
+        {
+            #[cfg(not(feature = "async-interface"))]
+            {
+                #expr
+            }
+
+            #[cfg(feature = "async-interface")]
+            {
+                #expr.await
+            }
+        }
+    };
+
+    quoted.into()
+}
+
+/// Awaits, if the `async-interface` feature is enabled, uses `tokio::Runtime::block_on()` otherwise
+///
+/// Requires the `tokio` crate as a dependecy with `rt-core` or `rt-threaded` to build.
+#[proc_macro]
+pub fn await_or_block(expr: TokenStream) -> TokenStream {
+    let expr: proc_macro2::TokenStream = expr.into();
+    let quoted = quote! {
+        {
+            #[cfg(not(feature = "async-interface"))]
+            {
+                tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap().block_on(#expr)
+            }
+
+            #[cfg(feature = "async-interface")]
+            {
+                #expr.await
+            }
+        }
+    };
+
+    quoted.into()
+}

src/blockchain/any.rs

@@ -0,0 +1,248 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Runtime-checked blockchain types
+//!
+//! This module provides the implementation of [`AnyBlockchain`] which allows switching the
+//! inner [`Blockchain`] type at runtime.
+//!
+//! ## Example
+//!
+//! When paired with the use of [`ConfigurableBlockchain`], it allows creating any
+//! blockchain type supported using a single line of code:
+//!
+//! ```no_run
+//! # use bitcoin::Network;
+//! # use bdk::blockchain::*;
+//! # #[cfg(all(feature = "esplora", feature = "ureq"))]
+//! # {
+//! let config = serde_json::from_str("...")?;
+//! let blockchain = AnyBlockchain::from_config(&config)?;
+//! let height = blockchain.get_height();
+//! # }
+//! # Ok::<(), bdk::Error>(())
+//! ```
+
+use super::*;
+
+macro_rules! impl_from {
+    ( boxed $from:ty, $to:ty, $variant:ident, $( $cfg:tt )* ) => {
+        $( $cfg )*
+        impl From<$from> for $to {
+            fn from(inner: $from) -> Self {
+                <$to>::$variant(Box::new(inner))
+            }
+        }
+    };
+    ( $from:ty, $to:ty, $variant:ident, $( $cfg:tt )* ) => {
+        $( $cfg )*
+        impl From<$from> for $to {
+            fn from(inner: $from) -> Self {
+                <$to>::$variant(inner)
+            }
+        }
+    };
+}
+
+macro_rules! impl_inner_method {
+    ( $self:expr, $name:ident $(, $args:expr)* ) => {
+        match $self {
+            #[cfg(feature = "electrum")]
+            AnyBlockchain::Electrum(inner) => inner.$name( $($args, )* ),
+            #[cfg(feature = "esplora")]
+            AnyBlockchain::Esplora(inner) => inner.$name( $($args, )* ),
+            #[cfg(feature = "compact_filters")]
+            AnyBlockchain::CompactFilters(inner) => inner.$name( $($args, )* ),
+            #[cfg(feature = "rpc")]
+            AnyBlockchain::Rpc(inner) => inner.$name( $($args, )* ),
+        }
+    }
+}
+
+/// Type that can contain any of the [`Blockchain`] types defined by the library
+///
+/// It allows switching backend at runtime
+///
+/// See [this module](crate::blockchain::any)'s documentation for a usage example.
+pub enum AnyBlockchain {
+    #[cfg(feature = "electrum")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "electrum")))]
+    /// Electrum client
+    Electrum(Box<electrum::ElectrumBlockchain>),
+    #[cfg(feature = "esplora")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "esplora")))]
+    /// Esplora client
+    Esplora(Box<esplora::EsploraBlockchain>),
+    #[cfg(feature = "compact_filters")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "compact_filters")))]
+    /// Compact filters client
+    CompactFilters(Box<compact_filters::CompactFiltersBlockchain>),
+    #[cfg(feature = "rpc")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "rpc")))]
+    /// RPC client
+    Rpc(Box<rpc::RpcBlockchain>),
+}
+
+#[maybe_async]
+impl Blockchain for AnyBlockchain {
+    fn get_capabilities(&self) -> HashSet<Capability> {
+        maybe_await!(impl_inner_method!(self, get_capabilities))
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(self, broadcast, tx))
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        maybe_await!(impl_inner_method!(self, estimate_fee, target))
+    }
+}
+
+#[maybe_async]
+impl GetHeight for AnyBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        maybe_await!(impl_inner_method!(self, get_height))
+    }
+}
+
+#[maybe_async]
+impl GetTx for AnyBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        maybe_await!(impl_inner_method!(self, get_tx, txid))
+    }
+}
+
+#[maybe_async]
+impl GetBlockHash for AnyBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        maybe_await!(impl_inner_method!(self, get_block_hash, height))
+    }
+}
+
+#[maybe_async]
+impl WalletSync for AnyBlockchain {
+    fn wallet_sync<D: BatchDatabase>(
+        &self,
+        database: &RefCell<D>,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(
+            self,
+            wallet_sync,
+            database,
+            progress_update
+        ))
+    }
+
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &RefCell<D>,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(
+            self,
+            wallet_setup,
+            database,
+            progress_update
+        ))
+    }
+}
+
+impl_from!(boxed electrum::ElectrumBlockchain, AnyBlockchain, Electrum, #[cfg(feature = "electrum")]);
+impl_from!(boxed esplora::EsploraBlockchain, AnyBlockchain, Esplora, #[cfg(feature = "esplora")]);
+impl_from!(boxed compact_filters::CompactFiltersBlockchain, AnyBlockchain, CompactFilters, #[cfg(feature = "compact_filters")]);
+impl_from!(boxed rpc::RpcBlockchain, AnyBlockchain, Rpc, #[cfg(feature = "rpc")]);
+
+/// Type that can contain any of the blockchain configurations defined by the library
+///
+/// This allows storing a single configuration that can be loaded into an [`AnyBlockchain`]
+/// instance. Wallets that plan to offer users the ability to switch blockchain backend at runtime
+/// will find this particularly useful.
+///
+/// This type can be serialized from a JSON object like:
+///
+/// ```
+/// # #[cfg(feature = "electrum")]
+/// # {
+/// use bdk::blockchain::{electrum::ElectrumBlockchainConfig, AnyBlockchainConfig};
+/// let config: AnyBlockchainConfig = serde_json::from_str(
+///     r#"{
+///    "type" : "electrum",
+///    "url" : "ssl://electrum.blockstream.info:50002",
+///    "retry": 2,
+///    "stop_gap": 20,
+///    "validate_domain": true
+/// }"#,
+/// )
+/// .unwrap();
+/// assert_eq!(
+///     config,
+///     AnyBlockchainConfig::Electrum(ElectrumBlockchainConfig {
+///         url: "ssl://electrum.blockstream.info:50002".into(),
+///         retry: 2,
+///         socks5: None,
+///         timeout: None,
+///         stop_gap: 20,
+///         validate_domain: true,
+///     })
+/// );
+/// # }
+/// ```
+#[derive(Debug, serde::Serialize, serde::Deserialize, Clone, PartialEq, Eq)]
+#[serde(tag = "type", rename_all = "snake_case")]
+pub enum AnyBlockchainConfig {
+    #[cfg(feature = "electrum")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "electrum")))]
+    /// Electrum client
+    Electrum(electrum::ElectrumBlockchainConfig),
+    #[cfg(feature = "esplora")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "esplora")))]
+    /// Esplora client
+    Esplora(esplora::EsploraBlockchainConfig),
+    #[cfg(feature = "compact_filters")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "compact_filters")))]
+    /// Compact filters client
+    CompactFilters(compact_filters::CompactFiltersBlockchainConfig),
+    #[cfg(feature = "rpc")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "rpc")))]
+    /// RPC client configuration
+    Rpc(rpc::RpcConfig),
+}
+
+impl ConfigurableBlockchain for AnyBlockchain {
+    type Config = AnyBlockchainConfig;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        Ok(match config {
+            #[cfg(feature = "electrum")]
+            AnyBlockchainConfig::Electrum(inner) => {
+                AnyBlockchain::Electrum(Box::new(electrum::ElectrumBlockchain::from_config(inner)?))
+            }
+            #[cfg(feature = "esplora")]
+            AnyBlockchainConfig::Esplora(inner) => {
+                AnyBlockchain::Esplora(Box::new(esplora::EsploraBlockchain::from_config(inner)?))
+            }
+            #[cfg(feature = "compact_filters")]
+            AnyBlockchainConfig::CompactFilters(inner) => AnyBlockchain::CompactFilters(Box::new(
+                compact_filters::CompactFiltersBlockchain::from_config(inner)?,
+            )),
+            #[cfg(feature = "rpc")]
+            AnyBlockchainConfig::Rpc(inner) => {
+                AnyBlockchain::Rpc(Box::new(rpc::RpcBlockchain::from_config(inner)?))
+            }
+        })
+    }
+}
+
+impl_from!(electrum::ElectrumBlockchainConfig, AnyBlockchainConfig, Electrum, #[cfg(feature = "electrum")]);
+impl_from!(esplora::EsploraBlockchainConfig, AnyBlockchainConfig, Esplora, #[cfg(feature = "esplora")]);
+impl_from!(compact_filters::CompactFiltersBlockchainConfig, AnyBlockchainConfig, CompactFilters, #[cfg(feature = "compact_filters")]);
+impl_from!(rpc::RpcConfig, AnyBlockchainConfig, Rpc, #[cfg(feature = "rpc")]);

src/blockchain/compact_filters/mod.rs

@@ -0,0 +1,618 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Compact Filters
+//!
+//! This module contains a multithreaded implementation of an [`Blockchain`] backend that
+//! uses BIP157 (aka "Neutrino") to populate the wallet's [database](crate::database::Database)
+//! by downloading compact filters from the P2P network.
+//!
+//! Since there are currently very few peers "in the wild" that advertise the required service
+//! flag, this implementation requires that one or more known peers are provided by the user.
+//! No dns or other kinds of peer discovery are done internally.
+//!
+//! Moreover, this module doesn't currently support detecting and resolving conflicts between
+//! messages received by different peers. Thus, it's recommended to use this module by only
+//! connecting to a single peer at a time, optionally by opening multiple connections if it's
+//! desirable to use multiple threads at once to sync in parallel.
+//!
+//! This is an **EXPERIMENTAL** feature, API and other major changes are expected.
+//!
+//! ## Example
+//!
+//! ```no_run
+//! # use std::sync::Arc;
+//! # use bitcoin::*;
+//! # use bdk::*;
+//! # use bdk::blockchain::compact_filters::*;
+//! let num_threads = 4;
+//!
+//! let mempool = Arc::new(Mempool::default());
+//! let peers = (0..num_threads)
+//!     .map(|_| {
+//!         Peer::connect(
+//!             "btcd-mainnet.lightning.computer:8333",
+//!             Arc::clone(&mempool),
+//!             Network::Bitcoin,
+//!         )
+//!     })
+//!     .collect::<Result<_, _>>()?;
+//! let blockchain = CompactFiltersBlockchain::new(peers, "./wallet-filters", Some(500_000))?;
+//! # Ok::<(), CompactFiltersError>(())
+//! ```
+
+use std::collections::HashSet;
+use std::fmt;
+use std::ops::DerefMut;
+use std::path::Path;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use std::sync::{Arc, Mutex};
+
+#[allow(unused_imports)]
+use log::{debug, error, info, trace};
+
+use bitcoin::network::message_blockdata::Inventory;
+use bitcoin::{Network, OutPoint, Transaction, Txid};
+
+use rocksdb::{Options, SliceTransform, DB};
+
+mod peer;
+mod store;
+mod sync;
+
+use crate::blockchain::*;
+use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
+use crate::error::Error;
+use crate::types::{KeychainKind, LocalUtxo, TransactionDetails};
+use crate::{BlockTime, FeeRate};
+
+use peer::*;
+use store::*;
+use sync::*;
+
+pub use peer::{Mempool, Peer};
+
+const SYNC_HEADERS_COST: f32 = 1.0;
+const SYNC_FILTERS_COST: f32 = 11.6 * 1_000.0;
+const PROCESS_BLOCKS_COST: f32 = 20_000.0;
+
+/// Structure implementing the required blockchain traits
+///
+/// ## Example
+/// See the [`blockchain::compact_filters`](crate::blockchain::compact_filters) module for a usage example.
+#[derive(Debug)]
+pub struct CompactFiltersBlockchain {
+    peers: Vec<Arc<Peer>>,
+    headers: Arc<ChainStore<Full>>,
+    skip_blocks: Option<usize>,
+}
+
+impl CompactFiltersBlockchain {
+    /// Construct a new instance given a list of peers, a path to store headers and block
+    /// filters downloaded during the sync and optionally a number of blocks to ignore starting
+    /// from the genesis while scanning for the wallet's outputs.
+    ///
+    /// For each [`Peer`] specified a new thread will be spawned to download and verify the filters
+    /// in parallel. It's currently recommended to only connect to a single peer to avoid
+    /// inconsistencies in the data returned, optionally with multiple connections in parallel to
+    /// speed-up the sync process.
+    pub fn new<P: AsRef<Path>>(
+        peers: Vec<Peer>,
+        storage_dir: P,
+        skip_blocks: Option<usize>,
+    ) -> Result<Self, CompactFiltersError> {
+        if peers.is_empty() {
+            return Err(CompactFiltersError::NoPeers);
+        }
+
+        let mut opts = Options::default();
+        opts.create_if_missing(true);
+        opts.set_prefix_extractor(SliceTransform::create_fixed_prefix(16));
+
+        let network = peers[0].get_network();
+
+        let cfs = DB::list_cf(&opts, &storage_dir).unwrap_or_else(|_| vec!["default".to_string()]);
+        let db = DB::open_cf(&opts, &storage_dir, &cfs)?;
+        let headers = Arc::new(ChainStore::new(db, network)?);
+
+        // try to recover partial snapshots
+        for cf_name in &cfs {
+            if !cf_name.starts_with("_headers:") {
+                continue;
+            }
+
+            info!("Trying to recover: {:?}", cf_name);
+            headers.recover_snapshot(cf_name)?;
+        }
+
+        Ok(CompactFiltersBlockchain {
+            peers: peers.into_iter().map(Arc::new).collect(),
+            headers,
+            skip_blocks,
+        })
+    }
+
+    /// Process a transaction by looking for inputs that spend from a UTXO in the database or
+    /// outputs that send funds to a know script_pubkey.
+    fn process_tx<D: BatchDatabase>(
+        &self,
+        database: &mut D,
+        tx: &Transaction,
+        height: Option<u32>,
+        timestamp: Option<u64>,
+        internal_max_deriv: &mut Option<u32>,
+        external_max_deriv: &mut Option<u32>,
+    ) -> Result<(), Error> {
+        let mut updates = database.begin_batch();
+
+        let mut incoming: u64 = 0;
+        let mut outgoing: u64 = 0;
+
+        let mut inputs_sum: u64 = 0;
+        let mut outputs_sum: u64 = 0;
+
+        // look for our own inputs
+        for (i, input) in tx.input.iter().enumerate() {
+            if let Some(previous_output) = database.get_previous_output(&input.previous_output)? {
+                inputs_sum += previous_output.value;
+
+                // this output is ours, we have a path to derive it
+                if let Some((keychain, _)) =
+                    database.get_path_from_script_pubkey(&previous_output.script_pubkey)?
+                {
+                    outgoing += previous_output.value;
+
+                    debug!("{} input #{} is mine, setting utxo as spent", tx.txid(), i);
+                    updates.set_utxo(&LocalUtxo {
+                        outpoint: input.previous_output,
+                        txout: previous_output.clone(),
+                        keychain,
+                        is_spent: true,
+                    })?;
+                }
+            }
+        }
+
+        for (i, output) in tx.output.iter().enumerate() {
+            // to compute the fees later
+            outputs_sum += output.value;
+
+            // this output is ours, we have a path to derive it
+            if let Some((keychain, child)) =
+                database.get_path_from_script_pubkey(&output.script_pubkey)?
+            {
+                debug!("{} output #{} is mine, adding utxo", tx.txid(), i);
+                updates.set_utxo(&LocalUtxo {
+                    outpoint: OutPoint::new(tx.txid(), i as u32),
+                    txout: output.clone(),
+                    keychain,
+                    is_spent: false,
+                })?;
+                incoming += output.value;
+
+                if keychain == KeychainKind::Internal
+                    && (internal_max_deriv.is_none() || child > internal_max_deriv.unwrap_or(0))
+                {
+                    *internal_max_deriv = Some(child);
+                } else if keychain == KeychainKind::External
+                    && (external_max_deriv.is_none() || child > external_max_deriv.unwrap_or(0))
+                {
+                    *external_max_deriv = Some(child);
+                }
+            }
+        }
+
+        if incoming > 0 || outgoing > 0 {
+            let tx = TransactionDetails {
+                txid: tx.txid(),
+                transaction: Some(tx.clone()),
+                received: incoming,
+                sent: outgoing,
+                confirmation_time: BlockTime::new(height, timestamp),
+                fee: Some(inputs_sum.saturating_sub(outputs_sum)),
+            };
+
+            info!("Saving tx {}", tx.txid);
+            updates.set_tx(&tx)?;
+        }
+
+        database.commit_batch(updates)?;
+
+        Ok(())
+    }
+}
+
+impl Blockchain for CompactFiltersBlockchain {
+    fn get_capabilities(&self) -> HashSet<Capability> {
+        vec![Capability::FullHistory].into_iter().collect()
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        self.peers[0].broadcast_tx(tx.clone())?;
+
+        Ok(())
+    }
+
+    fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
+        // TODO
+        Ok(FeeRate::default())
+    }
+}
+
+impl GetHeight for CompactFiltersBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.headers.get_height()? as u32)
+    }
+}
+
+impl GetTx for CompactFiltersBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.peers[0]
+            .get_mempool()
+            .get_tx(&Inventory::Transaction(*txid)))
+    }
+}
+
+impl GetBlockHash for CompactFiltersBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        self.headers
+            .get_block_hash(height as usize)?
+            .ok_or(Error::CompactFilters(
+                CompactFiltersError::BlockHashNotFound,
+            ))
+    }
+}
+
+impl WalletSync for CompactFiltersBlockchain {
+    #[allow(clippy::mutex_atomic)] // Mutex is easier to understand than a CAS loop.
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &RefCell<D>,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        let first_peer = &self.peers[0];
+
+        let skip_blocks = self.skip_blocks.unwrap_or(0);
+
+        let cf_sync = Arc::new(CfSync::new(Arc::clone(&self.headers), skip_blocks, 0x00)?);
+
+        let initial_height = self.headers.get_height()?;
+        let total_bundles = (first_peer.get_version().start_height as usize)
+            .checked_sub(skip_blocks)
+            .map(|x| x / 1000)
+            .unwrap_or(0)
+            + 1;
+        let expected_bundles_to_sync = total_bundles.saturating_sub(cf_sync.pruned_bundles()?);
+
+        let headers_cost = (first_peer.get_version().start_height as usize)
+            .saturating_sub(initial_height) as f32
+            * SYNC_HEADERS_COST;
+        let filters_cost = expected_bundles_to_sync as f32 * SYNC_FILTERS_COST;
+
+        let total_cost = headers_cost + filters_cost + PROCESS_BLOCKS_COST;
+
+        if let Some(snapshot) = sync::sync_headers(
+            Arc::clone(first_peer),
+            Arc::clone(&self.headers),
+            |new_height| {
+                let local_headers_cost =
+                    new_height.saturating_sub(initial_height) as f32 * SYNC_HEADERS_COST;
+                progress_update.update(
+                    local_headers_cost / total_cost * 100.0,
+                    Some(format!("Synced headers to {}", new_height)),
+                )
+            },
+        )? {
+            if snapshot.work()? > self.headers.work()? {
+                info!("Applying snapshot with work: {}", snapshot.work()?);
+                self.headers.apply_snapshot(snapshot)?;
+            }
+        }
+
+        let synced_height = self.headers.get_height()?;
+        let buried_height = synced_height.saturating_sub(sync::BURIED_CONFIRMATIONS);
+        info!("Synced headers to height: {}", synced_height);
+
+        cf_sync.prepare_sync(Arc::clone(first_peer))?;
+
+        let mut database = database.borrow_mut();
+        let database = database.deref_mut();
+
+        let all_scripts = Arc::new(
+            database
+                .iter_script_pubkeys(None)?
+                .into_iter()
+                .map(|s| s.to_bytes())
+                .collect::<Vec<_>>(),
+        );
+
+        #[allow(clippy::mutex_atomic)]
+        let last_synced_block = Arc::new(Mutex::new(synced_height));
+
+        let synced_bundles = Arc::new(AtomicUsize::new(0));
+        let progress_update = Arc::new(Mutex::new(progress_update));
+
+        let mut threads = Vec::with_capacity(self.peers.len());
+        for peer in &self.peers {
+            let cf_sync = Arc::clone(&cf_sync);
+            let peer = Arc::clone(peer);
+            let headers = Arc::clone(&self.headers);
+            let all_scripts = Arc::clone(&all_scripts);
+            let last_synced_block = Arc::clone(&last_synced_block);
+            let progress_update = Arc::clone(&progress_update);
+            let synced_bundles = Arc::clone(&synced_bundles);
+
+            let thread = std::thread::spawn(move || {
+                cf_sync.capture_thread_for_sync(
+                    peer,
+                    |block_hash, filter| {
+                        if !filter
+                            .match_any(block_hash, all_scripts.iter().map(|s| s.as_slice()))?
+                        {
+                            return Ok(false);
+                        }
+
+                        let block_height = headers.get_height_for(block_hash)?.unwrap_or(0);
+                        let saved_correct_block = matches!(headers.get_full_block(block_height)?, Some(block) if &block.block_hash() == block_hash);
+
+                        if saved_correct_block {
+                            Ok(false)
+                        } else {
+                            let mut last_synced_block = last_synced_block.lock().unwrap();
+
+                            // If we download a block older than `last_synced_block`, we update it so that
+                            // we know to delete and re-process all txs starting from that height
+                            if block_height < *last_synced_block {
+                                *last_synced_block = block_height;
+                            }
+
+                            Ok(true)
+                        }
+                    },
+                    |index| {
+                        let synced_bundles = synced_bundles.fetch_add(1, Ordering::SeqCst);
+                        let local_filters_cost = synced_bundles as f32 * SYNC_FILTERS_COST;
+                        progress_update.lock().unwrap().update(
+                            (headers_cost + local_filters_cost) / total_cost * 100.0,
+                            Some(format!(
+                                "Synced filters {} - {}",
+                                index * 1000 + 1,
+                                (index + 1) * 1000
+                            )),
+                        )
+                    },
+                )
+            });
+
+            threads.push(thread);
+        }
+
+        for t in threads {
+            t.join().unwrap()?;
+        }
+
+        progress_update.lock().unwrap().update(
+            (headers_cost + filters_cost) / total_cost * 100.0,
+            Some("Processing downloaded blocks and mempool".into()),
+        )?;
+
+        // delete all txs newer than last_synced_block
+        let last_synced_block = *last_synced_block.lock().unwrap();
+        log::debug!(
+            "Dropping transactions newer than `last_synced_block` = {}",
+            last_synced_block
+        );
+        let mut updates = database.begin_batch();
+        for details in database.iter_txs(false)? {
+            match details.confirmation_time {
+                Some(c) if (c.height as usize) < last_synced_block => continue,
+                _ => updates.del_tx(&details.txid, false)?,
+            };
+        }
+        database.commit_batch(updates)?;
+
+        match first_peer.ask_for_mempool() {
+            Err(CompactFiltersError::PeerBloomDisabled) => {
+                log::warn!("Peer has BLOOM disabled, we can't ask for the mempool")
+            }
+            e => e?,
+        };
+
+        let mut internal_max_deriv = None;
+        let mut external_max_deriv = None;
+
+        for (height, block) in self.headers.iter_full_blocks()? {
+            for tx in &block.txdata {
+                self.process_tx(
+                    database,
+                    tx,
+                    Some(height as u32),
+                    None,
+                    &mut internal_max_deriv,
+                    &mut external_max_deriv,
+                )?;
+            }
+        }
+        for tx in first_peer.get_mempool().iter_txs().iter() {
+            self.process_tx(
+                database,
+                tx,
+                None,
+                None,
+                &mut internal_max_deriv,
+                &mut external_max_deriv,
+            )?;
+        }
+
+        let current_ext = database
+            .get_last_index(KeychainKind::External)?
+            .unwrap_or(0);
+        let first_ext_new = external_max_deriv.map(|x| x + 1).unwrap_or(0);
+        if first_ext_new > current_ext {
+            info!("Setting external index to {}", first_ext_new);
+            database.set_last_index(KeychainKind::External, first_ext_new)?;
+        }
+
+        let current_int = database
+            .get_last_index(KeychainKind::Internal)?
+            .unwrap_or(0);
+        let first_int_new = internal_max_deriv.map(|x| x + 1).unwrap_or(0);
+        if first_int_new > current_int {
+            info!("Setting internal index to {}", first_int_new);
+            database.set_last_index(KeychainKind::Internal, first_int_new)?;
+        }
+
+        info!("Dropping blocks until {}", buried_height);
+        self.headers.delete_blocks_until(buried_height)?;
+
+        progress_update
+            .lock()
+            .unwrap()
+            .update(100.0, Some("Done".into()))?;
+
+        Ok(())
+    }
+}
+
+/// Data to connect to a Bitcoin P2P peer
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq, Eq)]
+pub struct BitcoinPeerConfig {
+    /// Peer address such as 127.0.0.1:18333
+    pub address: String,
+    /// Optional socks5 proxy
+    pub socks5: Option<String>,
+    /// Optional socks5 proxy credentials
+    pub socks5_credentials: Option<(String, String)>,
+}
+
+/// Configuration for a [`CompactFiltersBlockchain`]
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq, Eq)]
+pub struct CompactFiltersBlockchainConfig {
+    /// List of peers to try to connect to for asking headers and filters
+    pub peers: Vec<BitcoinPeerConfig>,
+    /// Network used
+    pub network: Network,
+    /// Storage dir to save partially downloaded headers and full blocks. Should be a separate directory per descriptor. Consider using [crate::wallet::wallet_name_from_descriptor] for this.
+    pub storage_dir: String,
+    /// Optionally skip initial `skip_blocks` blocks (default: 0)
+    pub skip_blocks: Option<usize>,
+}
+
+impl ConfigurableBlockchain for CompactFiltersBlockchain {
+    type Config = CompactFiltersBlockchainConfig;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        let mempool = Arc::new(Mempool::default());
+        let peers = config
+            .peers
+            .iter()
+            .map(|peer_conf| match &peer_conf.socks5 {
+                None => Peer::connect(&peer_conf.address, Arc::clone(&mempool), config.network),
+                Some(proxy) => Peer::connect_proxy(
+                    peer_conf.address.as_str(),
+                    proxy,
+                    peer_conf
+                        .socks5_credentials
+                        .as_ref()
+                        .map(|(a, b)| (a.as_str(), b.as_str())),
+                    Arc::clone(&mempool),
+                    config.network,
+                ),
+            })
+            .collect::<Result<_, _>>()?;
+
+        Ok(CompactFiltersBlockchain::new(
+            peers,
+            &config.storage_dir,
+            config.skip_blocks,
+        )?)
+    }
+}
+
+/// An error that can occur during sync with a [`CompactFiltersBlockchain`]
+#[derive(Debug)]
+pub enum CompactFiltersError {
+    /// A peer sent an invalid or unexpected response
+    InvalidResponse,
+    /// The headers returned are invalid
+    InvalidHeaders,
+    /// The compact filter headers returned are invalid
+    InvalidFilterHeader,
+    /// The compact filter returned is invalid
+    InvalidFilter,
+    /// The peer is missing a block in the valid chain
+    MissingBlock,
+    /// Block hash at specified height not found
+    BlockHashNotFound,
+    /// The data stored in the block filters storage are corrupted
+    DataCorruption,
+
+    /// A peer is not connected
+    NotConnected,
+    /// A peer took too long to reply to one of our messages
+    Timeout,
+    /// The peer doesn't advertise the [`BLOOM`](bitcoin::network::constants::ServiceFlags::BLOOM) service flag
+    PeerBloomDisabled,
+
+    /// No peers have been specified
+    NoPeers,
+
+    /// Internal database error
+    Db(rocksdb::Error),
+    /// Internal I/O error
+    Io(std::io::Error),
+    /// Invalid BIP158 filter
+    Bip158(bitcoin::bip158::Error),
+    /// Internal system time error
+    Time(std::time::SystemTimeError),
+
+    /// Wrapper for [`crate::error::Error`]
+    Global(Box<crate::error::Error>),
+}
+
+impl fmt::Display for CompactFiltersError {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            Self::InvalidResponse => write!(f, "A peer sent an invalid or unexpected response"),
+            Self::InvalidHeaders => write!(f, "Invalid headers"),
+            Self::InvalidFilterHeader => write!(f, "Invalid filter header"),
+            Self::InvalidFilter => write!(f, "Invalid filters"),
+            Self::MissingBlock => write!(f, "The peer is missing a block in the valid chain"),
+            Self::BlockHashNotFound => write!(f, "Block hash not found"),
+            Self::DataCorruption => write!(
+                f,
+                "The data stored in the block filters storage are corrupted"
+            ),
+            Self::NotConnected => write!(f, "A peer is not connected"),
+            Self::Timeout => write!(f, "A peer took too long to reply to one of our messages"),
+            Self::PeerBloomDisabled => write!(f, "Peer doesn't advertise the BLOOM service flag"),
+            Self::NoPeers => write!(f, "No peers have been specified"),
+            Self::Db(err) => write!(f, "Internal database error: {}", err),
+            Self::Io(err) => write!(f, "Internal I/O error: {}", err),
+            Self::Bip158(err) => write!(f, "Invalid BIP158 filter: {}", err),
+            Self::Time(err) => write!(f, "Invalid system time: {}", err),
+            Self::Global(err) => write!(f, "Generic error: {}", err),
+        }
+    }
+}
+
+impl std::error::Error for CompactFiltersError {}
+
+impl_error!(rocksdb::Error, Db, CompactFiltersError);
+impl_error!(std::io::Error, Io, CompactFiltersError);
+impl_error!(bitcoin::bip158::Error, Bip158, CompactFiltersError);
+impl_error!(std::time::SystemTimeError, Time, CompactFiltersError);
+
+impl From<crate::error::Error> for CompactFiltersError {
+    fn from(err: crate::error::Error) -> Self {
+        CompactFiltersError::Global(Box::new(err))
+    }
+}

src/blockchain/compact_filters/peer.rs

@@ -0,0 +1,576 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use std::collections::HashMap;
+use std::io::BufReader;
+use std::net::{TcpStream, ToSocketAddrs};
+use std::sync::{Arc, Condvar, Mutex, RwLock};
+use std::thread;
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+
+use socks::{Socks5Stream, ToTargetAddr};
+
+use rand::{thread_rng, Rng};
+
+use bitcoin::consensus::{Decodable, Encodable};
+use bitcoin::hash_types::BlockHash;
+use bitcoin::network::constants::ServiceFlags;
+use bitcoin::network::message::{NetworkMessage, RawNetworkMessage};
+use bitcoin::network::message_blockdata::*;
+use bitcoin::network::message_filter::*;
+use bitcoin::network::message_network::VersionMessage;
+use bitcoin::network::{Address, Magic};
+use bitcoin::{Block, Network, Transaction, Txid, Wtxid};
+
+use super::CompactFiltersError;
+
+type ResponsesMap = HashMap<&'static str, Arc<(Mutex<Vec<NetworkMessage>>, Condvar)>>;
+
+pub(crate) const TIMEOUT_SECS: u64 = 30;
+
+/// Container for unconfirmed, but valid Bitcoin transactions
+///
+/// It is normally shared between [`Peer`]s with the use of [`Arc`], so that transactions are not
+/// duplicated in memory.
+#[derive(Debug, Default)]
+pub struct Mempool(RwLock<InnerMempool>);
+
+#[derive(Debug, Default)]
+struct InnerMempool {
+    txs: HashMap<Txid, Transaction>,
+    wtxids: HashMap<Wtxid, Txid>,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+enum TxIdentifier {
+    Wtxid(Wtxid),
+    Txid(Txid),
+}
+
+impl Mempool {
+    /// Create a new empty mempool
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    /// Add a transaction to the mempool
+    ///
+    /// Note that this doesn't propagate the transaction to other
+    /// peers. To do that, [`broadcast`](crate::blockchain::Blockchain::broadcast) should be used.
+    pub fn add_tx(&self, tx: Transaction) {
+        let mut guard = self.0.write().unwrap();
+
+        guard.wtxids.insert(tx.wtxid(), tx.txid());
+        guard.txs.insert(tx.txid(), tx);
+    }
+
+    /// Look-up a transaction in the mempool given an [`Inventory`] request
+    pub fn get_tx(&self, inventory: &Inventory) -> Option<Transaction> {
+        let identifer = match inventory {
+            Inventory::Error
+            | Inventory::Block(_)
+            | Inventory::WitnessBlock(_)
+            | Inventory::CompactBlock(_) => return None,
+            Inventory::Transaction(txid) => TxIdentifier::Txid(*txid),
+            Inventory::WitnessTransaction(txid) => TxIdentifier::Txid(*txid),
+            Inventory::WTx(wtxid) => TxIdentifier::Wtxid(*wtxid),
+            Inventory::Unknown { inv_type, hash } => {
+                log::warn!(
+                    "Unknown inventory request type `{}`, hash `{:?}`",
+                    inv_type,
+                    hash
+                );
+                return None;
+            }
+        };
+
+        let txid = match identifer {
+            TxIdentifier::Txid(txid) => Some(txid),
+            TxIdentifier::Wtxid(wtxid) => self.0.read().unwrap().wtxids.get(&wtxid).cloned(),
+        };
+
+        txid.and_then(|txid| self.0.read().unwrap().txs.get(&txid).cloned())
+    }
+
+    /// Return whether or not the mempool contains a transaction with a given txid
+    pub fn has_tx(&self, txid: &Txid) -> bool {
+        self.0.read().unwrap().txs.contains_key(txid)
+    }
+
+    /// Return the list of transactions contained in the mempool
+    pub fn iter_txs(&self) -> Vec<Transaction> {
+        self.0.read().unwrap().txs.values().cloned().collect()
+    }
+}
+
+/// A Bitcoin peer
+#[derive(Debug)]
+#[allow(dead_code)]
+pub struct Peer {
+    writer: Arc<Mutex<TcpStream>>,
+    responses: Arc<RwLock<ResponsesMap>>,
+
+    reader_thread: thread::JoinHandle<()>,
+    connected: Arc<RwLock<bool>>,
+
+    mempool: Arc<Mempool>,
+
+    version: VersionMessage,
+    network: Network,
+}
+
+impl Peer {
+    /// Connect to a peer over a plaintext TCP connection
+    ///
+    /// This function internally spawns a new thread that will monitor incoming messages from the
+    /// peer, and optionally reply to some of them transparently, like [pings](bitcoin::network::message::NetworkMessage::Ping)
+    pub fn connect<A: ToSocketAddrs>(
+        address: A,
+        mempool: Arc<Mempool>,
+        network: Network,
+    ) -> Result<Self, CompactFiltersError> {
+        let stream = TcpStream::connect(address)?;
+
+        Peer::from_stream(stream, mempool, network)
+    }
+
+    /// Connect to a peer through a SOCKS5 proxy, optionally by using some credentials, specified
+    /// as a tuple of `(username, password)`
+    ///
+    /// This function internally spawns a new thread that will monitor incoming messages from the
+    /// peer, and optionally reply to some of them transparently, like [pings](NetworkMessage::Ping)
+    pub fn connect_proxy<T: ToTargetAddr, P: ToSocketAddrs>(
+        target: T,
+        proxy: P,
+        credentials: Option<(&str, &str)>,
+        mempool: Arc<Mempool>,
+        network: Network,
+    ) -> Result<Self, CompactFiltersError> {
+        let socks_stream = if let Some((username, password)) = credentials {
+            Socks5Stream::connect_with_password(proxy, target, username, password)?
+        } else {
+            Socks5Stream::connect(proxy, target)?
+        };
+
+        Peer::from_stream(socks_stream.into_inner(), mempool, network)
+    }
+
+    /// Create a [`Peer`] from an already connected TcpStream
+    fn from_stream(
+        stream: TcpStream,
+        mempool: Arc<Mempool>,
+        network: Network,
+    ) -> Result<Self, CompactFiltersError> {
+        let writer = Arc::new(Mutex::new(stream.try_clone()?));
+        let responses: Arc<RwLock<ResponsesMap>> = Arc::new(RwLock::new(HashMap::new()));
+        let connected = Arc::new(RwLock::new(true));
+
+        let mut locked_writer = writer.lock().unwrap();
+
+        let reader_thread_responses = Arc::clone(&responses);
+        let reader_thread_writer = Arc::clone(&writer);
+        let reader_thread_mempool = Arc::clone(&mempool);
+        let reader_thread_connected = Arc::clone(&connected);
+        let reader_thread = thread::spawn(move || {
+            Self::reader_thread(
+                network,
+                stream,
+                reader_thread_responses,
+                reader_thread_writer,
+                reader_thread_mempool,
+                reader_thread_connected,
+            )
+        });
+
+        let timestamp = SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs() as i64;
+        let nonce = thread_rng().gen();
+        let receiver = Address::new(&locked_writer.peer_addr()?, ServiceFlags::NONE);
+        let sender = Address {
+            services: ServiceFlags::NONE,
+            address: [0u16; 8],
+            port: 0,
+        };
+
+        Self::_send(
+            &mut locked_writer,
+            network.magic(),
+            NetworkMessage::Version(VersionMessage::new(
+                ServiceFlags::WITNESS,
+                timestamp,
+                receiver,
+                sender,
+                nonce,
+                "MagicalBitcoinWallet".into(),
+                0,
+            )),
+        )?;
+        let version = if let NetworkMessage::Version(version) =
+            Self::_recv(&responses, "version", None).unwrap()
+        {
+            version
+        } else {
+            return Err(CompactFiltersError::InvalidResponse);
+        };
+
+        if let NetworkMessage::Verack = Self::_recv(&responses, "verack", None).unwrap() {
+            Self::_send(&mut locked_writer, network.magic(), NetworkMessage::Verack)?;
+        } else {
+            return Err(CompactFiltersError::InvalidResponse);
+        }
+
+        std::mem::drop(locked_writer);
+
+        Ok(Peer {
+            writer,
+            responses,
+            reader_thread,
+            connected,
+            mempool,
+            version,
+            network,
+        })
+    }
+
+    /// Send a Bitcoin network message
+    fn _send(
+        writer: &mut TcpStream,
+        magic: Magic,
+        payload: NetworkMessage,
+    ) -> Result<(), CompactFiltersError> {
+        log::trace!("==> {:?}", payload);
+
+        let raw_message = RawNetworkMessage { magic, payload };
+
+        raw_message
+            .consensus_encode(writer)
+            .map_err(|_| CompactFiltersError::DataCorruption)?;
+
+        Ok(())
+    }
+
+    /// Wait for a specific incoming Bitcoin message, optionally with a timeout
+    fn _recv(
+        responses: &Arc<RwLock<ResponsesMap>>,
+        wait_for: &'static str,
+        timeout: Option<Duration>,
+    ) -> Option<NetworkMessage> {
+        let message_resp = {
+            let mut lock = responses.write().unwrap();
+            let message_resp = lock.entry(wait_for).or_default();
+            Arc::clone(message_resp)
+        };
+
+        let (lock, cvar) = &*message_resp;
+
+        let mut messages = lock.lock().unwrap();
+        while messages.is_empty() {
+            match timeout {
+                None => messages = cvar.wait(messages).unwrap(),
+                Some(t) => {
+                    let result = cvar.wait_timeout(messages, t).unwrap();
+                    if result.1.timed_out() {
+                        return None;
+                    }
+                    messages = result.0;
+                }
+            }
+        }
+
+        messages.pop()
+    }
+
+    /// Return the [`VersionMessage`] sent by the peer
+    pub fn get_version(&self) -> &VersionMessage {
+        &self.version
+    }
+
+    /// Return the Bitcoin [`Network`] in use
+    pub fn get_network(&self) -> Network {
+        self.network
+    }
+
+    /// Return the mempool used by this peer
+    pub fn get_mempool(&self) -> Arc<Mempool> {
+        Arc::clone(&self.mempool)
+    }
+
+    /// Return whether or not the peer is still connected
+    pub fn is_connected(&self) -> bool {
+        *self.connected.read().unwrap()
+    }
+
+    /// Internal function called once the `reader_thread` is spawned
+    fn reader_thread(
+        network: Network,
+        connection: TcpStream,
+        reader_thread_responses: Arc<RwLock<ResponsesMap>>,
+        reader_thread_writer: Arc<Mutex<TcpStream>>,
+        reader_thread_mempool: Arc<Mempool>,
+        reader_thread_connected: Arc<RwLock<bool>>,
+    ) {
+        macro_rules! check_disconnect {
+            ($call:expr) => {
+                match $call {
+                    Ok(good) => good,
+                    Err(e) => {
+                        log::debug!("Error {:?}", e);
+                        *reader_thread_connected.write().unwrap() = false;
+
+                        break;
+                    }
+                }
+            };
+        }
+
+        let mut reader = BufReader::new(connection);
+        loop {
+            let raw_message: RawNetworkMessage =
+                check_disconnect!(Decodable::consensus_decode(&mut reader));
+
+            let in_message = if raw_message.magic != network.magic() {
+                continue;
+            } else {
+                raw_message.payload
+            };
+
+            log::trace!("<== {:?}", in_message);
+
+            match in_message {
+                NetworkMessage::Ping(nonce) => {
+                    check_disconnect!(Self::_send(
+                        &mut reader_thread_writer.lock().unwrap(),
+                        network.magic(),
+                        NetworkMessage::Pong(nonce),
+                    ));
+
+                    continue;
+                }
+                NetworkMessage::Alert(_) => continue,
+                NetworkMessage::GetData(ref inv) => {
+                    let (found, not_found): (Vec<_>, Vec<_>) = inv
+                        .iter()
+                        .map(|item| (*item, reader_thread_mempool.get_tx(item)))
+                        .partition(|(_, d)| d.is_some());
+                    for (_, found_tx) in found {
+                        check_disconnect!(Self::_send(
+                            &mut reader_thread_writer.lock().unwrap(),
+                            network.magic(),
+                            NetworkMessage::Tx(found_tx.unwrap()),
+                        ));
+                    }
+
+                    if !not_found.is_empty() {
+                        check_disconnect!(Self::_send(
+                            &mut reader_thread_writer.lock().unwrap(),
+                            network.magic(),
+                            NetworkMessage::NotFound(
+                                not_found.into_iter().map(|(i, _)| i).collect(),
+                            ),
+                        ));
+                    }
+                }
+                _ => {}
+            }
+
+            let message_resp = {
+                let mut lock = reader_thread_responses.write().unwrap();
+                let message_resp = lock.entry(in_message.cmd()).or_default();
+                Arc::clone(message_resp)
+            };
+
+            let (lock, cvar) = &*message_resp;
+            let mut messages = lock.lock().unwrap();
+            messages.push(in_message);
+            cvar.notify_all();
+        }
+    }
+
+    /// Send a raw Bitcoin message to the peer
+    pub fn send(&self, payload: NetworkMessage) -> Result<(), CompactFiltersError> {
+        let mut writer = self.writer.lock().unwrap();
+        Self::_send(&mut writer, self.network.magic(), payload)
+    }
+
+    /// Waits for a specific incoming Bitcoin message, optionally with a timeout
+    pub fn recv(
+        &self,
+        wait_for: &'static str,
+        timeout: Option<Duration>,
+    ) -> Result<Option<NetworkMessage>, CompactFiltersError> {
+        Ok(Self::_recv(&self.responses, wait_for, timeout))
+    }
+}
+
+pub trait CompactFiltersPeer {
+    fn get_cf_checkpt(
+        &self,
+        filter_type: u8,
+        stop_hash: BlockHash,
+    ) -> Result<CFCheckpt, CompactFiltersError>;
+    fn get_cf_headers(
+        &self,
+        filter_type: u8,
+        start_height: u32,
+        stop_hash: BlockHash,
+    ) -> Result<CFHeaders, CompactFiltersError>;
+    fn get_cf_filters(
+        &self,
+        filter_type: u8,
+        start_height: u32,
+        stop_hash: BlockHash,
+    ) -> Result<(), CompactFiltersError>;
+    fn pop_cf_filter_resp(&self) -> Result<CFilter, CompactFiltersError>;
+}
+
+impl CompactFiltersPeer for Peer {
+    fn get_cf_checkpt(
+        &self,
+        filter_type: u8,
+        stop_hash: BlockHash,
+    ) -> Result<CFCheckpt, CompactFiltersError> {
+        self.send(NetworkMessage::GetCFCheckpt(GetCFCheckpt {
+            filter_type,
+            stop_hash,
+        }))?;
+
+        let response = self
+            .recv("cfcheckpt", Some(Duration::from_secs(TIMEOUT_SECS)))?
+            .ok_or(CompactFiltersError::Timeout)?;
+        let response = match response {
+            NetworkMessage::CFCheckpt(response) => response,
+            _ => return Err(CompactFiltersError::InvalidResponse),
+        };
+
+        if response.filter_type != filter_type {
+            return Err(CompactFiltersError::InvalidResponse);
+        }
+
+        Ok(response)
+    }
+
+    fn get_cf_headers(
+        &self,
+        filter_type: u8,
+        start_height: u32,
+        stop_hash: BlockHash,
+    ) -> Result<CFHeaders, CompactFiltersError> {
+        self.send(NetworkMessage::GetCFHeaders(GetCFHeaders {
+            filter_type,
+            start_height,
+            stop_hash,
+        }))?;
+
+        let response = self
+            .recv("cfheaders", Some(Duration::from_secs(TIMEOUT_SECS)))?
+            .ok_or(CompactFiltersError::Timeout)?;
+        let response = match response {
+            NetworkMessage::CFHeaders(response) => response,
+            _ => return Err(CompactFiltersError::InvalidResponse),
+        };
+
+        if response.filter_type != filter_type {
+            return Err(CompactFiltersError::InvalidResponse);
+        }
+
+        Ok(response)
+    }
+
+    fn pop_cf_filter_resp(&self) -> Result<CFilter, CompactFiltersError> {
+        let response = self
+            .recv("cfilter", Some(Duration::from_secs(TIMEOUT_SECS)))?
+            .ok_or(CompactFiltersError::Timeout)?;
+        let response = match response {
+            NetworkMessage::CFilter(response) => response,
+            _ => return Err(CompactFiltersError::InvalidResponse),
+        };
+
+        Ok(response)
+    }
+
+    fn get_cf_filters(
+        &self,
+        filter_type: u8,
+        start_height: u32,
+        stop_hash: BlockHash,
+    ) -> Result<(), CompactFiltersError> {
+        self.send(NetworkMessage::GetCFilters(GetCFilters {
+            filter_type,
+            start_height,
+            stop_hash,
+        }))?;
+
+        Ok(())
+    }
+}
+
+pub trait InvPeer {
+    fn get_block(&self, block_hash: BlockHash) -> Result<Option<Block>, CompactFiltersError>;
+    fn ask_for_mempool(&self) -> Result<(), CompactFiltersError>;
+    fn broadcast_tx(&self, tx: Transaction) -> Result<(), CompactFiltersError>;
+}
+
+impl InvPeer for Peer {
+    fn get_block(&self, block_hash: BlockHash) -> Result<Option<Block>, CompactFiltersError> {
+        self.send(NetworkMessage::GetData(vec![Inventory::WitnessBlock(
+            block_hash,
+        )]))?;
+
+        match self.recv("block", Some(Duration::from_secs(TIMEOUT_SECS)))? {
+            None => Ok(None),
+            Some(NetworkMessage::Block(response)) => Ok(Some(response)),
+            _ => Err(CompactFiltersError::InvalidResponse),
+        }
+    }
+
+    fn ask_for_mempool(&self) -> Result<(), CompactFiltersError> {
+        if !self.version.services.has(ServiceFlags::BLOOM) {
+            return Err(CompactFiltersError::PeerBloomDisabled);
+        }
+
+        self.send(NetworkMessage::MemPool)?;
+        let inv = match self.recv("inv", Some(Duration::from_secs(5)))? {
+            None => return Ok(()), // empty mempool
+            Some(NetworkMessage::Inv(inv)) => inv,
+            _ => return Err(CompactFiltersError::InvalidResponse),
+        };
+
+        let getdata = inv
+            .iter()
+            .cloned()
+            .filter(
+                |item| matches!(item, Inventory::Transaction(txid) if !self.mempool.has_tx(txid)),
+            )
+            .collect::<Vec<_>>();
+        let num_txs = getdata.len();
+        self.send(NetworkMessage::GetData(getdata))?;
+
+        for _ in 0..num_txs {
+            let tx = self
+                .recv("tx", Some(Duration::from_secs(TIMEOUT_SECS)))?
+                .ok_or(CompactFiltersError::Timeout)?;
+            let tx = match tx {
+                NetworkMessage::Tx(tx) => tx,
+                _ => return Err(CompactFiltersError::InvalidResponse),
+            };
+
+            self.mempool.add_tx(tx);
+        }
+
+        Ok(())
+    }
+
+    fn broadcast_tx(&self, tx: Transaction) -> Result<(), CompactFiltersError> {
+        self.mempool.add_tx(tx.clone());
+        self.send(NetworkMessage::Tx(tx))?;
+
+        Ok(())
+    }
+}

src/blockchain/compact_filters/store.rs

@@ -0,0 +1,840 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use std::convert::TryInto;
+use std::fmt;
+use std::io::{Read, Write};
+use std::marker::PhantomData;
+use std::sync::Arc;
+use std::sync::RwLock;
+
+use rand::distributions::Alphanumeric;
+use rand::{thread_rng, Rng};
+
+use rocksdb::{Direction, IteratorMode, ReadOptions, WriteBatch, DB};
+
+use bitcoin::bip158::BlockFilter;
+use bitcoin::block::Header;
+use bitcoin::blockdata::constants::genesis_block;
+use bitcoin::consensus::{deserialize, encode::VarInt, serialize, Decodable, Encodable};
+use bitcoin::hash_types::{FilterHash, FilterHeader};
+use bitcoin::hashes::Hash;
+use bitcoin::pow::Work;
+use bitcoin::Block;
+use bitcoin::BlockHash;
+use bitcoin::Network;
+use bitcoin::ScriptBuf;
+
+use super::CompactFiltersError;
+
+pub trait StoreType: Default + fmt::Debug {}
+
+#[derive(Default, Debug)]
+pub struct Full;
+impl StoreType for Full {}
+#[derive(Default, Debug)]
+pub struct Snapshot;
+impl StoreType for Snapshot {}
+
+pub enum StoreEntry {
+    BlockHeader(Option<usize>),
+    Block(Option<usize>),
+    BlockHeaderIndex(Option<BlockHash>),
+    CFilterTable((u8, Option<usize>)),
+}
+
+impl StoreEntry {
+    pub fn get_prefix(&self) -> Vec<u8> {
+        match self {
+            StoreEntry::BlockHeader(_) => b"z",
+            StoreEntry::Block(_) => b"x",
+            StoreEntry::BlockHeaderIndex(_) => b"i",
+            StoreEntry::CFilterTable(_) => b"t",
+        }
+        .to_vec()
+    }
+
+    pub fn get_key(&self) -> Vec<u8> {
+        let mut prefix = self.get_prefix();
+        match self {
+            StoreEntry::BlockHeader(Some(height)) => {
+                prefix.extend_from_slice(&height.to_be_bytes())
+            }
+            StoreEntry::Block(Some(height)) => prefix.extend_from_slice(&height.to_be_bytes()),
+            StoreEntry::BlockHeaderIndex(Some(hash)) => {
+                prefix.extend_from_slice(hash.to_raw_hash().as_ref())
+            }
+            StoreEntry::CFilterTable((filter_type, bundle_index)) => {
+                prefix.push(*filter_type);
+                if let Some(bundle_index) = bundle_index {
+                    prefix.extend_from_slice(&bundle_index.to_be_bytes());
+                }
+            }
+            _ => {}
+        }
+
+        prefix
+    }
+}
+
+pub trait SerializeDb: Sized {
+    fn serialize(&self) -> Vec<u8>;
+    fn deserialize(data: &[u8]) -> Result<Self, CompactFiltersError>;
+}
+
+impl<T> SerializeDb for T
+where
+    T: Encodable + Decodable,
+{
+    fn serialize(&self) -> Vec<u8> {
+        serialize(self)
+    }
+
+    fn deserialize(data: &[u8]) -> Result<Self, CompactFiltersError> {
+        deserialize(data).map_err(|_| CompactFiltersError::DataCorruption)
+    }
+}
+
+impl Encodable for BundleStatus {
+    fn consensus_encode<W: Write + ?Sized>(&self, e: &mut W) -> Result<usize, std::io::Error> {
+        let mut written = 0;
+
+        match self {
+            BundleStatus::Init => {
+                written += 0x00u8.consensus_encode(e)?;
+            }
+            BundleStatus::CfHeaders { cf_headers } => {
+                written += 0x01u8.consensus_encode(e)?;
+                written += VarInt(cf_headers.len() as u64).consensus_encode(e)?;
+                for header in cf_headers {
+                    written += header.consensus_encode(e)?;
+                }
+            }
+            BundleStatus::CFilters { cf_filters } => {
+                written += 0x02u8.consensus_encode(e)?;
+                written += VarInt(cf_filters.len() as u64).consensus_encode(e)?;
+                for filter in cf_filters {
+                    written += filter.consensus_encode(e)?;
+                }
+            }
+            BundleStatus::Processed { cf_filters } => {
+                written += 0x03u8.consensus_encode(e)?;
+                written += VarInt(cf_filters.len() as u64).consensus_encode(e)?;
+                for filter in cf_filters {
+                    written += filter.consensus_encode(e)?;
+                }
+            }
+            BundleStatus::Pruned => {
+                written += 0x04u8.consensus_encode(e)?;
+            }
+            BundleStatus::Tip { cf_filters } => {
+                written += 0x05u8.consensus_encode(e)?;
+                written += VarInt(cf_filters.len() as u64).consensus_encode(e)?;
+                for filter in cf_filters {
+                    written += filter.consensus_encode(e)?;
+                }
+            }
+        }
+
+        Ok(written)
+    }
+}
+
+impl Decodable for BundleStatus {
+    fn consensus_decode<D: Read + ?Sized>(
+        d: &mut D,
+    ) -> Result<Self, bitcoin::consensus::encode::Error> {
+        let byte_type = u8::consensus_decode(d)?;
+        match byte_type {
+            0x00 => Ok(BundleStatus::Init),
+            0x01 => {
+                let num = VarInt::consensus_decode(d)?;
+                let num = num.0 as usize;
+
+                let mut cf_headers = Vec::with_capacity(num);
+                for _ in 0..num {
+                    cf_headers.push(FilterHeader::consensus_decode(d)?);
+                }
+
+                Ok(BundleStatus::CfHeaders { cf_headers })
+            }
+            0x02 => {
+                let num = VarInt::consensus_decode(d)?;
+                let num = num.0 as usize;
+
+                let mut cf_filters = Vec::with_capacity(num);
+                for _ in 0..num {
+                    cf_filters.push(Vec::<u8>::consensus_decode(d)?);
+                }
+
+                Ok(BundleStatus::CFilters { cf_filters })
+            }
+            0x03 => {
+                let num = VarInt::consensus_decode(d)?;
+                let num = num.0 as usize;
+
+                let mut cf_filters = Vec::with_capacity(num);
+                for _ in 0..num {
+                    cf_filters.push(Vec::<u8>::consensus_decode(d)?);
+                }
+
+                Ok(BundleStatus::Processed { cf_filters })
+            }
+            0x04 => Ok(BundleStatus::Pruned),
+            0x05 => {
+                let num = VarInt::consensus_decode(d)?;
+                let num = num.0 as usize;
+
+                let mut cf_filters = Vec::with_capacity(num);
+                for _ in 0..num {
+                    cf_filters.push(Vec::<u8>::consensus_decode(d)?);
+                }
+
+                Ok(BundleStatus::Tip { cf_filters })
+            }
+            _ => Err(bitcoin::consensus::encode::Error::ParseFailed(
+                "Invalid byte type",
+            )),
+        }
+    }
+}
+
+pub struct ChainStore<T: StoreType> {
+    store: Arc<RwLock<DB>>,
+    cf_name: String,
+    min_height: usize,
+    network: Network,
+    phantom: PhantomData<T>,
+}
+
+impl ChainStore<Full> {
+    pub fn new(store: DB, network: Network) -> Result<Self, CompactFiltersError> {
+        let genesis = genesis_block(network);
+
+        let cf_name = "default".to_string();
+        let cf_handle = store.cf_handle(&cf_name).unwrap();
+
+        let genesis_key = StoreEntry::BlockHeader(Some(0)).get_key();
+
+        if store.get_pinned_cf(cf_handle, &genesis_key)?.is_none() {
+            let mut batch = WriteBatch::default();
+            batch.put_cf(
+                cf_handle,
+                genesis_key,
+                (genesis.header, genesis.header.work().to_be_bytes()).serialize(),
+            );
+            batch.put_cf(
+                cf_handle,
+                StoreEntry::BlockHeaderIndex(Some(genesis.block_hash())).get_key(),
+                0usize.to_be_bytes(),
+            );
+            store.write(batch)?;
+        }
+
+        Ok(ChainStore {
+            store: Arc::new(RwLock::new(store)),
+            cf_name,
+            min_height: 0,
+            network,
+            phantom: PhantomData,
+        })
+    }
+
+    pub fn get_locators(&self) -> Result<Vec<(BlockHash, usize)>, CompactFiltersError> {
+        let mut step = 1;
+        let mut index = self.get_height()?;
+        let mut answer = Vec::new();
+
+        let store_read = self.store.read().unwrap();
+        let cf_handle = store_read.cf_handle(&self.cf_name).unwrap();
+
+        loop {
+            if answer.len() > 10 {
+                step *= 2;
+            }
+
+            let (header, _): (Header, [u8; 32]) = SerializeDb::deserialize(
+                &store_read
+                    .get_pinned_cf(cf_handle, StoreEntry::BlockHeader(Some(index)).get_key())?
+                    .unwrap(),
+            )?;
+            answer.push((header.block_hash(), index));
+
+            if let Some(new_index) = index.checked_sub(step) {
+                index = new_index;
+            } else {
+                break;
+            }
+        }
+
+        Ok(answer)
+    }
+
+    pub fn start_snapshot(&self, from: usize) -> Result<ChainStore<Snapshot>, CompactFiltersError> {
+        let new_cf_name: String = thread_rng()
+            .sample_iter(&Alphanumeric)
+            .map(|byte| byte as char)
+            .take(16)
+            .collect();
+        let new_cf_name = format!("_headers:{}", new_cf_name);
+
+        let mut write_store = self.store.write().unwrap();
+
+        write_store.create_cf(&new_cf_name, &Default::default())?;
+
+        let cf_handle = write_store.cf_handle(&self.cf_name).unwrap();
+        let new_cf_handle = write_store.cf_handle(&new_cf_name).unwrap();
+
+        let (header, work): (Header, [u8; 32]) = SerializeDb::deserialize(
+            &write_store
+                .get_pinned_cf(cf_handle, StoreEntry::BlockHeader(Some(from)).get_key())?
+                .ok_or(CompactFiltersError::DataCorruption)?,
+        )?;
+        let work = Work::from_be_bytes(work);
+
+        let mut batch = WriteBatch::default();
+        batch.put_cf(
+            new_cf_handle,
+            StoreEntry::BlockHeaderIndex(Some(header.block_hash())).get_key(),
+            from.to_be_bytes(),
+        );
+        batch.put_cf(
+            new_cf_handle,
+            StoreEntry::BlockHeader(Some(from)).get_key(),
+            (header, work.to_be_bytes()).serialize(),
+        );
+        write_store.write(batch)?;
+
+        let store = Arc::clone(&self.store);
+        Ok(ChainStore {
+            store,
+            cf_name: new_cf_name,
+            min_height: from,
+            network: self.network,
+            phantom: PhantomData,
+        })
+    }
+
+    pub fn recover_snapshot(&self, cf_name: &str) -> Result<(), CompactFiltersError> {
+        let mut write_store = self.store.write().unwrap();
+        let snapshot_cf_handle = write_store.cf_handle(cf_name).unwrap();
+
+        let prefix = StoreEntry::BlockHeader(None).get_key();
+        let mut iterator = write_store.prefix_iterator_cf(snapshot_cf_handle, prefix);
+
+        let min_height = match iterator
+            .next()
+            .and_then(|(k, _)| k[1..].try_into().ok())
+            .map(usize::from_be_bytes)
+        {
+            None => {
+                std::mem::drop(iterator);
+                write_store.drop_cf(cf_name).ok();
+
+                return Ok(());
+            }
+            Some(x) => x,
+        };
+        std::mem::drop(iterator);
+        std::mem::drop(write_store);
+
+        let snapshot = ChainStore {
+            store: Arc::clone(&self.store),
+            cf_name: cf_name.into(),
+            min_height,
+            network: self.network,
+            phantom: PhantomData,
+        };
+        if snapshot.work()? > self.work()? {
+            self.apply_snapshot(snapshot)?;
+        }
+
+        Ok(())
+    }
+
+    pub fn apply_snapshot(
+        &self,
+        snaphost: ChainStore<Snapshot>,
+    ) -> Result<(), CompactFiltersError> {
+        let mut batch = WriteBatch::default();
+
+        let read_store = self.store.read().unwrap();
+        let cf_handle = read_store.cf_handle(&self.cf_name).unwrap();
+        let snapshot_cf_handle = read_store.cf_handle(&snaphost.cf_name).unwrap();
+
+        let from_key = StoreEntry::BlockHeader(Some(snaphost.min_height)).get_key();
+        let to_key = StoreEntry::BlockHeader(Some(usize::MAX)).get_key();
+
+        let mut opts = ReadOptions::default();
+        opts.set_iterate_upper_bound(to_key.clone());
+
+        log::debug!("Removing items");
+        batch.delete_range_cf(cf_handle, &from_key, &to_key);
+        for (_, v) in read_store.iterator_cf_opt(
+            cf_handle,
+            opts,
+            IteratorMode::From(&from_key, Direction::Forward),
+        ) {
+            let (header, _): (Header, [u8; 32]) = SerializeDb::deserialize(&v)?;
+
+            batch.delete_cf(
+                cf_handle,
+                StoreEntry::BlockHeaderIndex(Some(header.block_hash())).get_key(),
+            );
+        }
+
+        // Delete full blocks overridden by snapshot
+        let from_key = StoreEntry::Block(Some(snaphost.min_height)).get_key();
+        let to_key = StoreEntry::Block(Some(usize::MAX)).get_key();
+        batch.delete_range(&from_key, &to_key);
+
+        log::debug!("Copying over new items");
+        for (k, v) in read_store.iterator_cf(snapshot_cf_handle, IteratorMode::Start) {
+            batch.put_cf(cf_handle, k, v);
+        }
+
+        read_store.write(batch)?;
+        std::mem::drop(read_store);
+
+        self.store.write().unwrap().drop_cf(&snaphost.cf_name)?;
+
+        Ok(())
+    }
+
+    pub fn get_height_for(
+        &self,
+        block_hash: &BlockHash,
+    ) -> Result<Option<usize>, CompactFiltersError> {
+        let read_store = self.store.read().unwrap();
+        let cf_handle = read_store.cf_handle(&self.cf_name).unwrap();
+
+        let key = StoreEntry::BlockHeaderIndex(Some(*block_hash)).get_key();
+        let data = read_store.get_pinned_cf(cf_handle, key)?;
+        data.map(|data| {
+            Ok::<_, CompactFiltersError>(usize::from_be_bytes(
+                data.as_ref()
+                    .try_into()
+                    .map_err(|_| CompactFiltersError::DataCorruption)?,
+            ))
+        })
+        .transpose()
+    }
+
+    pub fn get_block_hash(&self, height: usize) -> Result<Option<BlockHash>, CompactFiltersError> {
+        let read_store = self.store.read().unwrap();
+        let cf_handle = read_store.cf_handle(&self.cf_name).unwrap();
+
+        let key = StoreEntry::BlockHeader(Some(height)).get_key();
+        let data = read_store.get_pinned_cf(cf_handle, key)?;
+        data.map(|data| {
+            let (header, _): (Header, [u8; 32]) =
+                deserialize(&data).map_err(|_| CompactFiltersError::DataCorruption)?;
+            Ok::<_, CompactFiltersError>(header.block_hash())
+        })
+        .transpose()
+    }
+
+    pub fn save_full_block(&self, block: &Block, height: usize) -> Result<(), CompactFiltersError> {
+        let key = StoreEntry::Block(Some(height)).get_key();
+        self.store.read().unwrap().put(key, block.serialize())?;
+
+        Ok(())
+    }
+
+    pub fn get_full_block(&self, height: usize) -> Result<Option<Block>, CompactFiltersError> {
+        let read_store = self.store.read().unwrap();
+
+        let key = StoreEntry::Block(Some(height)).get_key();
+        let opt_block = read_store.get_pinned(key)?;
+
+        opt_block
+            .map(|data| deserialize(&data))
+            .transpose()
+            .map_err(|_| CompactFiltersError::DataCorruption)
+    }
+
+    pub fn delete_blocks_until(&self, height: usize) -> Result<(), CompactFiltersError> {
+        let from_key = StoreEntry::Block(Some(0)).get_key();
+        let to_key = StoreEntry::Block(Some(height)).get_key();
+
+        let mut batch = WriteBatch::default();
+        batch.delete_range(&from_key, &to_key);
+
+        self.store.read().unwrap().write(batch)?;
+
+        Ok(())
+    }
+
+    pub fn iter_full_blocks(&self) -> Result<Vec<(usize, Block)>, CompactFiltersError> {
+        let read_store = self.store.read().unwrap();
+
+        let prefix = StoreEntry::Block(None).get_key();
+
+        let iterator = read_store.prefix_iterator(&prefix);
+        // FIXME: we have to filter manually because rocksdb sometimes returns stuff that doesn't
+        // have the right prefix
+        iterator
+            .filter(|(k, _)| k.starts_with(&prefix))
+            .map(|(k, v)| {
+                let height: usize = usize::from_be_bytes(
+                    k[1..]
+                        .try_into()
+                        .map_err(|_| CompactFiltersError::DataCorruption)?,
+                );
+                let block = SerializeDb::deserialize(&v)?;
+
+                Ok((height, block))
+            })
+            .collect::<Result<_, _>>()
+    }
+}
+
+impl<T: StoreType> ChainStore<T> {
+    pub fn work(&self) -> Result<Work, CompactFiltersError> {
+        let read_store = self.store.read().unwrap();
+        let cf_handle = read_store.cf_handle(&self.cf_name).unwrap();
+
+        let prefix = StoreEntry::BlockHeader(None).get_key();
+        let iterator = read_store.prefix_iterator_cf(cf_handle, prefix);
+
+        Ok(iterator
+            .last()
+            .map(|(_, v)| -> Result<_, CompactFiltersError> {
+                let (_, work): (Header, [u8; 32]) = SerializeDb::deserialize(&v)?;
+                let work = Work::from_be_bytes(work);
+
+                Ok(work)
+            })
+            .transpose()?
+            .unwrap_or_else(|| Work::from_be_bytes([0; 32])))
+    }
+
+    pub fn get_height(&self) -> Result<usize, CompactFiltersError> {
+        let read_store = self.store.read().unwrap();
+        let cf_handle = read_store.cf_handle(&self.cf_name).unwrap();
+
+        let prefix = StoreEntry::BlockHeader(None).get_key();
+        let iterator = read_store.prefix_iterator_cf(cf_handle, prefix);
+
+        Ok(iterator
+            .last()
+            .map(|(k, _)| -> Result<_, CompactFiltersError> {
+                let height = usize::from_be_bytes(
+                    k[1..]
+                        .try_into()
+                        .map_err(|_| CompactFiltersError::DataCorruption)?,
+                );
+
+                Ok(height)
+            })
+            .transpose()?
+            .unwrap_or_default())
+    }
+
+    pub fn get_tip_hash(&self) -> Result<Option<BlockHash>, CompactFiltersError> {
+        let read_store = self.store.read().unwrap();
+        let cf_handle = read_store.cf_handle(&self.cf_name).unwrap();
+
+        let prefix = StoreEntry::BlockHeader(None).get_key();
+        let iterator = read_store.prefix_iterator_cf(cf_handle, prefix);
+
+        iterator
+            .last()
+            .map(|(_, v)| -> Result<_, CompactFiltersError> {
+                let (header, _): (Header, [u8; 32]) = SerializeDb::deserialize(&v)?;
+
+                Ok(header.block_hash())
+            })
+            .transpose()
+    }
+
+    pub fn apply(
+        &mut self,
+        from: usize,
+        headers: Vec<Header>,
+    ) -> Result<BlockHash, CompactFiltersError> {
+        let mut batch = WriteBatch::default();
+
+        let read_store = self.store.read().unwrap();
+        let cf_handle = read_store.cf_handle(&self.cf_name).unwrap();
+
+        let (mut last_hash, mut accumulated_work) = read_store
+            .get_pinned_cf(cf_handle, StoreEntry::BlockHeader(Some(from)).get_key())?
+            .map(|result| {
+                let (header, work): (Header, [u8; 32]) = SerializeDb::deserialize(&result)?;
+                let work = Work::from_be_bytes(work);
+                Ok::<_, CompactFiltersError>((header.block_hash(), work))
+            })
+            .transpose()?
+            .ok_or(CompactFiltersError::DataCorruption)?;
+
+        for (index, header) in headers.into_iter().enumerate() {
+            if header.prev_blockhash != last_hash {
+                return Err(CompactFiltersError::InvalidHeaders);
+            }
+
+            last_hash = header.block_hash();
+            accumulated_work = accumulated_work + header.work();
+
+            let height = from + index + 1;
+            batch.put_cf(
+                cf_handle,
+                StoreEntry::BlockHeaderIndex(Some(header.block_hash())).get_key(),
+                (height).to_be_bytes(),
+            );
+            batch.put_cf(
+                cf_handle,
+                StoreEntry::BlockHeader(Some(height)).get_key(),
+                (header, accumulated_work.to_be_bytes()).serialize(),
+            );
+        }
+
+        std::mem::drop(read_store);
+
+        self.store.write().unwrap().write(batch)?;
+        Ok(last_hash)
+    }
+}
+
+impl<T: StoreType> fmt::Debug for ChainStore<T> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.debug_struct(&format!("ChainStore<{:?}>", T::default()))
+            .field("cf_name", &self.cf_name)
+            .field("min_height", &self.min_height)
+            .field("network", &self.network)
+            .field("headers_height", &self.get_height())
+            .field("tip_hash", &self.get_tip_hash())
+            .finish()
+    }
+}
+
+pub enum BundleStatus {
+    Init,
+    CfHeaders { cf_headers: Vec<FilterHeader> },
+    CFilters { cf_filters: Vec<Vec<u8>> },
+    Processed { cf_filters: Vec<Vec<u8>> },
+    Tip { cf_filters: Vec<Vec<u8>> },
+    Pruned,
+}
+
+pub struct CfStore {
+    store: Arc<RwLock<DB>>,
+    filter_type: u8,
+}
+
+type BundleEntry = (BundleStatus, FilterHeader);
+
+impl CfStore {
+    pub fn new(
+        headers_store: &ChainStore<Full>,
+        filter_type: u8,
+    ) -> Result<Self, CompactFiltersError> {
+        let cf_store = CfStore {
+            store: Arc::clone(&headers_store.store),
+            filter_type,
+        };
+
+        let genesis = genesis_block(headers_store.network);
+
+        let filter = BlockFilter::new_script_filter(&genesis, |utxo| {
+            Err::<ScriptBuf, _>(bitcoin::bip158::Error::UtxoMissing(*utxo))
+        })?;
+        let first_key = StoreEntry::CFilterTable((filter_type, Some(0))).get_key();
+
+        // Add the genesis' filter
+        {
+            let read_store = cf_store.store.read().unwrap();
+            if read_store.get_pinned(&first_key)?.is_none() {
+                read_store.put(
+                    &first_key,
+                    (
+                        BundleStatus::Init,
+                        filter.filter_header(&FilterHeader::from_raw_hash(Hash::all_zeros())),
+                    )
+                        .serialize(),
+                )?;
+            }
+        }
+
+        Ok(cf_store)
+    }
+
+    pub fn get_filter_type(&self) -> u8 {
+        self.filter_type
+    }
+
+    pub fn get_bundles(&self) -> Result<Vec<BundleEntry>, CompactFiltersError> {
+        let read_store = self.store.read().unwrap();
+
+        let prefix = StoreEntry::CFilterTable((self.filter_type, None)).get_key();
+        let iterator = read_store.prefix_iterator(&prefix);
+
+        // FIXME: we have to filter manually because rocksdb sometimes returns stuff that doesn't
+        // have the right prefix
+        iterator
+            .filter(|(k, _)| k.starts_with(&prefix))
+            .map(|(_, data)| BundleEntry::deserialize(&data))
+            .collect::<Result<_, _>>()
+    }
+
+    pub fn get_checkpoints(&self) -> Result<Vec<FilterHeader>, CompactFiltersError> {
+        let read_store = self.store.read().unwrap();
+
+        let prefix = StoreEntry::CFilterTable((self.filter_type, None)).get_key();
+        let iterator = read_store.prefix_iterator(&prefix);
+
+        // FIXME: we have to filter manually because rocksdb sometimes returns stuff that doesn't
+        // have the right prefix
+        iterator
+            .filter(|(k, _)| k.starts_with(&prefix))
+            .skip(1)
+            .map(|(_, data)| Ok::<_, CompactFiltersError>(BundleEntry::deserialize(&data)?.1))
+            .collect::<Result<_, _>>()
+    }
+
+    pub fn replace_checkpoints(
+        &self,
+        checkpoints: Vec<FilterHeader>,
+    ) -> Result<(), CompactFiltersError> {
+        let current_checkpoints = self.get_checkpoints()?;
+
+        let mut equal_bundles = 0;
+        for (index, (our, their)) in current_checkpoints
+            .iter()
+            .zip(checkpoints.iter())
+            .enumerate()
+        {
+            equal_bundles = index;
+
+            if our != their {
+                break;
+            }
+        }
+
+        let read_store = self.store.read().unwrap();
+        let mut batch = WriteBatch::default();
+
+        for (index, filter_hash) in checkpoints.iter().enumerate().skip(equal_bundles) {
+            let key = StoreEntry::CFilterTable((self.filter_type, Some(index + 1))).get_key(); // +1 to skip the genesis' filter
+
+            if let Some((BundleStatus::Tip { .. }, _)) = read_store
+                .get_pinned(&key)?
+                .map(|data| BundleEntry::deserialize(&data))
+                .transpose()?
+            {
+                println!("Keeping bundle #{} as Tip", index);
+            } else {
+                batch.put(&key, (BundleStatus::Init, *filter_hash).serialize());
+            }
+        }
+
+        read_store.write(batch)?;
+
+        Ok(())
+    }
+
+    pub fn advance_to_cf_headers(
+        &self,
+        bundle: usize,
+        checkpoint: FilterHeader,
+        filter_hashes: Vec<FilterHash>,
+    ) -> Result<BundleStatus, CompactFiltersError> {
+        let cf_headers: Vec<FilterHeader> = filter_hashes
+            .into_iter()
+            .scan(checkpoint, |prev_header, filter_hash| {
+                let filter_header = filter_hash.filter_header(prev_header);
+                *prev_header = filter_header;
+
+                Some(filter_header)
+            })
+            .collect();
+
+        let read_store = self.store.read().unwrap();
+
+        let next_key = StoreEntry::CFilterTable((self.filter_type, Some(bundle + 1))).get_key(); // +1 to skip the genesis' filter
+        if let Some((_, next_checkpoint)) = read_store
+            .get_pinned(&next_key)?
+            .map(|data| BundleEntry::deserialize(&data))
+            .transpose()?
+        {
+            // check connection with the next bundle if present
+            if cf_headers.iter().last() != Some(&next_checkpoint) {
+                return Err(CompactFiltersError::InvalidFilterHeader);
+            }
+        }
+
+        let key = StoreEntry::CFilterTable((self.filter_type, Some(bundle))).get_key();
+        let value = (BundleStatus::CfHeaders { cf_headers }, checkpoint);
+
+        read_store.put(key, value.serialize())?;
+
+        Ok(value.0)
+    }
+
+    pub fn advance_to_cf_filters(
+        &self,
+        bundle: usize,
+        checkpoint: FilterHeader,
+        headers: Vec<FilterHeader>,
+        filters: Vec<(usize, Vec<u8>)>,
+    ) -> Result<BundleStatus, CompactFiltersError> {
+        let cf_filters = filters
+            .into_iter()
+            .zip(headers.into_iter())
+            .scan(checkpoint, |prev_header, ((_, filter_content), header)| {
+                let filter = BlockFilter::new(&filter_content);
+                if header != filter.filter_header(prev_header) {
+                    return Some(Err(CompactFiltersError::InvalidFilter));
+                }
+                *prev_header = header;
+
+                Some(Ok::<_, CompactFiltersError>(filter_content))
+            })
+            .collect::<Result<_, _>>()?;
+
+        let key = StoreEntry::CFilterTable((self.filter_type, Some(bundle))).get_key();
+        let value = (BundleStatus::CFilters { cf_filters }, checkpoint);
+
+        let read_store = self.store.read().unwrap();
+        read_store.put(key, value.serialize())?;
+
+        Ok(value.0)
+    }
+
+    pub fn prune_filters(
+        &self,
+        bundle: usize,
+        checkpoint: FilterHeader,
+    ) -> Result<BundleStatus, CompactFiltersError> {
+        let key = StoreEntry::CFilterTable((self.filter_type, Some(bundle))).get_key();
+        let value = (BundleStatus::Pruned, checkpoint);
+
+        let read_store = self.store.read().unwrap();
+        read_store.put(key, value.serialize())?;
+
+        Ok(value.0)
+    }
+
+    pub fn mark_as_tip(
+        &self,
+        bundle: usize,
+        cf_filters: Vec<Vec<u8>>,
+        checkpoint: FilterHeader,
+    ) -> Result<BundleStatus, CompactFiltersError> {
+        let key = StoreEntry::CFilterTable((self.filter_type, Some(bundle))).get_key();
+        let value = (BundleStatus::Tip { cf_filters }, checkpoint);
+
+        let read_store = self.store.read().unwrap();
+        read_store.put(key, value.serialize())?;
+
+        Ok(value.0)
+    }
+}

src/blockchain/compact_filters/sync.rs

@@ -0,0 +1,297 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use std::collections::{BTreeMap, HashMap, VecDeque};
+use std::sync::{Arc, Mutex};
+use std::time::Duration;
+
+use bitcoin::bip158::BlockFilter;
+use bitcoin::hash_types::{BlockHash, FilterHeader};
+use bitcoin::hashes::Hash;
+use bitcoin::network::message::NetworkMessage;
+use bitcoin::network::message_blockdata::GetHeadersMessage;
+
+use super::peer::*;
+use super::store::*;
+use super::CompactFiltersError;
+use crate::error::Error;
+
+pub(crate) const BURIED_CONFIRMATIONS: usize = 100;
+
+pub struct CfSync {
+    headers_store: Arc<ChainStore<Full>>,
+    cf_store: Arc<CfStore>,
+    skip_blocks: usize,
+    bundles: Mutex<VecDeque<(BundleStatus, FilterHeader, usize)>>,
+}
+
+impl CfSync {
+    pub fn new(
+        headers_store: Arc<ChainStore<Full>>,
+        skip_blocks: usize,
+        filter_type: u8,
+    ) -> Result<Self, CompactFiltersError> {
+        let cf_store = Arc::new(CfStore::new(&headers_store, filter_type)?);
+
+        Ok(CfSync {
+            headers_store,
+            cf_store,
+            skip_blocks,
+            bundles: Mutex::new(VecDeque::new()),
+        })
+    }
+
+    pub fn pruned_bundles(&self) -> Result<usize, CompactFiltersError> {
+        Ok(self
+            .cf_store
+            .get_bundles()?
+            .into_iter()
+            .skip(self.skip_blocks / 1000)
+            .fold(0, |acc, (status, _)| match status {
+                BundleStatus::Pruned => acc + 1,
+                _ => acc,
+            }))
+    }
+
+    pub fn prepare_sync(&self, peer: Arc<Peer>) -> Result<(), CompactFiltersError> {
+        let mut bundles_lock = self.bundles.lock().unwrap();
+
+        let resp = peer.get_cf_checkpt(
+            self.cf_store.get_filter_type(),
+            self.headers_store.get_tip_hash()?.unwrap(),
+        )?;
+        self.cf_store.replace_checkpoints(resp.filter_headers)?;
+
+        bundles_lock.clear();
+        for (index, (status, checkpoint)) in self.cf_store.get_bundles()?.into_iter().enumerate() {
+            bundles_lock.push_back((status, checkpoint, index));
+        }
+
+        Ok(())
+    }
+
+    pub fn capture_thread_for_sync<F, Q>(
+        &self,
+        peer: Arc<Peer>,
+        process: F,
+        completed_bundle: Q,
+    ) -> Result<(), CompactFiltersError>
+    where
+        F: Fn(&BlockHash, &BlockFilter) -> Result<bool, CompactFiltersError>,
+        Q: Fn(usize) -> Result<(), Error>,
+    {
+        let current_height = self.headers_store.get_height()?; // TODO: we should update it in case headers_store is also updated
+
+        loop {
+            let (mut status, checkpoint, index) = match self.bundles.lock().unwrap().pop_front() {
+                None => break,
+                Some(x) => x,
+            };
+
+            log::debug!(
+                "Processing bundle #{} - height {} to {}",
+                index,
+                index * 1000 + 1,
+                (index + 1) * 1000
+            );
+
+            let process_received_filters =
+                |expected_filters| -> Result<BTreeMap<usize, Vec<u8>>, CompactFiltersError> {
+                    let mut filters_map = BTreeMap::new();
+                    for _ in 0..expected_filters {
+                        let filter = peer.pop_cf_filter_resp()?;
+                        if filter.filter_type != self.cf_store.get_filter_type() {
+                            return Err(CompactFiltersError::InvalidResponse);
+                        }
+
+                        match self.headers_store.get_height_for(&filter.block_hash)? {
+                            Some(height) => filters_map.insert(height, filter.filter),
+                            None => return Err(CompactFiltersError::InvalidFilter),
+                        };
+                    }
+
+                    Ok(filters_map)
+                };
+
+            let start_height = index * 1000 + 1;
+            let mut already_processed = 0;
+
+            if start_height < self.skip_blocks {
+                status = self.cf_store.prune_filters(index, checkpoint)?;
+            }
+
+            let stop_height = std::cmp::min(current_height, start_height + 999);
+            let stop_hash = self.headers_store.get_block_hash(stop_height)?.unwrap();
+
+            if let BundleStatus::Init = status {
+                log::trace!("status: Init");
+
+                let resp = peer.get_cf_headers(0x00, start_height as u32, stop_hash)?;
+
+                assert_eq!(resp.previous_filter_header, checkpoint);
+                status =
+                    self.cf_store
+                        .advance_to_cf_headers(index, checkpoint, resp.filter_hashes)?;
+            }
+            if let BundleStatus::Tip { cf_filters } = status {
+                log::trace!("status: Tip (beginning) ");
+
+                already_processed = cf_filters.len();
+                let headers_resp = peer.get_cf_headers(0x00, start_height as u32, stop_hash)?;
+
+                let cf_headers = match self.cf_store.advance_to_cf_headers(
+                    index,
+                    checkpoint,
+                    headers_resp.filter_hashes,
+                )? {
+                    BundleStatus::CfHeaders { cf_headers } => cf_headers,
+                    _ => return Err(CompactFiltersError::InvalidResponse),
+                };
+
+                peer.get_cf_filters(
+                    self.cf_store.get_filter_type(),
+                    (start_height + cf_filters.len()) as u32,
+                    stop_hash,
+                )?;
+                let expected_filters = stop_height - start_height + 1 - cf_filters.len();
+                let filters_map = process_received_filters(expected_filters)?;
+                let filters = cf_filters
+                    .into_iter()
+                    .enumerate()
+                    .chain(filters_map.into_iter())
+                    .collect();
+                status = self
+                    .cf_store
+                    .advance_to_cf_filters(index, checkpoint, cf_headers, filters)?;
+            }
+            if let BundleStatus::CfHeaders { cf_headers } = status {
+                log::trace!("status: CFHeaders");
+
+                peer.get_cf_filters(
+                    self.cf_store.get_filter_type(),
+                    start_height as u32,
+                    stop_hash,
+                )?;
+                let expected_filters = stop_height - start_height + 1;
+                let filters_map = process_received_filters(expected_filters)?;
+                status = self.cf_store.advance_to_cf_filters(
+                    index,
+                    checkpoint,
+                    cf_headers,
+                    filters_map.into_iter().collect(),
+                )?;
+            }
+            if let BundleStatus::CFilters { cf_filters } = status {
+                log::trace!("status: CFilters");
+
+                let last_sync_buried_height =
+                    (start_height + already_processed).saturating_sub(BURIED_CONFIRMATIONS);
+
+                for (filter_index, filter) in cf_filters.iter().enumerate() {
+                    let height = filter_index + start_height;
+
+                    // do not download blocks that were already "buried" since the last sync
+                    if height < last_sync_buried_height {
+                        continue;
+                    }
+
+                    let block_hash = self.headers_store.get_block_hash(height)?.unwrap();
+
+                    // TODO: also download random blocks?
+                    if process(&block_hash, &BlockFilter::new(filter))? {
+                        log::debug!("Downloading block {}", block_hash);
+
+                        let block = peer
+                            .get_block(block_hash)?
+                            .ok_or(CompactFiltersError::MissingBlock)?;
+                        self.headers_store.save_full_block(&block, height)?;
+                    }
+                }
+
+                status = BundleStatus::Processed { cf_filters };
+            }
+            if let BundleStatus::Processed { cf_filters } = status {
+                log::trace!("status: Processed");
+
+                if current_height - stop_height > 1000 {
+                    status = self.cf_store.prune_filters(index, checkpoint)?;
+                } else {
+                    status = self.cf_store.mark_as_tip(index, cf_filters, checkpoint)?;
+                }
+
+                completed_bundle(index)?;
+            }
+            if let BundleStatus::Pruned = status {
+                log::trace!("status: Pruned");
+            }
+            if let BundleStatus::Tip { .. } = status {
+                log::trace!("status: Tip");
+            }
+        }
+
+        Ok(())
+    }
+}
+
+pub fn sync_headers<F>(
+    peer: Arc<Peer>,
+    store: Arc<ChainStore<Full>>,
+    sync_fn: F,
+) -> Result<Option<ChainStore<Snapshot>>, CompactFiltersError>
+where
+    F: Fn(usize) -> Result<(), Error>,
+{
+    let locators = store.get_locators()?;
+    let locators_vec = locators.iter().map(|(hash, _)| hash).cloned().collect();
+    let locators_map: HashMap<_, _> = locators.into_iter().collect();
+
+    peer.send(NetworkMessage::GetHeaders(GetHeadersMessage::new(
+        locators_vec,
+        Hash::all_zeros(),
+    )))?;
+    let (mut snapshot, mut last_hash) = if let NetworkMessage::Headers(headers) = peer
+        .recv("headers", Some(Duration::from_secs(TIMEOUT_SECS)))?
+        .ok_or(CompactFiltersError::Timeout)?
+    {
+        if headers.is_empty() {
+            return Ok(None);
+        }
+
+        match locators_map.get(&headers[0].prev_blockhash) {
+            None => return Err(CompactFiltersError::InvalidHeaders),
+            Some(from) => (store.start_snapshot(*from)?, headers[0].prev_blockhash),
+        }
+    } else {
+        return Err(CompactFiltersError::InvalidResponse);
+    };
+
+    let mut sync_height = store.get_height()?;
+    while sync_height < peer.get_version().start_height as usize {
+        peer.send(NetworkMessage::GetHeaders(GetHeadersMessage::new(
+            vec![last_hash],
+            Hash::all_zeros(),
+        )))?;
+        if let NetworkMessage::Headers(headers) = peer
+            .recv("headers", Some(Duration::from_secs(TIMEOUT_SECS)))?
+            .ok_or(CompactFiltersError::Timeout)?
+        {
+            let batch_len = headers.len();
+            last_hash = snapshot.apply(sync_height, headers)?;
+
+            sync_height += batch_len;
+            sync_fn(sync_height)?;
+        } else {
+            return Err(CompactFiltersError::InvalidResponse);
+        }
+    }
+
+    Ok(Some(snapshot))
+}

src/blockchain/electrum.rs

@@ -1,147 +1,432 @@
-use std::collections::HashSet;
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Electrum
+//!
+//! This module defines a [`Blockchain`] struct that wraps an [`electrum_client::Client`]
+//! and implements the logic required to populate the wallet's [database](crate::database::Database) by
+//! querying the inner client.
+//!
+//! ## Example
+//!
+//! ```no_run
+//! # use bdk::blockchain::electrum::ElectrumBlockchain;
+//! let client = electrum_client::Client::new("ssl://electrum.blockstream.info:50002")?;
+//! let blockchain = ElectrumBlockchain::from(client);
+//! # Ok::<(), bdk::Error>(())
+//! ```
+
+use std::collections::{HashMap, HashSet};
+use std::ops::{Deref, DerefMut};
 
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
-use bitcoin::{Script, Transaction, Txid};
+use bitcoin::{Transaction, Txid};
 
-use electrum_client::tokio::io::{AsyncRead, AsyncWrite};
-use electrum_client::Client;
+use electrum_client::{Client, ConfigBuilder, ElectrumApi, Socks5Config};
 
-use self::utils::{ELSGetHistoryRes, ELSListUnspentRes, ElectrumLikeSync};
+use super::script_sync::Request;
 use super::*;
-use crate::database::{BatchDatabase, DatabaseUtils};
+use crate::database::{BatchDatabase, Database};
 use crate::error::Error;
+use crate::{BlockTime, FeeRate};
 
-pub struct ElectrumBlockchain<T: AsyncRead + AsyncWrite + Send>(Option<Client<T>>);
+/// Wrapper over an Electrum Client that implements the required blockchain traits
+///
+/// ## Example
+/// See the [`blockchain::electrum`](crate::blockchain::electrum) module for a usage example.
+pub struct ElectrumBlockchain {
+    client: Client,
+    stop_gap: usize,
+}
 
-impl<T: AsyncRead + AsyncWrite + Send> std::convert::From<Client<T>> for ElectrumBlockchain<T> {
-    fn from(client: Client<T>) -> Self {
-        ElectrumBlockchain(Some(client))
+impl std::convert::From<Client> for ElectrumBlockchain {
+    fn from(client: Client) -> Self {
+        ElectrumBlockchain {
+            client,
+            stop_gap: 20,
+        }
     }
 }
 
-impl<T: AsyncRead + AsyncWrite + Send> Blockchain for ElectrumBlockchain<T> {
-    fn offline() -> Self {
-        ElectrumBlockchain(None)
+impl Blockchain for ElectrumBlockchain {
+    fn get_capabilities(&self) -> HashSet<Capability> {
+        vec![
+            Capability::FullHistory,
+            Capability::GetAnyTx,
+            Capability::AccurateFees,
+        ]
+        .into_iter()
+        .collect()
     }
 
-    fn is_online(&self) -> bool {
-        self.0.is_some()
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        Ok(self.client.transaction_broadcast(tx).map(|_| ())?)
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        Ok(FeeRate::from_btc_per_kvb(
+            self.client.estimate_fee(target)? as f32
+        ))
     }
 }
 
-#[async_trait(?Send)]
-impl<T: AsyncRead + AsyncWrite + Send> OnlineBlockchain for ElectrumBlockchain<T> {
-    async fn get_capabilities(&self) -> HashSet<Capability> {
-        vec![Capability::FullHistory, Capability::GetAnyTx]
-            .into_iter()
-            .collect()
-    }
+impl Deref for ElectrumBlockchain {
+    type Target = Client;
 
-    async fn setup<D: BatchDatabase + DatabaseUtils, P: Progress>(
-        &mut self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        self.0
-            .as_mut()
-            .ok_or(Error::OfflineClient)?
-            .electrum_like_setup(stop_gap, database, progress_update)
-            .await
+    fn deref(&self) -> &Self::Target {
+        &self.client
     }
+}
 
-    async fn get_tx(&mut self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self
-            .0
-            .as_mut()
-            .ok_or(Error::OfflineClient)?
-            .transaction_get(txid)
-            .await
-            .map(Option::Some)?)
-    }
+impl StatelessBlockchain for ElectrumBlockchain {}
 
-    async fn broadcast(&mut self, tx: &Transaction) -> Result<(), Error> {
-        Ok(self
-            .0
-            .as_mut()
-            .ok_or(Error::OfflineClient)?
-            .transaction_broadcast(tx)
-            .await
-            .map(|_| ())?)
-    }
-
-    async fn get_height(&mut self) -> Result<usize, Error> {
+impl GetHeight for ElectrumBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
         // TODO: unsubscribe when added to the client, or is there a better call to use here?
 
         Ok(self
-            .0
-            .as_mut()
-            .ok_or(Error::OfflineClient)?
+            .client
             .block_headers_subscribe()
-            .await
-            .map(|data| data.height)?)
+            .map(|data| data.height as u32)?)
     }
 }
 
-#[async_trait(?Send)]
-impl<T: AsyncRead + AsyncWrite + Send> ElectrumLikeSync for Client<T> {
-    async fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script>>(
-        &mut self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ELSGetHistoryRes>>, Error> {
-        self.batch_script_get_history(scripts)
-            .await
-            .map(|v| {
-                v.into_iter()
-                    .map(|v| {
-                        v.into_iter()
-                            .map(
-                                |electrum_client::GetHistoryRes {
-                                     height, tx_hash, ..
-                                 }| ELSGetHistoryRes {
-                                    height,
-                                    tx_hash,
-                                },
-                            )
-                            .collect()
-                    })
-                    .collect()
-            })
-            .map_err(Error::Electrum)
-    }
-
-    async fn els_batch_script_list_unspent<'s, I: IntoIterator<Item = &'s Script>>(
-        &mut self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ELSListUnspentRes>>, Error> {
-        self.batch_script_list_unspent(scripts)
-            .await
-            .map(|v| {
-                v.into_iter()
-                    .map(|v| {
-                        v.into_iter()
-                            .map(
-                                |electrum_client::ListUnspentRes {
-                                     height,
-                                     tx_hash,
-                                     tx_pos,
-                                     ..
-                                 }| ELSListUnspentRes {
-                                    height,
-                                    tx_hash,
-                                    tx_pos,
-                                },
-                            )
-                            .collect()
-                    })
-                    .collect()
-            })
-            .map_err(Error::Electrum)
-    }
-
-    async fn els_transaction_get(&mut self, txid: &Txid) -> Result<Transaction, Error> {
-        self.transaction_get(txid).await.map_err(Error::Electrum)
+impl GetTx for ElectrumBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.client.transaction_get(txid).map(Option::Some)?)
+    }
+}
+
+impl GetBlockHash for ElectrumBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        let block_header = self.client.block_header(height as usize)?;
+        Ok(block_header.block_hash())
+    }
+}
+
+impl WalletSync for ElectrumBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &RefCell<D>,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        let mut database = database.borrow_mut();
+        let database = database.deref_mut();
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut block_times = HashMap::<u32, u32>::new();
+        let mut txid_to_height = HashMap::<Txid, u32>::new();
+        let mut tx_cache = TxCache::new(database, &self.client);
+
+        // Set chunk_size to the smallest value capable of finding a gap greater than stop_gap.
+        let chunk_size = self.stop_gap + 1;
+
+        // The electrum server has been inconsistent somehow in its responses during sync. For
+        // example, we do a batch request of transactions and the response contains less
+        // tranascations than in the request. This should never happen but we don't want to panic.
+        let electrum_goof = || Error::Generic("electrum server misbehaving".to_string());
+
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let scripts = script_req.request().take(chunk_size);
+                    let txids_per_script: Vec<Vec<_>> = self
+                        .client
+                        .batch_script_get_history(scripts)
+                        .map_err(Error::Electrum)?
+                        .into_iter()
+                        .map(|txs| {
+                            txs.into_iter()
+                                .map(|tx| {
+                                    let tx_height = match tx.height {
+                                        none if none <= 0 => None,
+                                        height => {
+                                            txid_to_height.insert(tx.tx_hash, height as u32);
+                                            Some(height as u32)
+                                        }
+                                    };
+                                    (tx.tx_hash, tx_height)
+                                })
+                                .collect()
+                        })
+                        .collect();
+
+                    script_req.satisfy(txids_per_script)?
+                }
+
+                Request::Conftime(conftime_req) => {
+                    // collect up to chunk_size heights to fetch from electrum
+                    let needs_block_height = conftime_req
+                        .request()
+                        .filter_map(|txid| txid_to_height.get(txid).cloned())
+                        .filter(|height| block_times.get(height).is_none())
+                        .take(chunk_size)
+                        .collect::<HashSet<u32>>();
+
+                    let new_block_headers = self
+                        .client
+                        .batch_block_header(needs_block_height.iter().cloned())?;
+
+                    for (height, header) in needs_block_height.into_iter().zip(new_block_headers) {
+                        block_times.insert(height, header.time);
+                    }
+
+                    let conftimes = conftime_req
+                        .request()
+                        .take(chunk_size)
+                        .map(|txid| {
+                            let confirmation_time = txid_to_height
+                                .get(txid)
+                                .map(|height| {
+                                    let timestamp =
+                                        *block_times.get(height).ok_or_else(electrum_goof)?;
+                                    Result::<_, Error>::Ok(BlockTime {
+                                        height: *height,
+                                        timestamp: timestamp.into(),
+                                    })
+                                })
+                                .transpose()?;
+                            Ok(confirmation_time)
+                        })
+                        .collect::<Result<_, Error>>()?;
+
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let needs_full = tx_req.request().take(chunk_size);
+                    tx_cache.save_txs(needs_full.clone())?;
+                    let full_transactions = needs_full
+                        .map(|txid| tx_cache.get(*txid).ok_or_else(electrum_goof))
+                        .collect::<Result<Vec<_>, _>>()?;
+                    let input_txs = full_transactions.iter().flat_map(|tx| {
+                        tx.input
+                            .iter()
+                            .filter(|input| !input.previous_output.is_null())
+                            .map(|input| &input.previous_output.txid)
+                    });
+                    tx_cache.save_txs(input_txs)?;
+
+                    let full_details = full_transactions
+                        .into_iter()
+                        .map(|tx| {
+                            let mut input_index = 0usize;
+                            let prev_outputs = tx
+                                .input
+                                .iter()
+                                .map(|input| {
+                                    if input.previous_output.is_null() {
+                                        return Ok(None);
+                                    }
+                                    let prev_tx = tx_cache
+                                        .get(input.previous_output.txid)
+                                        .ok_or_else(electrum_goof)?;
+                                    let txout = prev_tx
+                                        .output
+                                        .get(input.previous_output.vout as usize)
+                                        .ok_or_else(electrum_goof)?;
+                                    input_index += 1;
+                                    Ok(Some(txout.clone()))
+                                })
+                                .collect::<Result<Vec<_>, Error>>()?;
+                            Ok((prev_outputs, tx))
+                        })
+                        .collect::<Result<Vec<_>, Error>>()?;
+
+                    tx_req.satisfy(full_details)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+        Ok(())
+    }
+}
+
+struct TxCache<'a, 'b, D> {
+    db: &'a D,
+    client: &'b Client,
+    cache: HashMap<Txid, Transaction>,
+}
+
+impl<'a, 'b, D: Database> TxCache<'a, 'b, D> {
+    fn new(db: &'a D, client: &'b Client) -> Self {
+        TxCache {
+            db,
+            client,
+            cache: HashMap::default(),
+        }
+    }
+    fn save_txs<'c>(&mut self, txids: impl Iterator<Item = &'c Txid>) -> Result<(), Error> {
+        let mut need_fetch = vec![];
+        for txid in txids {
+            if self.cache.get(txid).is_some() {
+                continue;
+            } else if let Some(transaction) = self.db.get_raw_tx(txid)? {
+                self.cache.insert(*txid, transaction);
+            } else {
+                need_fetch.push(txid);
+            }
+        }
+
+        if !need_fetch.is_empty() {
+            let txs = self
+                .client
+                .batch_transaction_get(need_fetch.clone())
+                .map_err(Error::Electrum)?;
+            let mut txs: HashMap<_, _> = txs.into_iter().map(|tx| (tx.txid(), tx)).collect();
+            for txid in need_fetch {
+                if let Some(tx) = txs.remove(txid) {
+                    self.cache.insert(*txid, tx);
+                }
+            }
+        }
+
+        Ok(())
+    }
+
+    fn get(&self, txid: Txid) -> Option<Transaction> {
+        self.cache.get(&txid).map(Clone::clone)
+    }
+}
+
+/// Configuration for an [`ElectrumBlockchain`]
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq, Eq)]
+pub struct ElectrumBlockchainConfig {
+    /// URL of the Electrum server (such as ElectrumX, Esplora, BWT) may start with `ssl://` or `tcp://` and include a port
+    ///
+    /// eg. `ssl://electrum.blockstream.info:60002`
+    pub url: String,
+    /// URL of the socks5 proxy server or a Tor service
+    pub socks5: Option<String>,
+    /// Request retry count
+    pub retry: u8,
+    /// Request timeout (seconds)
+    pub timeout: Option<u8>,
+    /// Stop searching addresses for transactions after finding an unused gap of this length
+    pub stop_gap: usize,
+    /// Validate the domain when using SSL
+    pub validate_domain: bool,
+}
+
+impl ConfigurableBlockchain for ElectrumBlockchain {
+    type Config = ElectrumBlockchainConfig;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        let socks5 = config.socks5.as_ref().map(Socks5Config::new);
+        let electrum_config = ConfigBuilder::new()
+            .retry(config.retry)
+            .timeout(config.timeout)
+            .socks5(socks5)
+            .validate_domain(config.validate_domain)
+            .build();
+
+        Ok(ElectrumBlockchain {
+            client: Client::from_config(config.url.as_str(), electrum_config)?,
+            stop_gap: config.stop_gap,
+        })
+    }
+}
+
+#[cfg(test)]
+#[cfg(feature = "test-electrum")]
+mod test {
+    use std::sync::Arc;
+
+    use super::*;
+    use crate::database::MemoryDatabase;
+    use crate::testutils::blockchain_tests::TestClient;
+    use crate::testutils::configurable_blockchain_tests::ConfigurableBlockchainTester;
+    use crate::wallet::{AddressIndex, Wallet};
+
+    crate::bdk_blockchain_tests! {
+        fn test_instance(test_client: &TestClient) -> ElectrumBlockchain {
+            ElectrumBlockchain::from(Client::new(&test_client.electrsd.electrum_url).unwrap())
+        }
+    }
+
+    fn get_factory() -> (TestClient, Arc<ElectrumBlockchain>) {
+        let test_client = TestClient::default();
+
+        let factory = Arc::new(ElectrumBlockchain::from(
+            Client::new(&test_client.electrsd.electrum_url).unwrap(),
+        ));
+
+        (test_client, factory)
+    }
+
+    #[test]
+    fn test_electrum_blockchain_factory() {
+        let (_test_client, factory) = get_factory();
+
+        let a = factory.build("aaaaaa", None).unwrap();
+        let b = factory.build("bbbbbb", None).unwrap();
+
+        assert_eq!(
+            a.client.block_headers_subscribe().unwrap().height,
+            b.client.block_headers_subscribe().unwrap().height
+        );
+    }
+
+    #[test]
+    fn test_electrum_blockchain_factory_sync_wallet() {
+        let (mut test_client, factory) = get_factory();
+
+        let db = MemoryDatabase::new();
+        let wallet = Wallet::new(
+            "wpkh(L5EZftvrYaSudiozVRzTqLcHLNDoVn7H5HSfM9BAN6tMJX8oTWz6)",
+            None,
+            bitcoin::Network::Regtest,
+            db,
+        )
+        .unwrap();
+
+        let address = wallet.get_address(AddressIndex::New).unwrap();
+
+        let tx = testutils! {
+            @tx ( (@addr address.address) => 50_000 )
+        };
+        test_client.receive(tx);
+
+        factory
+            .sync_wallet(&wallet, None, Default::default())
+            .unwrap();
+
+        assert_eq!(wallet.get_balance().unwrap().untrusted_pending, 50_000);
+    }
+
+    #[test]
+    fn test_electrum_with_variable_configs() {
+        struct ElectrumTester;
+
+        impl ConfigurableBlockchainTester<ElectrumBlockchain> for ElectrumTester {
+            const BLOCKCHAIN_NAME: &'static str = "Electrum";
+
+            fn config_with_stop_gap(
+                &self,
+                test_client: &mut TestClient,
+                stop_gap: usize,
+            ) -> Option<ElectrumBlockchainConfig> {
+                Some(ElectrumBlockchainConfig {
+                    url: test_client.electrsd.electrum_url.clone(),
+                    socks5: None,
+                    retry: 0,
+                    timeout: None,
+                    stop_gap: stop_gap,
+                    validate_domain: true,
+                })
+            }
+        }
+
+        ElectrumTester.run();
     }
 }

src/blockchain/esplora.rs

@@ -1,314 +0,0 @@
-use std::collections::HashSet;
-
-use futures::stream::{self, StreamExt, TryStreamExt};
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
-
-use serde::Deserialize;
-
-use reqwest::Client;
-use reqwest::StatusCode;
-
-use bitcoin::consensus::{deserialize, serialize};
-use bitcoin::hashes::hex::ToHex;
-use bitcoin::hashes::{sha256, Hash};
-use bitcoin::{Script, Transaction, Txid};
-
-use self::utils::{ELSGetHistoryRes, ELSListUnspentRes, ElectrumLikeSync};
-use super::*;
-use crate::database::{BatchDatabase, DatabaseUtils};
-use crate::error::Error;
-
-#[derive(Debug)]
-pub struct UrlClient {
-    url: String,
-    client: Client,
-}
-
-#[derive(Debug)]
-pub struct EsploraBlockchain(Option<UrlClient>);
-
-impl std::convert::From<UrlClient> for EsploraBlockchain {
-    fn from(url_client: UrlClient) -> Self {
-        EsploraBlockchain(Some(url_client))
-    }
-}
-
-impl EsploraBlockchain {
-    pub fn new(base_url: &str) -> Self {
-        EsploraBlockchain(Some(UrlClient {
-            url: base_url.to_string(),
-            client: Client::new(),
-        }))
-    }
-}
-
-impl Blockchain for EsploraBlockchain {
-    fn offline() -> Self {
-        EsploraBlockchain(None)
-    }
-
-    fn is_online(&self) -> bool {
-        self.0.is_some()
-    }
-}
-
-#[async_trait(?Send)]
-impl OnlineBlockchain for EsploraBlockchain {
-    async fn get_capabilities(&self) -> HashSet<Capability> {
-        vec![Capability::FullHistory, Capability::GetAnyTx]
-            .into_iter()
-            .collect()
-    }
-
-    async fn setup<D: BatchDatabase + DatabaseUtils, P: Progress>(
-        &mut self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
-    ) -> Result<(), Error> {
-        self.0
-            .as_mut()
-            .ok_or(Error::OfflineClient)?
-            .electrum_like_setup(stop_gap, database, progress_update)
-            .await
-    }
-
-    async fn get_tx(&mut self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self
-            .0
-            .as_mut()
-            .ok_or(Error::OfflineClient)?
-            ._get_tx(txid)
-            .await?)
-    }
-
-    async fn broadcast(&mut self, tx: &Transaction) -> Result<(), Error> {
-        Ok(self
-            .0
-            .as_mut()
-            .ok_or(Error::OfflineClient)?
-            ._broadcast(tx)
-            .await?)
-    }
-
-    async fn get_height(&mut self) -> Result<usize, Error> {
-        Ok(self
-            .0
-            .as_mut()
-            .ok_or(Error::OfflineClient)?
-            ._get_height()
-            .await?)
-    }
-}
-
-impl UrlClient {
-    fn script_to_scripthash(script: &Script) -> String {
-        sha256::Hash::hash(script.as_bytes()).into_inner().to_hex()
-    }
-
-    async fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
-        let resp = self
-            .client
-            .get(&format!("{}/api/tx/{}/raw", self.url, txid))
-            .send()
-            .await?;
-
-        if let StatusCode::NOT_FOUND = resp.status() {
-            return Ok(None);
-        }
-
-        Ok(Some(deserialize(&resp.error_for_status()?.bytes().await?)?))
-    }
-
-    async fn _broadcast(&self, transaction: &Transaction) -> Result<(), EsploraError> {
-        self.client
-            .post(&format!("{}/api/tx", self.url))
-            .body(serialize(transaction).to_hex())
-            .send()
-            .await?
-            .error_for_status()?;
-
-        Ok(())
-    }
-
-    async fn _get_height(&self) -> Result<usize, EsploraError> {
-        Ok(self
-            .client
-            .get(&format!("{}/api/blocks/tip/height", self.url))
-            .send()
-            .await?
-            .error_for_status()?
-            .text()
-            .await?
-            .parse()?)
-    }
-
-    async fn _script_get_history(
-        &self,
-        script: &Script,
-    ) -> Result<Vec<ELSGetHistoryRes>, EsploraError> {
-        let mut result = Vec::new();
-        let scripthash = Self::script_to_scripthash(script);
-
-        // Add the unconfirmed transactions first
-        result.extend(
-            self.client
-                .get(&format!(
-                    "{}/api/scripthash/{}/txs/mempool",
-                    self.url, scripthash
-                ))
-                .send()
-                .await?
-                .error_for_status()?
-                .json::<Vec<EsploraGetHistory>>()
-                .await?
-                .into_iter()
-                .map(|x| ELSGetHistoryRes {
-                    tx_hash: x.txid,
-                    height: x.status.block_height.unwrap_or(0) as i32,
-                }),
-        );
-
-        debug!(
-            "Found {} mempool txs for {} - {:?}",
-            result.len(),
-            scripthash,
-            script
-        );
-
-        // Then go through all the pages of confirmed transactions
-        let mut last_txid = String::new();
-        loop {
-            let response = self
-                .client
-                .get(&format!(
-                    "{}/api/scripthash/{}/txs/chain/{}",
-                    self.url, scripthash, last_txid
-                ))
-                .send()
-                .await?
-                .error_for_status()?
-                .json::<Vec<EsploraGetHistory>>()
-                .await?;
-            let len = response.len();
-            if let Some(elem) = response.last() {
-                last_txid = elem.txid.to_hex();
-            }
-
-            debug!("... adding {} confirmed transactions", len);
-
-            result.extend(response.into_iter().map(|x| ELSGetHistoryRes {
-                tx_hash: x.txid,
-                height: x.status.block_height.unwrap_or(0) as i32,
-            }));
-
-            if len < 25 {
-                break;
-            }
-        }
-
-        Ok(result)
-    }
-
-    async fn _script_list_unspent(
-        &self,
-        script: &Script,
-    ) -> Result<Vec<ELSListUnspentRes>, EsploraError> {
-        Ok(self
-            .client
-            .get(&format!(
-                "{}/api/scripthash/{}/utxo",
-                self.url,
-                Self::script_to_scripthash(script)
-            ))
-            .send()
-            .await?
-            .error_for_status()?
-            .json::<Vec<EsploraListUnspent>>()
-            .await?
-            .into_iter()
-            .map(|x| ELSListUnspentRes {
-                tx_hash: x.txid,
-                height: x.status.block_height.unwrap_or(0),
-                tx_pos: x.vout,
-            })
-            .collect())
-    }
-}
-
-#[async_trait(?Send)]
-impl ElectrumLikeSync for UrlClient {
-    async fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script>>(
-        &mut self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ELSGetHistoryRes>>, Error> {
-        Ok(stream::iter(scripts)
-            .then(|script| self._script_get_history(&script))
-            .try_collect()
-            .await?)
-    }
-
-    async fn els_batch_script_list_unspent<'s, I: IntoIterator<Item = &'s Script>>(
-        &mut self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ELSListUnspentRes>>, Error> {
-        Ok(stream::iter(scripts)
-            .then(|script| self._script_list_unspent(&script))
-            .try_collect()
-            .await?)
-    }
-
-    async fn els_transaction_get(&mut self, txid: &Txid) -> Result<Transaction, Error> {
-        Ok(self
-            ._get_tx(txid)
-            .await?
-            .ok_or_else(|| EsploraError::TransactionNotFound(*txid))?)
-    }
-}
-
-#[derive(Deserialize)]
-struct EsploraGetHistoryStatus {
-    block_height: Option<usize>,
-}
-
-#[derive(Deserialize)]
-struct EsploraGetHistory {
-    txid: Txid,
-    status: EsploraGetHistoryStatus,
-}
-
-#[derive(Deserialize)]
-struct EsploraListUnspent {
-    txid: Txid,
-    vout: usize,
-    status: EsploraGetHistoryStatus,
-}
-
-#[derive(Debug)]
-pub enum EsploraError {
-    Reqwest(reqwest::Error),
-    Parsing(std::num::ParseIntError),
-    BitcoinEncoding(bitcoin::consensus::encode::Error),
-
-    TransactionNotFound(Txid),
-}
-
-impl From<reqwest::Error> for EsploraError {
-    fn from(other: reqwest::Error) -> Self {
-        EsploraError::Reqwest(other)
-    }
-}
-
-impl From<std::num::ParseIntError> for EsploraError {
-    fn from(other: std::num::ParseIntError) -> Self {
-        EsploraError::Parsing(other)
-    }
-}
-
-impl From<bitcoin::consensus::encode::Error> for EsploraError {
-    fn from(other: bitcoin::consensus::encode::Error) -> Self {
-        EsploraError::BitcoinEncoding(other)
-    }
-}

src/blockchain/esplora/async.rs

@@ -0,0 +1,252 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Esplora by way of `reqwest` HTTP client.
+
+use std::collections::{HashMap, HashSet};
+use std::ops::{Deref, DerefMut};
+
+use bitcoin::{Transaction, Txid};
+
+#[allow(unused_imports)]
+use log::{debug, error, info, trace};
+
+use esplora_client::{convert_fee_rate, AsyncClient, Builder, Tx};
+use futures::stream::{FuturesOrdered, TryStreamExt};
+
+use crate::blockchain::*;
+use crate::database::BatchDatabase;
+use crate::error::Error;
+use crate::FeeRate;
+
+/// Structure that implements the logic to sync with Esplora
+///
+/// ## Example
+/// See the [`blockchain::esplora`](crate::blockchain::esplora) module for a usage example.
+#[derive(Debug)]
+pub struct EsploraBlockchain {
+    url_client: AsyncClient,
+    stop_gap: usize,
+    concurrency: u8,
+}
+
+impl std::convert::From<AsyncClient> for EsploraBlockchain {
+    fn from(url_client: AsyncClient) -> Self {
+        EsploraBlockchain {
+            url_client,
+            stop_gap: 20,
+            concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
+        }
+    }
+}
+
+impl EsploraBlockchain {
+    /// Create a new instance of the client from a base URL and `stop_gap`.
+    pub fn new(base_url: &str, stop_gap: usize) -> Self {
+        let url_client = Builder::new(base_url)
+            .build_async()
+            .expect("Should never fail with no proxy and timeout");
+
+        Self::from_client(url_client, stop_gap)
+    }
+
+    /// Build a new instance given a client
+    pub fn from_client(url_client: AsyncClient, stop_gap: usize) -> Self {
+        EsploraBlockchain {
+            url_client,
+            stop_gap,
+            concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
+        }
+    }
+
+    /// Set the concurrency to use when doing batch queries against the Esplora instance.
+    pub fn with_concurrency(mut self, concurrency: u8) -> Self {
+        self.concurrency = concurrency;
+        self
+    }
+}
+
+#[maybe_async]
+impl Blockchain for EsploraBlockchain {
+    fn get_capabilities(&self) -> HashSet<Capability> {
+        vec![
+            Capability::FullHistory,
+            Capability::GetAnyTx,
+            Capability::AccurateFees,
+        ]
+        .into_iter()
+        .collect()
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        Ok(await_or_block!(self.url_client.broadcast(tx))?)
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        let estimates = await_or_block!(self.url_client.get_fee_estimates())?;
+        Ok(FeeRate::from_sat_per_vb(convert_fee_rate(
+            target, estimates,
+        )?))
+    }
+}
+
+impl Deref for EsploraBlockchain {
+    type Target = AsyncClient;
+
+    fn deref(&self) -> &Self::Target {
+        &self.url_client
+    }
+}
+
+impl StatelessBlockchain for EsploraBlockchain {}
+
+#[maybe_async]
+impl GetHeight for EsploraBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(await_or_block!(self.url_client.get_height())?)
+    }
+}
+
+#[maybe_async]
+impl GetTx for EsploraBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(await_or_block!(self.url_client.get_tx(txid))?)
+    }
+}
+
+#[maybe_async]
+impl GetBlockHash for EsploraBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        Ok(await_or_block!(self
+            .url_client
+            .get_block_hash(height as u32))?)
+    }
+}
+
+#[maybe_async]
+impl WalletSync for EsploraBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &RefCell<D>,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        use crate::blockchain::script_sync::Request;
+        let mut database = database.borrow_mut();
+        let database = database.deref_mut();
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut tx_index: HashMap<Txid, Tx> = HashMap::new();
+
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let futures: FuturesOrdered<_> = script_req
+                        .request()
+                        .take(self.concurrency as usize)
+                        .map(|script| async move {
+                            let mut related_txs: Vec<Tx> =
+                                self.url_client.scripthash_txs(script, None).await?;
+
+                            let n_confirmed =
+                                related_txs.iter().filter(|tx| tx.status.confirmed).count();
+                            // esplora pages on 25 confirmed transactions. If there's 25 or more we
+                            // keep requesting to see if there's more.
+                            if n_confirmed >= 25 {
+                                loop {
+                                    let new_related_txs: Vec<Tx> = self
+                                        .url_client
+                                        .scripthash_txs(
+                                            script,
+                                            Some(related_txs.last().unwrap().txid),
+                                        )
+                                        .await?;
+                                    let n = new_related_txs.len();
+                                    related_txs.extend(new_related_txs);
+                                    // we've reached the end
+                                    if n < 25 {
+                                        break;
+                                    }
+                                }
+                            }
+                            Result::<_, Error>::Ok(related_txs)
+                        })
+                        .collect();
+                    let txs_per_script: Vec<Vec<Tx>> = await_or_block!(futures.try_collect())?;
+                    let mut satisfaction = vec![];
+
+                    for txs in txs_per_script {
+                        satisfaction.push(
+                            txs.iter()
+                                .map(|tx| (tx.txid, tx.status.block_height))
+                                .collect(),
+                        );
+                        for tx in txs {
+                            tx_index.insert(tx.txid, tx);
+                        }
+                    }
+
+                    script_req.satisfy(satisfaction)?
+                }
+                Request::Conftime(conftime_req) => {
+                    let conftimes = conftime_req
+                        .request()
+                        .map(|txid| {
+                            tx_index
+                                .get(txid)
+                                .expect("must be in index")
+                                .confirmation_time()
+                                .map(Into::into)
+                        })
+                        .collect();
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let full_txs = tx_req
+                        .request()
+                        .map(|txid| {
+                            let tx = tx_index.get(txid).expect("must be in index");
+                            Ok((tx.previous_outputs(), tx.to_tx()))
+                        })
+                        .collect::<Result<_, Error>>()?;
+                    tx_req.satisfy(full_txs)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+        Ok(())
+    }
+}
+
+impl ConfigurableBlockchain for EsploraBlockchain {
+    type Config = super::EsploraBlockchainConfig;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        let mut builder = Builder::new(config.base_url.as_str());
+
+        if let Some(timeout) = config.timeout {
+            builder = builder.timeout(timeout);
+        }
+
+        if let Some(proxy) = &config.proxy {
+            builder = builder.proxy(proxy);
+        }
+
+        let mut blockchain =
+            EsploraBlockchain::from_client(builder.build_async()?, config.stop_gap);
+
+        if let Some(concurrency) = config.concurrency {
+            blockchain = blockchain.with_concurrency(concurrency);
+        }
+
+        Ok(blockchain)
+    }
+}

src/blockchain/esplora/blocking.rs

@@ -0,0 +1,241 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Esplora by way of `ureq` HTTP client.
+
+use std::collections::{HashMap, HashSet};
+use std::ops::DerefMut;
+
+#[allow(unused_imports)]
+use log::{debug, error, info, trace};
+
+use bitcoin::{Transaction, Txid};
+
+use esplora_client::{convert_fee_rate, BlockingClient, Builder, Tx};
+
+use crate::blockchain::*;
+use crate::database::BatchDatabase;
+use crate::error::Error;
+use crate::FeeRate;
+
+/// Structure that implements the logic to sync with Esplora
+///
+/// ## Example
+/// See the [`blockchain::esplora`](crate::blockchain::esplora) module for a usage example.
+#[derive(Debug)]
+pub struct EsploraBlockchain {
+    url_client: BlockingClient,
+    stop_gap: usize,
+    concurrency: u8,
+}
+
+impl EsploraBlockchain {
+    /// Create a new instance of the client from a base URL and the `stop_gap`.
+    pub fn new(base_url: &str, stop_gap: usize) -> Self {
+        let url_client = Builder::new(base_url)
+            .build_blocking()
+            .expect("Should never fail with no proxy and timeout");
+
+        Self::from_client(url_client, stop_gap)
+    }
+
+    /// Build a new instance given a client
+    pub fn from_client(url_client: BlockingClient, stop_gap: usize) -> Self {
+        EsploraBlockchain {
+            url_client,
+            concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
+            stop_gap,
+        }
+    }
+
+    /// Set the number of parallel requests the client can make.
+    pub fn with_concurrency(mut self, concurrency: u8) -> Self {
+        self.concurrency = concurrency;
+        self
+    }
+}
+
+impl Blockchain for EsploraBlockchain {
+    fn get_capabilities(&self) -> HashSet<Capability> {
+        vec![
+            Capability::FullHistory,
+            Capability::GetAnyTx,
+            Capability::AccurateFees,
+        ]
+        .into_iter()
+        .collect()
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        self.url_client.broadcast(tx)?;
+        Ok(())
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        let estimates = self.url_client.get_fee_estimates()?;
+        Ok(FeeRate::from_sat_per_vb(convert_fee_rate(
+            target, estimates,
+        )?))
+    }
+}
+
+impl Deref for EsploraBlockchain {
+    type Target = BlockingClient;
+
+    fn deref(&self) -> &Self::Target {
+        &self.url_client
+    }
+}
+
+impl StatelessBlockchain for EsploraBlockchain {}
+
+impl GetHeight for EsploraBlockchain {
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.url_client.get_height()?)
+    }
+}
+
+impl GetTx for EsploraBlockchain {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.url_client.get_tx(txid)?)
+    }
+}
+
+impl GetBlockHash for EsploraBlockchain {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        Ok(self.url_client.get_block_hash(height as u32)?)
+    }
+}
+
+impl WalletSync for EsploraBlockchain {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &RefCell<D>,
+        _progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        use crate::blockchain::script_sync::Request;
+        let mut database = database.borrow_mut();
+        let database = database.deref_mut();
+        let mut request = script_sync::start(database, self.stop_gap)?;
+        let mut tx_index: HashMap<Txid, Tx> = HashMap::new();
+        let batch_update = loop {
+            request = match request {
+                Request::Script(script_req) => {
+                    let scripts = script_req
+                        .request()
+                        .take(self.concurrency as usize)
+                        .map(bitcoin::ScriptBuf::from);
+
+                    let mut handles = vec![];
+                    for script in scripts {
+                        let client = self.url_client.clone();
+                        // make each request in its own thread.
+                        handles.push(std::thread::spawn(move || {
+                            let mut related_txs: Vec<Tx> = client.scripthash_txs(&script, None)?;
+
+                            let n_confirmed =
+                                related_txs.iter().filter(|tx| tx.status.confirmed).count();
+                            // esplora pages on 25 confirmed transactions. If there's 25 or more we
+                            // keep requesting to see if there's more.
+                            if n_confirmed >= 25 {
+                                loop {
+                                    let new_related_txs: Vec<Tx> = client.scripthash_txs(
+                                        &script,
+                                        Some(related_txs.last().unwrap().txid),
+                                    )?;
+                                    let n = new_related_txs.len();
+                                    related_txs.extend(new_related_txs);
+                                    // we've reached the end
+                                    if n < 25 {
+                                        break;
+                                    }
+                                }
+                            }
+                            Result::<_, Error>::Ok(related_txs)
+                        }));
+                    }
+
+                    let txs_per_script: Vec<Vec<Tx>> = handles
+                        .into_iter()
+                        .map(|handle| handle.join().unwrap())
+                        .collect::<Result<_, _>>()?;
+                    let mut satisfaction = vec![];
+
+                    for txs in txs_per_script {
+                        satisfaction.push(
+                            txs.iter()
+                                .map(|tx| (tx.txid, tx.status.block_height))
+                                .collect(),
+                        );
+                        for tx in txs {
+                            tx_index.insert(tx.txid, tx);
+                        }
+                    }
+
+                    script_req.satisfy(satisfaction)?
+                }
+                Request::Conftime(conftime_req) => {
+                    let conftimes = conftime_req
+                        .request()
+                        .map(|txid| {
+                            tx_index
+                                .get(txid)
+                                .expect("must be in index")
+                                .confirmation_time()
+                                .map(Into::into)
+                        })
+                        .collect();
+                    conftime_req.satisfy(conftimes)?
+                }
+                Request::Tx(tx_req) => {
+                    let full_txs = tx_req
+                        .request()
+                        .map(|txid| {
+                            let tx = tx_index.get(txid).expect("must be in index");
+                            Ok((tx.previous_outputs(), tx.to_tx()))
+                        })
+                        .collect::<Result<_, Error>>()?;
+                    tx_req.satisfy(full_txs)?
+                }
+                Request::Finish(batch_update) => break batch_update,
+            }
+        };
+
+        database.commit_batch(batch_update)?;
+
+        Ok(())
+    }
+}
+
+impl ConfigurableBlockchain for EsploraBlockchain {
+    type Config = super::EsploraBlockchainConfig;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        let mut builder = Builder::new(config.base_url.as_str());
+
+        if let Some(timeout) = config.timeout {
+            builder = builder.timeout(timeout);
+        }
+
+        if let Some(proxy) = &config.proxy {
+            builder = builder.proxy(proxy);
+        }
+
+        let mut blockchain =
+            EsploraBlockchain::from_client(builder.build_blocking()?, config.stop_gap);
+
+        if let Some(concurrency) = config.concurrency {
+            blockchain = blockchain.with_concurrency(concurrency);
+        }
+
+        Ok(blockchain)
+    }
+}

src/blockchain/esplora/mod.rs

@@ -0,0 +1,130 @@
+//! Esplora
+//!
+//! This module defines a [`EsploraBlockchain`] struct that can query an Esplora
+//! backend populate the wallet's [database](crate::database::Database) by:
+//!
+//! ## Example
+//!
+//! ```no_run
+//! # use bdk::blockchain::esplora::EsploraBlockchain;
+//! let blockchain = EsploraBlockchain::new("https://blockstream.info/testnet/api", 20);
+//! # Ok::<(), bdk::Error>(())
+//! ```
+//!
+//! Esplora blockchain can use either `ureq` or `reqwest` for the HTTP client
+//! depending on your needs (blocking or async respectively).
+//!
+//! Please note, to configure the Esplora HTTP client correctly use one of:
+//! Blocking:  --features='use-esplora-blocking'
+//! Async:     --features='async-interface,use-esplora-async' --no-default-features
+
+pub use esplora_client::Error as EsploraError;
+
+#[cfg(feature = "use-esplora-async")]
+mod r#async;
+
+#[cfg(feature = "use-esplora-async")]
+pub use self::r#async::*;
+
+#[cfg(feature = "use-esplora-blocking")]
+mod blocking;
+
+#[cfg(feature = "use-esplora-blocking")]
+pub use self::blocking::*;
+
+/// Configuration for an [`EsploraBlockchain`]
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq, Eq)]
+pub struct EsploraBlockchainConfig {
+    /// Base URL of the esplora service
+    ///
+    /// eg. `https://blockstream.info/api/`
+    pub base_url: String,
+    /// Optional URL of the proxy to use to make requests to the Esplora server
+    ///
+    /// The string should be formatted as: `<protocol>://<user>:<password>@host:<port>`.
+    ///
+    /// Note that the format of this value and the supported protocols change slightly between the
+    /// sync version of esplora (using `ureq`) and the async version (using `reqwest`). For more
+    /// details check with the documentation of the two crates. Both of them are compiled with
+    /// the `socks` feature enabled.
+    ///
+    /// The proxy is ignored when targeting `wasm32`.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub proxy: Option<String>,
+    /// Number of parallel requests sent to the esplora service (default: 4)
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub concurrency: Option<u8>,
+    /// Stop searching addresses for transactions after finding an unused gap of this length.
+    pub stop_gap: usize,
+    /// Socket timeout.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub timeout: Option<u64>,
+}
+
+impl EsploraBlockchainConfig {
+    /// create a config with default values given the base url and stop gap
+    pub fn new(base_url: String, stop_gap: usize) -> Self {
+        Self {
+            base_url,
+            proxy: None,
+            timeout: None,
+            stop_gap,
+            concurrency: None,
+        }
+    }
+}
+
+impl From<esplora_client::BlockTime> for crate::BlockTime {
+    fn from(esplora_client::BlockTime { timestamp, height }: esplora_client::BlockTime) -> Self {
+        Self { timestamp, height }
+    }
+}
+
+#[cfg(test)]
+#[cfg(feature = "test-esplora")]
+crate::bdk_blockchain_tests! {
+    fn test_instance(test_client: &TestClient) -> EsploraBlockchain {
+        EsploraBlockchain::new(&format!("http://{}",test_client.electrsd.esplora_url.as_ref().unwrap()), 20)
+    }
+}
+
+const DEFAULT_CONCURRENT_REQUESTS: u8 = 4;
+
+#[cfg(test)]
+mod test {
+    #[test]
+    #[cfg(feature = "test-esplora")]
+    fn test_esplora_with_variable_configs() {
+        use super::*;
+
+        use crate::testutils::{
+            blockchain_tests::TestClient,
+            configurable_blockchain_tests::ConfigurableBlockchainTester,
+        };
+
+        struct EsploraTester;
+
+        impl ConfigurableBlockchainTester<EsploraBlockchain> for EsploraTester {
+            const BLOCKCHAIN_NAME: &'static str = "Esplora";
+
+            fn config_with_stop_gap(
+                &self,
+                test_client: &mut TestClient,
+                stop_gap: usize,
+            ) -> Option<EsploraBlockchainConfig> {
+                Some(EsploraBlockchainConfig {
+                    base_url: format!(
+                        "http://{}",
+                        test_client.electrsd.esplora_url.as_ref().unwrap()
+                    ),
+                    proxy: None,
+                    concurrency: None,
+                    stop_gap: stop_gap,
+                    timeout: None,
+                })
+            }
+        }
+
+        EsploraTester.run();
+    }
+}

src/blockchain/mod.rs

@@ -1,84 +1,296 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Blockchain backends
+//!
+//! This module provides the implementation of a few commonly-used backends like
+//! [Electrum](crate::blockchain::electrum), [Esplora](crate::blockchain::esplora) and
+//! [Compact Filters/Neutrino](crate::blockchain::compact_filters), along with a generalized trait
+//! [`Blockchain`] that can be implemented to build customized backends.
+
+use std::cell::RefCell;
 use std::collections::HashSet;
+use std::ops::Deref;
 use std::sync::mpsc::{channel, Receiver, Sender};
+use std::sync::Arc;
 
-use bitcoin::{Transaction, Txid};
+use bitcoin::{BlockHash, Transaction, Txid};
 
-use crate::database::{BatchDatabase, DatabaseUtils};
+use crate::database::BatchDatabase;
 use crate::error::Error;
+use crate::wallet::{wallet_name_from_descriptor, Wallet};
+use crate::{FeeRate, KeychainKind};
 
-pub mod utils;
+#[cfg(any(
+    feature = "electrum",
+    feature = "esplora",
+    feature = "compact_filters",
+    feature = "rpc"
+))]
+pub mod any;
+mod script_sync;
+
+#[cfg(any(
+    feature = "electrum",
+    feature = "esplora",
+    feature = "compact_filters",
+    feature = "rpc"
+))]
+pub use any::{AnyBlockchain, AnyBlockchainConfig};
 
 #[cfg(feature = "electrum")]
+#[cfg_attr(docsrs, doc(cfg(feature = "electrum")))]
 pub mod electrum;
 #[cfg(feature = "electrum")]
 pub use self::electrum::ElectrumBlockchain;
+#[cfg(feature = "electrum")]
+pub use self::electrum::ElectrumBlockchainConfig;
+
+#[cfg(feature = "rpc")]
+#[cfg_attr(docsrs, doc(cfg(feature = "rpc")))]
+pub mod rpc;
+#[cfg(feature = "rpc")]
+pub use self::rpc::RpcBlockchain;
+#[cfg(feature = "rpc")]
+pub use self::rpc::RpcConfig;
 
 #[cfg(feature = "esplora")]
+#[cfg_attr(docsrs, doc(cfg(feature = "esplora")))]
 pub mod esplora;
 #[cfg(feature = "esplora")]
 pub use self::esplora::EsploraBlockchain;
 
+#[cfg(feature = "compact_filters")]
+#[cfg_attr(docsrs, doc(cfg(feature = "compact_filters")))]
+pub mod compact_filters;
+
+#[cfg(feature = "compact_filters")]
+pub use self::compact_filters::CompactFiltersBlockchain;
+
+/// Capabilities that can be supported by a [`Blockchain`] backend
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
 pub enum Capability {
+    /// Can recover the full history of a wallet and not only the set of currently spendable UTXOs
     FullHistory,
+    /// Can fetch any historical transaction given its txid
     GetAnyTx,
+    /// Can compute accurate fees for the transactions found during sync
+    AccurateFees,
 }
 
-pub trait Blockchain {
-    fn is_online(&self) -> bool;
-
-    fn offline() -> Self;
+/// Trait that defines the actions that must be supported by a blockchain backend
+#[maybe_async]
+pub trait Blockchain: WalletSync + GetHeight + GetTx + GetBlockHash {
+    /// Return the set of [`Capability`] supported by this backend
+    fn get_capabilities(&self) -> HashSet<Capability>;
+    /// Broadcast a transaction
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error>;
+    /// Estimate the fee rate required to confirm a transaction in a given `target` of blocks
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error>;
 }
 
-pub struct OfflineBlockchain;
-impl Blockchain for OfflineBlockchain {
-    fn offline() -> Self {
-        OfflineBlockchain
-    }
-
-    fn is_online(&self) -> bool {
-        false
-    }
+/// Trait for getting the current height of the blockchain.
+#[maybe_async]
+pub trait GetHeight {
+    /// Return the current height
+    fn get_height(&self) -> Result<u32, Error>;
 }
 
-#[async_trait(?Send)]
-pub trait OnlineBlockchain: Blockchain {
-    async fn get_capabilities(&self) -> HashSet<Capability>;
+#[maybe_async]
+/// Trait for getting a transaction by txid
+pub trait GetTx {
+    /// Fetch a transaction given its txid
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
+}
 
-    async fn setup<D: BatchDatabase + DatabaseUtils, P: Progress>(
-        &mut self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
+#[maybe_async]
+/// Trait for getting block hash by block height
+pub trait GetBlockHash {
+    /// fetch block hash given its height
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error>;
+}
+
+/// Trait for blockchains that can sync by updating the database directly.
+#[maybe_async]
+pub trait WalletSync {
+    /// Setup the backend and populate the internal database for the first time
+    ///
+    /// This method is the equivalent of [`Self::wallet_sync`], but it's guaranteed to only be
+    /// called once, at the first [`Wallet::sync`](crate::wallet::Wallet::sync).
+    ///
+    /// The rationale behind the distinction between `sync` and `setup` is that some custom backends
+    /// might need to perform specific actions only the first time they are synced.
+    ///
+    /// For types that do not have that distinction, only this method can be implemented, since
+    /// [`WalletSync::wallet_sync`] defaults to calling this internally if not overridden.
+    /// Populate the internal database with transactions and UTXOs
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &RefCell<D>,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error>;
-    async fn sync<D: BatchDatabase + DatabaseUtils, P: Progress>(
-        &mut self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        progress_update: P,
+
+    /// If not overridden, it defaults to calling [`Self::wallet_setup`] internally.
+    ///
+    /// This method should implement the logic required to iterate over the list of the wallet's
+    /// script_pubkeys using [`Database::iter_script_pubkeys`] and look for relevant transactions
+    /// in the blockchain to populate the database with [`BatchOperations::set_tx`] and
+    /// [`BatchOperations::set_utxo`].
+    ///
+    /// This method should also take care of removing UTXOs that are seen as spent in the
+    /// blockchain, using [`BatchOperations::del_utxo`].
+    ///
+    /// The `progress_update` object can be used to give the caller updates about the progress by using
+    /// [`Progress::update`].
+    ///
+    /// [`Database::iter_script_pubkeys`]: crate::database::Database::iter_script_pubkeys
+    /// [`BatchOperations::set_tx`]: crate::database::BatchOperations::set_tx
+    /// [`BatchOperations::set_utxo`]: crate::database::BatchOperations::set_utxo
+    /// [`BatchOperations::del_utxo`]: crate::database::BatchOperations::del_utxo
+    fn wallet_sync<D: BatchDatabase>(
+        &self,
+        database: &RefCell<D>,
+        progress_update: Box<dyn Progress>,
     ) -> Result<(), Error> {
-        self.setup(stop_gap, database, progress_update).await
+        maybe_await!(self.wallet_setup(database, progress_update))
     }
-
-    async fn get_tx(&mut self, txid: &Txid) -> Result<Option<Transaction>, Error>;
-    async fn broadcast(&mut self, tx: &Transaction) -> Result<(), Error>;
-
-    async fn get_height(&mut self) -> Result<usize, Error>;
 }
 
+/// Trait for [`Blockchain`] types that can be created given a configuration
+pub trait ConfigurableBlockchain: Blockchain + Sized {
+    /// Type that contains the configuration
+    type Config: std::fmt::Debug;
+
+    /// Create a new instance given a configuration
+    fn from_config(config: &Self::Config) -> Result<Self, Error>;
+}
+
+/// Trait for blockchains that don't contain any state
+///
+/// Statless blockchains can be used to sync multiple wallets with different descriptors.
+///
+/// [`BlockchainFactory`] is automatically implemented for `Arc<T>` where `T` is a stateless
+/// blockchain.
+pub trait StatelessBlockchain: Blockchain {}
+
+/// Trait for a factory of blockchains that share the underlying connection or configuration
+#[cfg_attr(
+    not(feature = "async-interface"),
+    doc = r##"
+## Example
+
+This example shows how to sync multiple walles and return the sum of their balances
+
+```no_run
+# use bdk::Error;
+# use bdk::blockchain::*;
+# use bdk::database::*;
+# use bdk::wallet::*;
+# use bdk::*;
+fn sum_of_balances<B: BlockchainFactory>(blockchain_factory: B, wallets: &[Wallet<MemoryDatabase>]) -> Result<Balance, Error> {
+    Ok(wallets
+        .iter()
+        .map(|w| -> Result<_, Error> {
+            blockchain_factory.sync_wallet(&w, None, SyncOptions::default())?;
+            w.get_balance()
+        })
+        .collect::<Result<Vec<_>, _>>()?
+        .into_iter()
+        .sum())
+}
+```
+"##
+)]
+pub trait BlockchainFactory {
+    /// The type returned when building a blockchain from this factory
+    type Inner: Blockchain;
+
+    /// Build a new blockchain for the given descriptor wallet_name
+    ///
+    /// If `override_skip_blocks` is `None`, the returned blockchain will inherit the number of blocks
+    /// from the factory. Since it's not possible to override the value to `None`, set it to
+    /// `Some(0)` to rescan from the genesis.
+    fn build(
+        &self,
+        wallet_name: &str,
+        override_skip_blocks: Option<u32>,
+    ) -> Result<Self::Inner, Error>;
+
+    /// Build a new blockchain for a given wallet
+    ///
+    /// Internally uses [`wallet_name_from_descriptor`] to derive the name, and then calls
+    /// [`BlockchainFactory::build`] to create the blockchain instance.
+    fn build_for_wallet<D: BatchDatabase>(
+        &self,
+        wallet: &Wallet<D>,
+        override_skip_blocks: Option<u32>,
+    ) -> Result<Self::Inner, Error> {
+        let wallet_name = wallet_name_from_descriptor(
+            wallet.public_descriptor(KeychainKind::External)?.unwrap(),
+            wallet.public_descriptor(KeychainKind::Internal)?,
+            wallet.network(),
+            wallet.secp_ctx(),
+        )?;
+        self.build(&wallet_name, override_skip_blocks)
+    }
+
+    /// Use [`BlockchainFactory::build_for_wallet`] to get a blockchain, then sync the wallet
+    ///
+    /// This can be used when a new blockchain would only be used to sync a wallet and then
+    /// immediately dropped. Keep in mind that specific blockchain factories may perform slow
+    /// operations to build a blockchain for a given wallet, so if a wallet needs to be synced
+    /// often it's recommended to use [`BlockchainFactory::build_for_wallet`] to reuse the same
+    /// blockchain multiple times.
+    #[cfg(not(feature = "async-interface"))]
+    #[cfg_attr(docsrs, doc(cfg(not(feature = "async-interface"))))]
+    fn sync_wallet<D: BatchDatabase>(
+        &self,
+        wallet: &Wallet<D>,
+        override_skip_blocks: Option<u32>,
+        sync_options: crate::wallet::SyncOptions,
+    ) -> Result<(), Error> {
+        let blockchain = self.build_for_wallet(wallet, override_skip_blocks)?;
+        wallet.sync(&blockchain, sync_options)
+    }
+}
+
+impl<T: StatelessBlockchain> BlockchainFactory for Arc<T> {
+    type Inner = Self;
+
+    fn build(&self, _wallet_name: &str, _override_skip_blocks: Option<u32>) -> Result<Self, Error> {
+        Ok(Arc::clone(self))
+    }
+}
+
+/// Data sent with a progress update over a [`channel`]
 pub type ProgressData = (f32, Option<String>);
 
-pub trait Progress {
+/// Trait for types that can receive and process progress updates during [`WalletSync::wallet_sync`] and
+/// [`WalletSync::wallet_setup`]
+pub trait Progress: Send + 'static + core::fmt::Debug {
+    /// Send a new progress update
+    ///
+    /// The `progress` value should be in the range 0.0 - 100.0, and the `message` value is an
+    /// optional text message that can be displayed to the user.
     fn update(&self, progress: f32, message: Option<String>) -> Result<(), Error>;
 }
 
+/// Shortcut to create a [`channel`] (pair of [`Sender`] and [`Receiver`]) that can transport [`ProgressData`]
 pub fn progress() -> (Sender<ProgressData>, Receiver<ProgressData>) {
     channel()
 }
 
 impl Progress for Sender<ProgressData> {
     fn update(&self, progress: f32, message: Option<String>) -> Result<(), Error> {
-        if progress < 0.0 || progress > 100.0 {
+        if !(0.0..=100.0).contains(&progress) {
             return Err(Error::InvalidProgressValue(progress));
         }
 
@@ -87,8 +299,11 @@ impl Progress for Sender<ProgressData> {
     }
 }
 
+/// Type that implements [`Progress`] and drops every update received
+#[derive(Clone, Copy, Default, Debug)]
 pub struct NoopProgress;
 
+/// Create a new instance of [`NoopProgress`]
 pub fn noop_progress() -> NoopProgress {
     NoopProgress
 }
@@ -98,3 +313,79 @@ impl Progress for NoopProgress {
         Ok(())
     }
 }
+
+/// Type that implements [`Progress`] and logs at level `INFO` every update received
+#[derive(Clone, Copy, Default, Debug)]
+pub struct LogProgress;
+
+/// Create a new instance of [`LogProgress`]
+pub fn log_progress() -> LogProgress {
+    LogProgress
+}
+
+impl Progress for LogProgress {
+    fn update(&self, progress: f32, message: Option<String>) -> Result<(), Error> {
+        log::info!(
+            "Sync {:.3}%: `{}`",
+            progress,
+            message.unwrap_or_else(|| "".into())
+        );
+
+        Ok(())
+    }
+}
+
+#[maybe_async]
+impl<T: Blockchain> Blockchain for Arc<T> {
+    fn get_capabilities(&self) -> HashSet<Capability> {
+        maybe_await!(self.deref().get_capabilities())
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        maybe_await!(self.deref().broadcast(tx))
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        maybe_await!(self.deref().estimate_fee(target))
+    }
+}
+
+#[maybe_async]
+impl<T: GetTx> GetTx for Arc<T> {
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        maybe_await!(self.deref().get_tx(txid))
+    }
+}
+
+#[maybe_async]
+impl<T: GetHeight> GetHeight for Arc<T> {
+    fn get_height(&self) -> Result<u32, Error> {
+        maybe_await!(self.deref().get_height())
+    }
+}
+
+#[maybe_async]
+impl<T: GetBlockHash> GetBlockHash for Arc<T> {
+    fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
+        maybe_await!(self.deref().get_block_hash(height))
+    }
+}
+
+#[maybe_async]
+impl<T: WalletSync> WalletSync for Arc<T> {
+    fn wallet_setup<D: BatchDatabase>(
+        &self,
+        database: &RefCell<D>,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(self.deref().wallet_setup(database, progress_update))
+    }
+
+    fn wallet_sync<D: BatchDatabase>(
+        &self,
+        database: &RefCell<D>,
+        progress_update: Box<dyn Progress>,
+    ) -> Result<(), Error> {
+        maybe_await!(self.deref().wallet_sync(database, progress_update))
+    }
+}

src/blockchain/script_sync.rs

@@ -0,0 +1,474 @@
+/*!
+This models a how a sync happens where you have a server that you send your script pubkeys to and it
+returns associated transactions i.e. electrum.
+*/
+#![allow(dead_code)]
+use crate::{
+    database::{BatchDatabase, BatchOperations, DatabaseUtils},
+    error::MissingCachedScripts,
+    wallet::time::Instant,
+    BlockTime, Error, KeychainKind, LocalUtxo, TransactionDetails,
+};
+use bitcoin::{hashes::Hash, OutPoint, Script, ScriptBuf, Transaction, TxOut, Txid};
+use log::*;
+use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque};
+
+/// A request for on-chain information
+pub enum Request<'a, D: BatchDatabase> {
+    /// A request for transactions related to script pubkeys.
+    Script(ScriptReq<'a, D>),
+    /// A request for confirmation times for some transactions.
+    Conftime(ConftimeReq<'a, D>),
+    /// A request for full transaction details of some transactions.
+    Tx(TxReq<'a, D>),
+    /// Requests are finished here's a batch database update to reflect data gathered.
+    Finish(D::Batch),
+}
+
+/// starts a sync
+pub fn start<D: BatchDatabase>(db: &D, stop_gap: usize) -> Result<Request<'_, D>, Error> {
+    use rand::seq::SliceRandom;
+    let mut keychains = vec![KeychainKind::Internal, KeychainKind::External];
+    // shuffling improve privacy, the server doesn't know my first request is from my internal or external addresses
+    keychains.shuffle(&mut rand::thread_rng());
+    let keychain = keychains.pop().unwrap();
+    let scripts_needed = db
+        .iter_script_pubkeys(Some(keychain))?
+        .into_iter()
+        .collect::<VecDeque<_>>();
+    let state = State::new(db);
+
+    Ok(Request::Script(ScriptReq {
+        state,
+        initial_scripts_needed: scripts_needed.len(),
+        scripts_needed,
+        script_index: 0,
+        stop_gap,
+        keychain,
+        next_keychains: keychains,
+    }))
+}
+
+pub struct ScriptReq<'a, D: BatchDatabase> {
+    state: State<'a, D>,
+    script_index: usize,
+    initial_scripts_needed: usize, // if this is 1, we assume the descriptor is not derivable
+    scripts_needed: VecDeque<ScriptBuf>,
+    stop_gap: usize,
+    keychain: KeychainKind,
+    next_keychains: Vec<KeychainKind>,
+}
+
+/// The sync starts by returning script pubkeys we are interested in.
+impl<'a, D: BatchDatabase> ScriptReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Script> + Clone {
+        self.scripts_needed.iter().map(|s| s.as_script())
+    }
+
+    pub fn satisfy(
+        mut self,
+        // we want to know the txids assoiciated with the script and their height
+        txids: Vec<Vec<(Txid, Option<u32>)>>,
+    ) -> Result<Request<'a, D>, Error> {
+        for (txid_list, script) in txids.iter().zip(self.scripts_needed.iter()) {
+            debug!(
+                "found {} transactions for script pubkey {}",
+                txid_list.len(),
+                script
+            );
+            if !txid_list.is_empty() {
+                // the address is active
+                self.state
+                    .last_active_index
+                    .insert(self.keychain, self.script_index);
+            }
+
+            for (txid, height) in txid_list {
+                // have we seen this txid already?
+                match self.state.db.get_tx(txid, true)? {
+                    Some(mut details) => {
+                        let old_height = details.confirmation_time.as_ref().map(|x| x.height);
+                        match (old_height, height) {
+                            (None, Some(_)) => {
+                                // It looks like the tx has confirmed since we last saw it -- we
+                                // need to know the confirmation time.
+                                self.state.tx_missing_conftime.insert(*txid, details);
+                            }
+                            (Some(old_height), Some(new_height)) if old_height != *new_height => {
+                                // The height of the tx has changed !? -- It's a reorg get the new confirmation time.
+                                self.state.tx_missing_conftime.insert(*txid, details);
+                            }
+                            (Some(_), None) => {
+                                // A re-org where the tx is not in the chain anymore.
+                                details.confirmation_time = None;
+                                self.state.finished_txs.push(details);
+                            }
+                            _ => self.state.finished_txs.push(details),
+                        }
+                    }
+                    None => {
+                        // we've never seen it let's get the whole thing
+                        self.state.tx_needed.insert(*txid);
+                    }
+                };
+            }
+
+            self.script_index += 1;
+        }
+
+        self.scripts_needed.drain(..txids.len());
+
+        // last active index: 0 => No last active
+        let last = self
+            .state
+            .last_active_index
+            .get(&self.keychain)
+            .map(|&l| l + 1)
+            .unwrap_or(0);
+        // remaining scripts left to check
+        let remaining = self.scripts_needed.len();
+        // difference between current index and last active index
+        let current_gap = self.script_index - last;
+
+        // this is a hack to check whether the scripts are coming from a derivable descriptor
+        // we assume for non-derivable descriptors, the initial script count is always 1
+        let is_derivable = self.initial_scripts_needed > 1;
+
+        debug!(
+            "sync: last={}, remaining={}, diff={}, stop_gap={}",
+            last, remaining, current_gap, self.stop_gap
+        );
+
+        if is_derivable {
+            if remaining > 0 {
+                // we still have scriptPubKeys to do requests for
+                return Ok(Request::Script(self));
+            }
+
+            if last > 0 && current_gap < self.stop_gap {
+                // current gap is not large enough to stop, but we are unable to keep checking since
+                // we have exhausted cached scriptPubKeys, so return error
+                let err = MissingCachedScripts {
+                    last_count: self.script_index,
+                    missing_count: self.stop_gap - current_gap,
+                };
+                return Err(Error::MissingCachedScripts(err));
+            }
+
+            // we have exhausted cached scriptPubKeys and found no txs, continue
+        }
+
+        debug!(
+            "finished scanning for txs of keychain {:?} at index {:?}",
+            self.keychain, last
+        );
+
+        if let Some(keychain) = self.next_keychains.pop() {
+            // we still have another keychain to request txs with
+            let scripts_needed = self
+                .state
+                .db
+                .iter_script_pubkeys(Some(keychain))?
+                .into_iter()
+                .collect::<VecDeque<_>>();
+
+            self.keychain = keychain;
+            self.script_index = 0;
+            self.initial_scripts_needed = scripts_needed.len();
+            self.scripts_needed = scripts_needed;
+            return Ok(Request::Script(self));
+        }
+
+        // We have finished requesting txids, let's get the actual txs.
+        Ok(Request::Tx(TxReq { state: self.state }))
+    }
+}
+
+/// Then we get full transactions
+pub struct TxReq<'a, D> {
+    state: State<'a, D>,
+}
+
+impl<'a, D: BatchDatabase> TxReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Txid> + Clone {
+        self.state.tx_needed.iter()
+    }
+
+    pub fn satisfy(
+        mut self,
+        tx_details: Vec<(Vec<Option<TxOut>>, Transaction)>,
+    ) -> Result<Request<'a, D>, Error> {
+        let tx_details: Vec<TransactionDetails> = tx_details
+            .into_iter()
+            .zip(self.state.tx_needed.iter())
+            .map(|((vout, tx), txid)| {
+                debug!("found tx_details for {}", txid);
+                assert_eq!(tx.txid(), *txid);
+                let mut sent: u64 = 0;
+                let mut received: u64 = 0;
+                let mut inputs_sum: u64 = 0;
+                let mut outputs_sum: u64 = 0;
+
+                for (txout, (_input_index, input)) in
+                    vout.into_iter().zip(tx.input.iter().enumerate())
+                {
+                    let txout = match txout {
+                        Some(txout) => txout,
+                        None => {
+                            // skip coinbase inputs
+                            debug_assert!(
+                                input.previous_output.is_null(),
+                                "prevout should only be missing for coinbase"
+                            );
+                            continue;
+                        }
+                    };
+                    // Verify this input if requested via feature flag
+                    #[cfg(feature = "verify")]
+                    {
+                        use crate::wallet::verify::VerifyError;
+                        let serialized_tx = bitcoin::consensus::serialize(&tx);
+                        bitcoinconsensus::verify(
+                            txout.script_pubkey.to_bytes().as_ref(),
+                            txout.value,
+                            &serialized_tx,
+                            _input_index,
+                        )
+                        .map_err(VerifyError::from)?;
+                    }
+                    inputs_sum += txout.value;
+                    if self.state.db.is_mine(&txout.script_pubkey)? {
+                        sent += txout.value;
+                    }
+                }
+
+                for out in &tx.output {
+                    outputs_sum += out.value;
+                    if self.state.db.is_mine(&out.script_pubkey)? {
+                        received += out.value;
+                    }
+                }
+                // we need to saturating sub since we want coinbase txs to map to 0 fee and
+                // this subtraction will be negative for coinbase txs.
+                let fee = inputs_sum.saturating_sub(outputs_sum);
+                Result::<_, Error>::Ok(TransactionDetails {
+                    txid: *txid,
+                    transaction: Some(tx),
+                    received,
+                    sent,
+                    // we're going to fill this in later
+                    confirmation_time: None,
+                    fee: Some(fee),
+                })
+            })
+            .collect::<Result<Vec<_>, _>>()?;
+
+        for tx_detail in tx_details {
+            self.state.tx_needed.remove(&tx_detail.txid);
+            self.state
+                .tx_missing_conftime
+                .insert(tx_detail.txid, tx_detail);
+        }
+
+        if !self.state.tx_needed.is_empty() {
+            Ok(Request::Tx(self))
+        } else {
+            Ok(Request::Conftime(ConftimeReq { state: self.state }))
+        }
+    }
+}
+
+/// Final step is to get confirmation times
+pub struct ConftimeReq<'a, D> {
+    state: State<'a, D>,
+}
+
+impl<'a, D: BatchDatabase> ConftimeReq<'a, D> {
+    pub fn request(&self) -> impl Iterator<Item = &Txid> + Clone {
+        self.state.tx_missing_conftime.keys()
+    }
+
+    pub fn satisfy(
+        mut self,
+        confirmation_times: Vec<Option<BlockTime>>,
+    ) -> Result<Request<'a, D>, Error> {
+        let conftime_needed = self
+            .request()
+            .cloned()
+            .take(confirmation_times.len())
+            .collect::<Vec<_>>();
+        for (confirmation_time, txid) in confirmation_times.into_iter().zip(conftime_needed.iter())
+        {
+            debug!("confirmation time for {} was {:?}", txid, confirmation_time);
+            if let Some(mut tx_details) = self.state.tx_missing_conftime.remove(txid) {
+                tx_details.confirmation_time = confirmation_time;
+                self.state.finished_txs.push(tx_details);
+            }
+        }
+
+        if self.state.tx_missing_conftime.is_empty() {
+            Ok(Request::Finish(self.state.into_db_update()?))
+        } else {
+            Ok(Request::Conftime(self))
+        }
+    }
+}
+
+struct State<'a, D> {
+    db: &'a D,
+    last_active_index: HashMap<KeychainKind, usize>,
+    /// Transactions where we need to get the full details
+    tx_needed: BTreeSet<Txid>,
+    /// Transacitions that we know everything about
+    finished_txs: Vec<TransactionDetails>,
+    /// Transactions that discovered conftimes should be inserted into
+    tx_missing_conftime: BTreeMap<Txid, TransactionDetails>,
+    /// The start of the sync
+    start_time: Instant,
+    /// Missing number of scripts to cache per keychain
+    missing_script_counts: HashMap<KeychainKind, usize>,
+}
+
+impl<'a, D: BatchDatabase> State<'a, D> {
+    fn new(db: &'a D) -> Self {
+        State {
+            db,
+            last_active_index: HashMap::default(),
+            finished_txs: vec![],
+            tx_needed: BTreeSet::default(),
+            tx_missing_conftime: BTreeMap::default(),
+            start_time: Instant::new(),
+            missing_script_counts: HashMap::default(),
+        }
+    }
+    fn into_db_update(self) -> Result<D::Batch, Error> {
+        debug_assert!(self.tx_needed.is_empty() && self.tx_missing_conftime.is_empty());
+        let existing_txs = self.db.iter_txs(false)?;
+        let existing_txids: HashSet<Txid> = existing_txs.iter().map(|tx| tx.txid).collect();
+        let finished_txs = make_txs_consistent(&self.finished_txs);
+        let observed_txids: HashSet<Txid> = finished_txs.iter().map(|tx| tx.txid).collect();
+        let txids_to_delete = existing_txids.difference(&observed_txids);
+
+        // Ensure `last_active_index` does not decrement database's current state.
+        let index_updates = self
+            .last_active_index
+            .iter()
+            .map(|(keychain, sync_index)| {
+                let sync_index = *sync_index as u32;
+                let index_res = match self.db.get_last_index(*keychain) {
+                    Ok(Some(db_index)) => Ok(std::cmp::max(db_index, sync_index)),
+                    Ok(None) => Ok(sync_index),
+                    Err(err) => Err(err),
+                };
+                index_res.map(|index| (*keychain, index))
+            })
+            .collect::<Result<Vec<(KeychainKind, u32)>, _>>()?;
+
+        let mut batch = self.db.begin_batch();
+
+        // Delete old txs that no longer exist
+        for txid in txids_to_delete {
+            if let Some(raw_tx) = self.db.get_raw_tx(txid)? {
+                for i in 0..raw_tx.output.len() {
+                    // Also delete any utxos from the txs that no longer exist.
+                    let _ = batch.del_utxo(&OutPoint {
+                        txid: *txid,
+                        vout: i as u32,
+                    })?;
+                }
+            } else {
+                unreachable!("we should always have the raw tx");
+            }
+            batch.del_tx(txid, true)?;
+        }
+
+        let mut spent_utxos = HashSet::new();
+
+        // track all the spent utxos
+        for finished_tx in &finished_txs {
+            let tx = finished_tx
+                .transaction
+                .as_ref()
+                .expect("transaction will always be present here");
+            for input in &tx.input {
+                spent_utxos.insert(&input.previous_output);
+            }
+        }
+
+        // set every utxo we observed, unless it's already spent
+        // we don't do this in the loop above as we want to know all the spent outputs before
+        // adding the non-spent to the batch in case there are new tranasactions
+        // that spend form each other.
+        for finished_tx in &finished_txs {
+            let tx = finished_tx
+                .transaction
+                .as_ref()
+                .expect("transaction will always be present here");
+            for (i, output) in tx.output.iter().enumerate() {
+                if let Some((keychain, _)) =
+                    self.db.get_path_from_script_pubkey(&output.script_pubkey)?
+                {
+                    // add utxos we own from the new transactions we've seen.
+                    let outpoint = OutPoint {
+                        txid: finished_tx.txid,
+                        vout: i as u32,
+                    };
+
+                    batch.set_utxo(&LocalUtxo {
+                        outpoint,
+                        txout: output.clone(),
+                        keychain,
+                        // Is this UTXO in the spent_utxos set?
+                        is_spent: spent_utxos.get(&outpoint).is_some(),
+                    })?;
+                }
+            }
+
+            batch.set_tx(finished_tx)?;
+        }
+
+        // apply index updates
+        for (keychain, new_index) in index_updates {
+            debug!("updating index ({}, {})", keychain.as_byte(), new_index);
+            batch.set_last_index(keychain, new_index)?;
+        }
+
+        info!(
+            "finished setup, elapsed {:?}ms",
+            self.start_time.elapsed().as_millis()
+        );
+        Ok(batch)
+    }
+}
+
+/// Remove conflicting transactions -- tie breaking them by fee.
+fn make_txs_consistent(txs: &[TransactionDetails]) -> Vec<&TransactionDetails> {
+    let mut utxo_index: HashMap<OutPoint, &TransactionDetails> = HashMap::default();
+    let mut coinbase_txs = vec![];
+    for tx in txs {
+        for input in &tx.transaction.as_ref().unwrap().input {
+            if input.previous_output.txid == Txid::all_zeros() {
+                coinbase_txs.push(tx);
+                break;
+            }
+
+            utxo_index
+                .entry(input.previous_output)
+                .and_modify(|existing| match (tx.fee, existing.fee) {
+                    (Some(fee), Some(existing_fee)) if fee > existing_fee => *existing = tx,
+                    (Some(_), None) => *existing = tx,
+                    _ => { /* leave it the same */ }
+                })
+                .or_insert(tx);
+        }
+    }
+
+    utxo_index
+        .into_iter()
+        .map(|(_, tx)| (tx.txid, tx))
+        .collect::<HashMap<_, _>>()
+        .into_iter()
+        .map(|(_, tx)| tx)
+        .chain(coinbase_txs)
+        .collect()
+}

src/blockchain/utils.rs

@@ -1,306 +0,0 @@
-use std::cmp;
-use std::collections::{HashSet, VecDeque};
-use std::convert::TryFrom;
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
-
-use bitcoin::{Address, Network, OutPoint, Script, Transaction, Txid};
-
-use super::*;
-use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
-use crate::error::Error;
-use crate::types::{ScriptType, TransactionDetails, UTXO};
-use crate::wallet::utils::ChunksIterator;
-
-#[derive(Debug)]
-pub struct ELSGetHistoryRes {
-    pub height: i32,
-    pub tx_hash: Txid,
-}
-
-#[derive(Debug)]
-pub struct ELSListUnspentRes {
-    pub height: usize,
-    pub tx_hash: Txid,
-    pub tx_pos: usize,
-}
-
-/// Implements the synchronization logic for an Electrum-like client.
-#[async_trait(?Send)]
-pub trait ElectrumLikeSync {
-    async fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script>>(
-        &mut self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ELSGetHistoryRes>>, Error>;
-
-    async fn els_batch_script_list_unspent<'s, I: IntoIterator<Item = &'s Script>>(
-        &mut self,
-        scripts: I,
-    ) -> Result<Vec<Vec<ELSListUnspentRes>>, Error>;
-
-    async fn els_transaction_get(&mut self, txid: &Txid) -> Result<Transaction, Error>;
-
-    // Provided methods down here...
-
-    async fn electrum_like_setup<D: BatchDatabase + DatabaseUtils, P: Progress>(
-        &mut self,
-        stop_gap: Option<usize>,
-        database: &mut D,
-        _progress_update: P,
-    ) -> Result<(), Error> {
-        // TODO: progress
-
-        let stop_gap = stop_gap.unwrap_or(20);
-        let batch_query_size = 20;
-
-        // check unconfirmed tx, delete so they are retrieved later
-        let mut del_batch = database.begin_batch();
-        for tx in database.iter_txs(false)? {
-            if tx.height.is_none() {
-                del_batch.del_tx(&tx.txid, false)?;
-            }
-        }
-        database.commit_batch(del_batch)?;
-
-        // maximum derivation index for a change address that we've seen during sync
-        let mut change_max_deriv = 0;
-
-        let mut already_checked: HashSet<Script> = HashSet::new();
-        let mut to_check_later = VecDeque::with_capacity(batch_query_size);
-
-        // insert the first chunk
-        let mut iter_scriptpubkeys = database
-            .iter_script_pubkeys(Some(ScriptType::External))?
-            .into_iter();
-        let chunk: Vec<Script> = iter_scriptpubkeys.by_ref().take(batch_query_size).collect();
-        for item in chunk.into_iter().rev() {
-            to_check_later.push_front(item);
-        }
-
-        let mut iterating_external = true;
-        let mut index = 0;
-        let mut last_found = 0;
-        while !to_check_later.is_empty() {
-            trace!("to_check_later size {}", to_check_later.len());
-
-            let until = cmp::min(to_check_later.len(), batch_query_size);
-            let chunk: Vec<Script> = to_check_later.drain(..until).collect();
-            let call_result = self.els_batch_script_get_history(chunk.iter()).await?;
-
-            for (script, history) in chunk.into_iter().zip(call_result.into_iter()) {
-                trace!("received history for {:?}, size {}", script, history.len());
-
-                if !history.is_empty() {
-                    last_found = index;
-
-                    let mut check_later_scripts = self
-                        .check_history(database, script, history, &mut change_max_deriv)
-                        .await?
-                        .into_iter()
-                        .filter(|x| already_checked.insert(x.clone()))
-                        .collect();
-                    to_check_later.append(&mut check_later_scripts);
-                }
-
-                index += 1;
-            }
-
-            match iterating_external {
-                true if index - last_found >= stop_gap => iterating_external = false,
-                true => {
-                    trace!("pushing one more batch from `iter_scriptpubkeys`. index = {}, last_found = {}, stop_gap = {}", index, last_found, stop_gap);
-
-                    let chunk: Vec<Script> =
-                        iter_scriptpubkeys.by_ref().take(batch_query_size).collect();
-                    for item in chunk.into_iter().rev() {
-                        to_check_later.push_front(item);
-                    }
-                }
-                _ => {}
-            }
-        }
-
-        // check utxo
-        // TODO: try to minimize network requests and re-use scripts if possible
-        let mut batch = database.begin_batch();
-        for chunk in ChunksIterator::new(database.iter_utxos()?.into_iter(), batch_query_size) {
-            let scripts: Vec<_> = chunk.iter().map(|u| &u.txout.script_pubkey).collect();
-            let call_result = self.els_batch_script_list_unspent(scripts).await?;
-
-            // check which utxos are actually still unspent
-            for (utxo, list_unspent) in chunk.into_iter().zip(call_result.iter()) {
-                debug!(
-                    "outpoint {:?} is unspent for me, list unspent is {:?}",
-                    utxo.outpoint, list_unspent
-                );
-
-                let mut spent = true;
-                for unspent in list_unspent {
-                    let res_outpoint = OutPoint::new(unspent.tx_hash, unspent.tx_pos as u32);
-                    if utxo.outpoint == res_outpoint {
-                        spent = false;
-                        break;
-                    }
-                }
-                if spent {
-                    info!("{} not anymore unspent, removing", utxo.outpoint);
-                    batch.del_utxo(&utxo.outpoint)?;
-                }
-            }
-        }
-
-        let current_ext = database.get_last_index(ScriptType::External)?.unwrap_or(0);
-        let first_ext_new = last_found as u32 + 1;
-        if first_ext_new > current_ext {
-            info!("Setting external index to {}", first_ext_new);
-            database.set_last_index(ScriptType::External, first_ext_new)?;
-        }
-
-        let current_int = database.get_last_index(ScriptType::Internal)?.unwrap_or(0);
-        let first_int_new = change_max_deriv + 1;
-        if first_int_new > current_int {
-            info!("Setting internal index to {}", first_int_new);
-            database.set_last_index(ScriptType::Internal, first_int_new)?;
-        }
-
-        database.commit_batch(batch)?;
-
-        Ok(())
-    }
-
-    async fn check_tx_and_descendant<D: DatabaseUtils + BatchDatabase>(
-        &mut self,
-        database: &mut D,
-        txid: &Txid,
-        height: Option<u32>,
-        cur_script: &Script,
-        change_max_deriv: &mut u32,
-    ) -> Result<Vec<Script>, Error> {
-        debug!(
-            "check_tx_and_descendant of {}, height: {:?}, script: {}",
-            txid, height, cur_script
-        );
-        let mut updates = database.begin_batch();
-        let tx = match database.get_tx(&txid, true)? {
-            // TODO: do we need the raw?
-            Some(mut saved_tx) => {
-                // update the height if it's different (in case of reorg)
-                if saved_tx.height != height {
-                    info!(
-                        "updating height from {:?} to {:?} for tx {}",
-                        saved_tx.height, height, txid
-                    );
-                    saved_tx.height = height;
-                    updates.set_tx(&saved_tx)?;
-                }
-
-                debug!("already have {} in db, returning the cached version", txid);
-
-                // unwrap since we explicitly ask for the raw_tx, if it's not present something
-                // went wrong
-                saved_tx.transaction.unwrap()
-            }
-            None => self.els_transaction_get(&txid).await?,
-        };
-
-        let mut incoming: u64 = 0;
-        let mut outgoing: u64 = 0;
-
-        // look for our own inputs
-        for (i, input) in tx.input.iter().enumerate() {
-            // the fact that we visit addresses in a BFS fashion starting from the external addresses
-            // should ensure that this query is always consistent (i.e. when we get to call this all
-            // the transactions at a lower depth have already been indexed, so if an outpoint is ours
-            // we are guaranteed to have it in the db).
-            if let Some(previous_output) = database.get_previous_output(&input.previous_output)? {
-                if database.is_mine(&previous_output.script_pubkey)? {
-                    outgoing += previous_output.value;
-
-                    debug!("{} input #{} is mine, removing from utxo", txid, i);
-                    updates.del_utxo(&input.previous_output)?;
-                }
-            }
-        }
-
-        let mut to_check_later = vec![];
-        for (i, output) in tx.output.iter().enumerate() {
-            // this output is ours, we have a path to derive it
-            if let Some((script_type, path)) =
-                database.get_path_from_script_pubkey(&output.script_pubkey)?
-            {
-                debug!("{} output #{} is mine, adding utxo", txid, i);
-                updates.set_utxo(&UTXO {
-                    outpoint: OutPoint::new(tx.txid(), i as u32),
-                    txout: output.clone(),
-                })?;
-                incoming += output.value;
-
-                if output.script_pubkey != *cur_script {
-                    debug!("{} output #{} script {} was not current script, adding script to be checked later", txid, i, output.script_pubkey);
-                    to_check_later.push(output.script_pubkey.clone())
-                }
-
-                // derive as many change addrs as external addresses that we've seen
-                if script_type == ScriptType::Internal
-                    && u32::from(path.as_ref()[0]) > *change_max_deriv
-                {
-                    *change_max_deriv = u32::from(path.as_ref()[0]);
-                }
-            }
-        }
-
-        let tx = TransactionDetails {
-            txid: tx.txid(),
-            transaction: Some(tx),
-            received: incoming,
-            sent: outgoing,
-            height,
-            timestamp: 0,
-        };
-        info!("Saving tx {}", txid);
-        updates.set_tx(&tx)?;
-
-        database.commit_batch(updates)?;
-
-        Ok(to_check_later)
-    }
-
-    async fn check_history<D: DatabaseUtils + BatchDatabase>(
-        &mut self,
-        database: &mut D,
-        script_pubkey: Script,
-        txs: Vec<ELSGetHistoryRes>,
-        change_max_deriv: &mut u32,
-    ) -> Result<Vec<Script>, Error> {
-        let mut to_check_later = Vec::new();
-
-        debug!(
-            "history of {} script {} has {} tx",
-            Address::from_script(&script_pubkey, Network::Testnet).unwrap(),
-            script_pubkey,
-            txs.len()
-        );
-
-        for tx in txs {
-            let height: Option<u32> = match tx.height {
-                0 | -1 => None,
-                x => u32::try_from(x).ok(),
-            };
-
-            to_check_later.extend_from_slice(
-                &self
-                    .check_tx_and_descendant(
-                        database,
-                        &tx.tx_hash,
-                        height,
-                        &script_pubkey,
-                        change_max_deriv,
-                    )
-                    .await?,
-            );
-        }
-
-        Ok(to_check_later)
-    }
-}

src/cli.rs

@@ -1,436 +0,0 @@
-use std::collections::BTreeMap;
-use std::str::FromStr;
-
-use clap::{App, Arg, ArgMatches, SubCommand};
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace, LevelFilter};
-
-use bitcoin::consensus::encode::{deserialize, serialize, serialize_hex};
-use bitcoin::hashes::hex::{FromHex, ToHex};
-use bitcoin::util::psbt::PartiallySignedTransaction;
-use bitcoin::{Address, OutPoint};
-
-use crate::error::Error;
-use crate::types::ScriptType;
-use crate::Wallet;
-
-fn parse_addressee(s: &str) -> Result<(Address, u64), String> {
-    let parts: Vec<_> = s.split(":").collect();
-    if parts.len() != 2 {
-        return Err("Invalid format".to_string());
-    }
-
-    let addr = Address::from_str(&parts[0]);
-    if let Err(e) = addr {
-        return Err(format!("{:?}", e));
-    }
-    let val = u64::from_str(&parts[1]);
-    if let Err(e) = val {
-        return Err(format!("{:?}", e));
-    }
-
-    Ok((addr.unwrap(), val.unwrap()))
-}
-
-fn parse_outpoint(s: &str) -> Result<OutPoint, String> {
-    OutPoint::from_str(s).map_err(|e| format!("{:?}", e))
-}
-
-fn addressee_validator(s: String) -> Result<(), String> {
-    parse_addressee(&s).map(|_| ())
-}
-
-fn outpoint_validator(s: String) -> Result<(), String> {
-    parse_outpoint(&s).map(|_| ())
-}
-
-pub fn make_cli_subcommands<'a, 'b>() -> App<'a, 'b> {
-    App::new("Magical Bitcoin Wallet")
-        .version(option_env!("CARGO_PKG_VERSION").unwrap_or("unknown"))
-        .author(option_env!("CARGO_PKG_AUTHORS").unwrap_or(""))
-        .about("A modern, lightweight, descriptor-based wallet")
-        .subcommand(
-            SubCommand::with_name("get_new_address").about("Generates a new external address"),
-        )
-        .subcommand(SubCommand::with_name("sync").about("Syncs with the chosen Electrum server"))
-        .subcommand(
-            SubCommand::with_name("list_unspent").about("Lists the available spendable UTXOs"),
-        )
-        .subcommand(
-            SubCommand::with_name("get_balance").about("Returns the current wallet balance"),
-        )
-        .subcommand(
-            SubCommand::with_name("create_tx")
-                .about("Creates a new unsigned tranasaction")
-                .arg(
-                    Arg::with_name("to")
-                        .long("to")
-                        .value_name("ADDRESS:SAT")
-                        .help("Adds an addressee to the transaction")
-                        .takes_value(true)
-                        .number_of_values(1)
-                        .required(true)
-                        .multiple(true)
-                        .validator(addressee_validator),
-                )
-                .arg(
-                    Arg::with_name("send_all")
-                        .short("all")
-                        .long("send_all")
-                        .help("Sends all the funds (or all the selected utxos). Requires only one addressees of value 0"),
-                )
-                .arg(
-                    Arg::with_name("utxos")
-                        .long("utxos")
-                        .value_name("TXID:VOUT")
-                        .help("Selects which utxos *must* be spent")
-                        .takes_value(true)
-                        .number_of_values(1)
-                        .multiple(true)
-                        .validator(outpoint_validator),
-                )
-                .arg(
-                    Arg::with_name("unspendable")
-                        .long("unspendable")
-                        .value_name("TXID:VOUT")
-                        .help("Marks an utxo as unspendable")
-                        .takes_value(true)
-                        .number_of_values(1)
-                        .multiple(true)
-                        .validator(outpoint_validator),
-                )
-                .arg(
-                    Arg::with_name("fee_rate")
-                        .short("fee")
-                        .long("fee_rate")
-                        .value_name("SATS_VBYTE")
-                        .help("Fee rate to use in sat/vbyte")
-                        .takes_value(true),
-                )
-                .arg(
-                    Arg::with_name("policy")
-                        .long("policy")
-                        .value_name("POLICY")
-                        .help("Selects which policy should be used to satisfy the descriptor")
-                        .takes_value(true)
-                        .number_of_values(1),
-                ),
-        )
-        .subcommand(
-            SubCommand::with_name("policies")
-                .about("Returns the available spending policies for the descriptor")
-            )
-        .subcommand(
-            SubCommand::with_name("public_descriptor")
-                .about("Returns the public version of the wallet's descriptor(s)")
-            )
-        .subcommand(
-            SubCommand::with_name("sign")
-                .about("Signs and tries to finalize a PSBT")
-                .arg(
-                    Arg::with_name("psbt")
-                        .long("psbt")
-                        .value_name("BASE64_PSBT")
-                        .help("Sets the PSBT to sign")
-                        .takes_value(true)
-                        .number_of_values(1)
-                        .required(true),
-                )
-                .arg(
-                    Arg::with_name("assume_height")
-                        .long("assume_height")
-                        .value_name("HEIGHT")
-                        .help("Assume the blockchain has reached a specific height. This affects the transaction finalization, if there are timelocks in the descriptor")
-                        .takes_value(true)
-                        .number_of_values(1)
-                        .required(false),
-                ))
-        .subcommand(
-            SubCommand::with_name("broadcast")
-                .about("Broadcasts a transaction to the network. Takes either a raw transaction or a PSBT to extract")
-                .arg(
-                    Arg::with_name("psbt")
-                        .long("psbt")
-                        .value_name("BASE64_PSBT")
-                        .help("Sets the PSBT to extract and broadcast")
-                        .takes_value(true)
-                        .required_unless("tx")
-                        .number_of_values(1))
-                .arg(
-                    Arg::with_name("tx")
-                        .long("tx")
-                        .value_name("RAWTX")
-                        .help("Sets the raw transaction to broadcast")
-                        .takes_value(true)
-                        .required_unless("psbt")
-                        .number_of_values(1))
-                )
-        .subcommand(
-            SubCommand::with_name("extract_psbt")
-                .about("Extracts a raw transaction from a PSBT")
-                .arg(
-                    Arg::with_name("psbt")
-                        .long("psbt")
-                        .value_name("BASE64_PSBT")
-                        .help("Sets the PSBT to extract")
-                        .takes_value(true)
-                        .required(true)
-                        .number_of_values(1))
-                )
-        .subcommand(
-            SubCommand::with_name("finalize_psbt")
-                .about("Finalizes a psbt")
-                .arg(
-                    Arg::with_name("psbt")
-                        .long("psbt")
-                        .value_name("BASE64_PSBT")
-                        .help("Sets the PSBT to finalize")
-                        .takes_value(true)
-                        .required(true)
-                        .number_of_values(1))
-                .arg(
-                    Arg::with_name("assume_height")
-                        .long("assume_height")
-                        .value_name("HEIGHT")
-                        .help("Assume the blockchain has reached a specific height")
-                        .takes_value(true)
-                        .number_of_values(1)
-                        .required(false))
-                )
-        .subcommand(
-            SubCommand::with_name("combine_psbt")
-                .about("Combines multiple PSBTs into one")
-                .arg(
-                    Arg::with_name("psbt")
-                        .long("psbt")
-                        .value_name("BASE64_PSBT")
-                        .help("Add one PSBT to comine. This option can be repeated multiple times, one for each PSBT")
-                        .takes_value(true)
-                        .number_of_values(1)
-                        .required(true)
-                        .multiple(true))
-                )
-}
-
-pub fn add_global_flags<'a, 'b>(app: App<'a, 'b>) -> App<'a, 'b> {
-    app.arg(
-        Arg::with_name("network")
-            .short("n")
-            .long("network")
-            .value_name("NETWORK")
-            .help("Sets the network")
-            .takes_value(true)
-            .default_value("testnet")
-            .possible_values(&["testnet", "regtest"]),
-    )
-    .arg(
-        Arg::with_name("wallet")
-            .short("w")
-            .long("wallet")
-            .value_name("WALLET_NAME")
-            .help("Selects the wallet to use")
-            .takes_value(true)
-            .default_value("main"),
-    )
-    .arg(
-        Arg::with_name("server")
-            .short("s")
-            .long("server")
-            .value_name("SERVER:PORT")
-            .help("Sets the Electrum server to use")
-            .takes_value(true)
-            .default_value("tn.not.fyi:55001"),
-    )
-    .arg(
-        Arg::with_name("descriptor")
-            .short("d")
-            .long("descriptor")
-            .value_name("DESCRIPTOR")
-            .help("Sets the descriptor to use for the external addresses")
-            .required(true)
-            .takes_value(true),
-    )
-    .arg(
-        Arg::with_name("change_descriptor")
-            .short("c")
-            .long("change_descriptor")
-            .value_name("DESCRIPTOR")
-            .help("Sets the descriptor to use for internal addresses")
-            .takes_value(true),
-    )
-    .arg(
-        Arg::with_name("v")
-            .short("v")
-            .multiple(true)
-            .help("Sets the level of verbosity"),
-    )
-    .subcommand(SubCommand::with_name("repl").about("Opens an interactive shell"))
-}
-
-pub async fn handle_matches<C, D>(
-    wallet: &Wallet<C, D>,
-    matches: ArgMatches<'_>,
-) -> Result<Option<String>, Error>
-where
-    C: crate::blockchain::OnlineBlockchain,
-    D: crate::database::BatchDatabase,
-{
-    if let Some(_sub_matches) = matches.subcommand_matches("get_new_address") {
-        Ok(Some(format!("{}", wallet.get_new_address()?)))
-    } else if let Some(_sub_matches) = matches.subcommand_matches("sync") {
-        wallet.sync(None, None).await?;
-        Ok(None)
-    } else if let Some(_sub_matches) = matches.subcommand_matches("list_unspent") {
-        let mut res = String::new();
-        for utxo in wallet.list_unspent()? {
-            res += &format!("{} value {} SAT\n", utxo.outpoint, utxo.txout.value);
-        }
-
-        Ok(Some(res))
-    } else if let Some(_sub_matches) = matches.subcommand_matches("get_balance") {
-        Ok(Some(format!("{} SAT", wallet.get_balance()?)))
-    } else if let Some(sub_matches) = matches.subcommand_matches("create_tx") {
-        let addressees = sub_matches
-            .values_of("to")
-            .unwrap()
-            .map(|s| parse_addressee(s))
-            .collect::<Result<Vec<_>, _>>()
-            .map_err(|s| Error::Generic(s))?;
-        let send_all = sub_matches.is_present("send_all");
-        let fee_rate = sub_matches
-            .value_of("fee_rate")
-            .map(|s| f32::from_str(s).unwrap())
-            .unwrap_or(1.0);
-        let utxos = sub_matches
-            .values_of("utxos")
-            .map(|s| s.map(|i| parse_outpoint(i).unwrap()).collect());
-        let unspendable = sub_matches
-            .values_of("unspendable")
-            .map(|s| s.map(|i| parse_outpoint(i).unwrap()).collect());
-        let policy: Option<_> = sub_matches
-            .value_of("policy")
-            .map(|s| serde_json::from_str::<BTreeMap<String, Vec<usize>>>(&s).unwrap());
-
-        let result = wallet.create_tx(
-            addressees,
-            send_all,
-            fee_rate * 1e-5,
-            policy,
-            utxos,
-            unspendable,
-        )?;
-        Ok(Some(format!(
-            "{:#?}\nPSBT: {}",
-            result.1,
-            base64::encode(&serialize(&result.0))
-        )))
-    } else if let Some(_sub_matches) = matches.subcommand_matches("policies") {
-        Ok(Some(format!(
-            "External: {}\nInternal:{}",
-            serde_json::to_string(&wallet.policies(ScriptType::External)?).unwrap(),
-            serde_json::to_string(&wallet.policies(ScriptType::Internal)?).unwrap(),
-        )))
-    } else if let Some(_sub_matches) = matches.subcommand_matches("public_descriptor") {
-        let external = match wallet.public_descriptor(ScriptType::External)? {
-            Some(desc) => format!("{}", desc),
-            None => "<NONE>".into(),
-        };
-        let internal = match wallet.public_descriptor(ScriptType::Internal)? {
-            Some(desc) => format!("{}", desc),
-            None => "<NONE>".into(),
-        };
-
-        Ok(Some(format!(
-            "External: {}\nInternal:{}",
-            external, internal
-        )))
-    } else if let Some(sub_matches) = matches.subcommand_matches("sign") {
-        let psbt = base64::decode(sub_matches.value_of("psbt").unwrap()).unwrap();
-        let psbt: PartiallySignedTransaction = deserialize(&psbt).unwrap();
-        let assume_height = sub_matches
-            .value_of("assume_height")
-            .and_then(|s| Some(s.parse().unwrap()));
-        let (psbt, finalized) = wallet.sign(psbt, assume_height)?;
-
-        let mut res = String::new();
-
-        res += &format!("PSBT: {}\n", base64::encode(&serialize(&psbt)));
-        res += &format!("Finalized: {}", finalized);
-        if finalized {
-            res += &format!("\nExtracted: {}", serialize_hex(&psbt.extract_tx()));
-        }
-
-        Ok(Some(res))
-    } else if let Some(sub_matches) = matches.subcommand_matches("broadcast") {
-        let tx = if sub_matches.value_of("psbt").is_some() {
-            let psbt = base64::decode(&sub_matches.value_of("psbt").unwrap()).unwrap();
-            let psbt: PartiallySignedTransaction = deserialize(&psbt).unwrap();
-            psbt.extract_tx()
-        } else if sub_matches.value_of("tx").is_some() {
-            deserialize(&Vec::<u8>::from_hex(&sub_matches.value_of("tx").unwrap()).unwrap())
-                .unwrap()
-        } else {
-            panic!("Missing `psbt` and `tx` option");
-        };
-
-        let txid = wallet.broadcast(tx).await?;
-
-        Ok(Some(format!("TXID: {}", txid)))
-    } else if let Some(sub_matches) = matches.subcommand_matches("extract_psbt") {
-        let psbt = base64::decode(&sub_matches.value_of("psbt").unwrap()).unwrap();
-        let psbt: PartiallySignedTransaction = deserialize(&psbt).unwrap();
-
-        Ok(Some(format!(
-            "TX: {}",
-            serialize(&psbt.extract_tx()).to_hex()
-        )))
-    } else if let Some(sub_matches) = matches.subcommand_matches("finalize_psbt") {
-        let psbt = base64::decode(&sub_matches.value_of("psbt").unwrap()).unwrap();
-        let mut psbt: PartiallySignedTransaction = deserialize(&psbt).unwrap();
-
-        let assume_height = sub_matches
-            .value_of("assume_height")
-            .and_then(|s| Some(s.parse().unwrap()));
-
-        let finalized = wallet.finalize_psbt(&mut psbt, assume_height)?;
-
-        let mut res = String::new();
-        res += &format!("PSBT: {}\n", base64::encode(&serialize(&psbt)));
-        res += &format!("Finalized: {}", finalized);
-        if finalized {
-            res += &format!("\nExtracted: {}", serialize_hex(&psbt.extract_tx()));
-        }
-
-        Ok(Some(res))
-    } else if let Some(sub_matches) = matches.subcommand_matches("combine_psbt") {
-        let mut psbts = sub_matches
-            .values_of("psbt")
-            .unwrap()
-            .map(|s| {
-                let psbt = base64::decode(&s).unwrap();
-                let psbt: PartiallySignedTransaction = deserialize(&psbt).unwrap();
-
-                psbt
-            })
-            .collect::<Vec<_>>();
-
-        let init_psbt = psbts.pop().unwrap();
-        let final_psbt = psbts
-            .into_iter()
-            .try_fold::<_, _, Result<PartiallySignedTransaction, Error>>(
-                init_psbt,
-                |mut acc, x| {
-                    acc.merge(x)?;
-                    Ok(acc)
-                },
-            )?;
-
-        Ok(Some(format!(
-            "PSBT: {}",
-            base64::encode(&serialize(&final_psbt))
-        )))
-    } else {
-        Ok(None)
-    }
-}

src/database/any.rs

@@ -0,0 +1,427 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Runtime-checked database types
+//!
+//! This module provides the implementation of [`AnyDatabase`] which allows switching the
+//! inner [`Database`] type at runtime.
+//!
+//! ## Example
+//!
+//! In this example, `wallet_memory` and `wallet_sled` have the same type of `Wallet<(), AnyDatabase>`.
+//!
+//! ```no_run
+//! # use bitcoin::Network;
+//! # use bdk::database::{AnyDatabase, MemoryDatabase};
+//! # use bdk::{Wallet};
+//! let memory = MemoryDatabase::default();
+//! let wallet_memory = Wallet::new("...", None, Network::Testnet, memory)?;
+//!
+//! # #[cfg(feature = "key-value-db")]
+//! # {
+//! let sled = sled::open("my-database")?.open_tree("default_tree")?;
+//! let wallet_sled = Wallet::new("...", None, Network::Testnet, sled)?;
+//! # }
+//! # Ok::<(), bdk::Error>(())
+//! ```
+//!
+//! When paired with the use of [`ConfigurableDatabase`], it allows creating wallets with any
+//! database supported using a single line of code:
+//!
+//! ```no_run
+//! # use bitcoin::Network;
+//! # use bdk::database::*;
+//! # use bdk::{Wallet};
+//! let config = serde_json::from_str("...")?;
+//! let database = AnyDatabase::from_config(&config)?;
+//! let wallet = Wallet::new("...", None, Network::Testnet, database)?;
+//! # Ok::<(), bdk::Error>(())
+//! ```
+
+use super::*;
+
+macro_rules! impl_from {
+    ( $from:ty, $to:ty, $variant:ident, $( $cfg:tt )* ) => {
+        $( $cfg )*
+        impl From<$from> for $to {
+            fn from(inner: $from) -> Self {
+                <$to>::$variant(inner)
+            }
+        }
+    };
+}
+
+macro_rules! impl_inner_method {
+    ( $enum_name:ident, $self:expr, $name:ident $(, $args:expr)* ) => {
+        #[allow(deprecated)]
+        match $self {
+            $enum_name::Memory(inner) => inner.$name( $($args, )* ),
+            #[cfg(feature = "key-value-db")]
+            $enum_name::Sled(inner) => inner.$name( $($args, )* ),
+            #[cfg(feature = "sqlite")]
+            $enum_name::Sqlite(inner) => inner.$name( $($args, )* ),
+        }
+    }
+}
+
+/// Type that can contain any of the [`Database`] types defined by the library
+///
+/// It allows switching database type at runtime.
+///
+/// See [this module](crate::database::any)'s documentation for a usage example.
+#[derive(Debug)]
+pub enum AnyDatabase {
+    /// In-memory ephemeral database
+    Memory(memory::MemoryDatabase),
+    #[cfg(feature = "key-value-db")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "key-value-db")))]
+    /// Simple key-value embedded database based on [`sled`]
+    Sled(sled::Tree),
+    #[cfg(feature = "sqlite")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "sqlite")))]
+    /// Sqlite embedded database using [`rusqlite`]
+    Sqlite(sqlite::SqliteDatabase),
+}
+
+impl_from!(memory::MemoryDatabase, AnyDatabase, Memory,);
+impl_from!(sled::Tree, AnyDatabase, Sled, #[cfg(feature = "key-value-db")]);
+impl_from!(sqlite::SqliteDatabase, AnyDatabase, Sqlite, #[cfg(feature = "sqlite")]);
+
+/// Type that contains any of the [`BatchDatabase::Batch`] types defined by the library
+pub enum AnyBatch {
+    /// In-memory ephemeral database
+    Memory(<memory::MemoryDatabase as BatchDatabase>::Batch),
+    #[cfg(feature = "key-value-db")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "key-value-db")))]
+    /// Simple key-value embedded database based on [`sled`]
+    Sled(<sled::Tree as BatchDatabase>::Batch),
+    #[cfg(feature = "sqlite")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "sqlite")))]
+    /// Sqlite embedded database using [`rusqlite`]
+    Sqlite(<sqlite::SqliteDatabase as BatchDatabase>::Batch),
+}
+
+impl_from!(
+    <memory::MemoryDatabase as BatchDatabase>::Batch,
+    AnyBatch,
+    Memory,
+);
+impl_from!(<sled::Tree as BatchDatabase>::Batch, AnyBatch, Sled, #[cfg(feature = "key-value-db")]);
+impl_from!(<sqlite::SqliteDatabase as BatchDatabase>::Batch, AnyBatch, Sqlite, #[cfg(feature = "sqlite")]);
+
+impl BatchOperations for AnyDatabase {
+    fn set_script_pubkey(
+        &mut self,
+        script: &Script,
+        keychain: KeychainKind,
+        child: u32,
+    ) -> Result<(), Error> {
+        impl_inner_method!(
+            AnyDatabase,
+            self,
+            set_script_pubkey,
+            script,
+            keychain,
+            child
+        )
+    }
+    fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error> {
+        impl_inner_method!(AnyDatabase, self, set_utxo, utxo)
+    }
+    fn set_raw_tx(&mut self, transaction: &Transaction) -> Result<(), Error> {
+        impl_inner_method!(AnyDatabase, self, set_raw_tx, transaction)
+    }
+    fn set_tx(&mut self, transaction: &TransactionDetails) -> Result<(), Error> {
+        impl_inner_method!(AnyDatabase, self, set_tx, transaction)
+    }
+    fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error> {
+        impl_inner_method!(AnyDatabase, self, set_last_index, keychain, value)
+    }
+    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error> {
+        impl_inner_method!(AnyDatabase, self, set_sync_time, sync_time)
+    }
+
+    fn del_script_pubkey_from_path(
+        &mut self,
+        keychain: KeychainKind,
+        child: u32,
+    ) -> Result<Option<ScriptBuf>, Error> {
+        impl_inner_method!(
+            AnyDatabase,
+            self,
+            del_script_pubkey_from_path,
+            keychain,
+            child
+        )
+    }
+    fn del_path_from_script_pubkey(
+        &mut self,
+        script: &Script,
+    ) -> Result<Option<(KeychainKind, u32)>, Error> {
+        impl_inner_method!(AnyDatabase, self, del_path_from_script_pubkey, script)
+    }
+    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+        impl_inner_method!(AnyDatabase, self, del_utxo, outpoint)
+    }
+    fn del_raw_tx(&mut self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        impl_inner_method!(AnyDatabase, self, del_raw_tx, txid)
+    }
+    fn del_tx(
+        &mut self,
+        txid: &Txid,
+        include_raw: bool,
+    ) -> Result<Option<TransactionDetails>, Error> {
+        impl_inner_method!(AnyDatabase, self, del_tx, txid, include_raw)
+    }
+    fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
+        impl_inner_method!(AnyDatabase, self, del_last_index, keychain)
+    }
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+        impl_inner_method!(AnyDatabase, self, del_sync_time)
+    }
+}
+
+impl Database for AnyDatabase {
+    fn check_descriptor_checksum<B: AsRef<[u8]>>(
+        &mut self,
+        keychain: KeychainKind,
+        bytes: B,
+    ) -> Result<(), Error> {
+        impl_inner_method!(
+            AnyDatabase,
+            self,
+            check_descriptor_checksum,
+            keychain,
+            bytes
+        )
+    }
+
+    fn iter_script_pubkeys(&self, keychain: Option<KeychainKind>) -> Result<Vec<ScriptBuf>, Error> {
+        impl_inner_method!(AnyDatabase, self, iter_script_pubkeys, keychain)
+    }
+    fn iter_utxos(&self) -> Result<Vec<LocalUtxo>, Error> {
+        impl_inner_method!(AnyDatabase, self, iter_utxos)
+    }
+    fn iter_raw_txs(&self) -> Result<Vec<Transaction>, Error> {
+        impl_inner_method!(AnyDatabase, self, iter_raw_txs)
+    }
+    fn iter_txs(&self, include_raw: bool) -> Result<Vec<TransactionDetails>, Error> {
+        impl_inner_method!(AnyDatabase, self, iter_txs, include_raw)
+    }
+
+    fn get_script_pubkey_from_path(
+        &self,
+        keychain: KeychainKind,
+        child: u32,
+    ) -> Result<Option<ScriptBuf>, Error> {
+        impl_inner_method!(
+            AnyDatabase,
+            self,
+            get_script_pubkey_from_path,
+            keychain,
+            child
+        )
+    }
+    fn get_path_from_script_pubkey(
+        &self,
+        script: &Script,
+    ) -> Result<Option<(KeychainKind, u32)>, Error> {
+        impl_inner_method!(AnyDatabase, self, get_path_from_script_pubkey, script)
+    }
+    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+        impl_inner_method!(AnyDatabase, self, get_utxo, outpoint)
+    }
+    fn get_raw_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        impl_inner_method!(AnyDatabase, self, get_raw_tx, txid)
+    }
+    fn get_tx(&self, txid: &Txid, include_raw: bool) -> Result<Option<TransactionDetails>, Error> {
+        impl_inner_method!(AnyDatabase, self, get_tx, txid, include_raw)
+    }
+    fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
+        impl_inner_method!(AnyDatabase, self, get_last_index, keychain)
+    }
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        impl_inner_method!(AnyDatabase, self, get_sync_time)
+    }
+
+    fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
+        impl_inner_method!(AnyDatabase, self, increment_last_index, keychain)
+    }
+}
+
+impl BatchOperations for AnyBatch {
+    fn set_script_pubkey(
+        &mut self,
+        script: &Script,
+        keychain: KeychainKind,
+        child: u32,
+    ) -> Result<(), Error> {
+        impl_inner_method!(AnyBatch, self, set_script_pubkey, script, keychain, child)
+    }
+    fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error> {
+        impl_inner_method!(AnyBatch, self, set_utxo, utxo)
+    }
+    fn set_raw_tx(&mut self, transaction: &Transaction) -> Result<(), Error> {
+        impl_inner_method!(AnyBatch, self, set_raw_tx, transaction)
+    }
+    fn set_tx(&mut self, transaction: &TransactionDetails) -> Result<(), Error> {
+        impl_inner_method!(AnyBatch, self, set_tx, transaction)
+    }
+    fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error> {
+        impl_inner_method!(AnyBatch, self, set_last_index, keychain, value)
+    }
+    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error> {
+        impl_inner_method!(AnyBatch, self, set_sync_time, sync_time)
+    }
+
+    fn del_script_pubkey_from_path(
+        &mut self,
+        keychain: KeychainKind,
+        child: u32,
+    ) -> Result<Option<ScriptBuf>, Error> {
+        impl_inner_method!(AnyBatch, self, del_script_pubkey_from_path, keychain, child)
+    }
+    fn del_path_from_script_pubkey(
+        &mut self,
+        script: &Script,
+    ) -> Result<Option<(KeychainKind, u32)>, Error> {
+        impl_inner_method!(AnyBatch, self, del_path_from_script_pubkey, script)
+    }
+    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+        impl_inner_method!(AnyBatch, self, del_utxo, outpoint)
+    }
+    fn del_raw_tx(&mut self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        impl_inner_method!(AnyBatch, self, del_raw_tx, txid)
+    }
+    fn del_tx(
+        &mut self,
+        txid: &Txid,
+        include_raw: bool,
+    ) -> Result<Option<TransactionDetails>, Error> {
+        impl_inner_method!(AnyBatch, self, del_tx, txid, include_raw)
+    }
+    fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
+        impl_inner_method!(AnyBatch, self, del_last_index, keychain)
+    }
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+        impl_inner_method!(AnyBatch, self, del_sync_time)
+    }
+}
+
+impl BatchDatabase for AnyDatabase {
+    type Batch = AnyBatch;
+
+    fn begin_batch(&self) -> Self::Batch {
+        match self {
+            AnyDatabase::Memory(inner) => inner.begin_batch().into(),
+            #[cfg(feature = "key-value-db")]
+            AnyDatabase::Sled(inner) => inner.begin_batch().into(),
+            #[cfg(feature = "sqlite")]
+            AnyDatabase::Sqlite(inner) => inner.begin_batch().into(),
+        }
+    }
+    fn commit_batch(&mut self, batch: Self::Batch) -> Result<(), Error> {
+        match self {
+            AnyDatabase::Memory(db) => match batch {
+                AnyBatch::Memory(batch) => db.commit_batch(batch),
+                #[cfg(any(feature = "key-value-db", feature = "sqlite"))]
+                _ => unimplemented!("Other batch shouldn't be used with Memory db."),
+            },
+            #[cfg(feature = "key-value-db")]
+            AnyDatabase::Sled(db) => match batch {
+                AnyBatch::Sled(batch) => db.commit_batch(batch),
+                _ => unimplemented!("Other batch shouldn't be used with Sled db."),
+            },
+            #[cfg(feature = "sqlite")]
+            AnyDatabase::Sqlite(db) => match batch {
+                AnyBatch::Sqlite(batch) => db.commit_batch(batch),
+                _ => unimplemented!("Other batch shouldn't be used with Sqlite db."),
+            },
+        }
+    }
+}
+
+/// Configuration type for a [`sled::Tree`] database
+#[cfg(feature = "key-value-db")]
+#[derive(Debug, serde::Serialize, serde::Deserialize)]
+pub struct SledDbConfiguration {
+    /// Main directory of the db
+    pub path: String,
+    /// Name of the database tree, a separated namespace for the data
+    pub tree_name: String,
+}
+
+#[cfg(feature = "key-value-db")]
+impl ConfigurableDatabase for sled::Tree {
+    type Config = SledDbConfiguration;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        Ok(sled::open(&config.path)?.open_tree(&config.tree_name)?)
+    }
+}
+
+/// Configuration type for a [`sqlite::SqliteDatabase`] database
+#[cfg(feature = "sqlite")]
+#[derive(Debug, serde::Serialize, serde::Deserialize)]
+pub struct SqliteDbConfiguration {
+    /// Main directory of the db
+    pub path: String,
+}
+
+#[cfg(feature = "sqlite")]
+impl ConfigurableDatabase for sqlite::SqliteDatabase {
+    type Config = SqliteDbConfiguration;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        Ok(sqlite::SqliteDatabase::new(config.path.clone()))
+    }
+}
+
+/// Type that can contain any of the database configurations defined by the library
+///
+/// This allows storing a single configuration that can be loaded into an [`AnyDatabase`]
+/// instance. Wallets that plan to offer users the ability to switch blockchain backend at runtime
+/// will find this particularly useful.
+#[derive(Debug, serde::Serialize, serde::Deserialize)]
+pub enum AnyDatabaseConfig {
+    /// Memory database has no config
+    Memory(()),
+    #[cfg(feature = "key-value-db")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "key-value-db")))]
+    /// Simple key-value embedded database based on [`sled`]
+    Sled(SledDbConfiguration),
+    #[cfg(feature = "sqlite")]
+    #[cfg_attr(docsrs, doc(cfg(feature = "sqlite")))]
+    /// Sqlite embedded database using [`rusqlite`]
+    Sqlite(SqliteDbConfiguration),
+}
+
+impl ConfigurableDatabase for AnyDatabase {
+    type Config = AnyDatabaseConfig;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        Ok(match config {
+            AnyDatabaseConfig::Memory(inner) => {
+                AnyDatabase::Memory(memory::MemoryDatabase::from_config(inner)?)
+            }
+            #[cfg(feature = "key-value-db")]
+            AnyDatabaseConfig::Sled(inner) => AnyDatabase::Sled(sled::Tree::from_config(inner)?),
+            #[cfg(feature = "sqlite")]
+            AnyDatabaseConfig::Sqlite(inner) => {
+                AnyDatabase::Sqlite(sqlite::SqliteDatabase::from_config(inner)?)
+            }
+        })
+    }
+}
+
+impl_from!((), AnyDatabaseConfig, Memory,);
+impl_from!(SledDbConfiguration, AnyDatabaseConfig, Sled, #[cfg(feature = "key-value-db")]);
+impl_from!(SqliteDbConfiguration, AnyDatabaseConfig, Sqlite, #[cfg(feature = "sqlite")]);

src/database/keyvalue.rs

@@ -1,38 +1,51 @@
-use std::convert::{From, TryInto};
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use std::convert::TryInto;
 
 use sled::{Batch, Tree};
 
 use bitcoin::consensus::encode::{deserialize, serialize};
 use bitcoin::hash_types::Txid;
-use bitcoin::util::bip32::{ChildNumber, DerivationPath};
-use bitcoin::{OutPoint, Script, Transaction};
+use bitcoin::{OutPoint, Script, ScriptBuf, Transaction};
 
 use crate::database::memory::MapKey;
-use crate::database::{BatchDatabase, BatchOperations, Database};
+use crate::database::{BatchDatabase, BatchOperations, Database, SyncTime};
 use crate::error::Error;
 use crate::types::*;
 
 macro_rules! impl_batch_operations {
     ( { $($after_insert:tt)* }, $process_delete:ident ) => {
-        fn set_script_pubkey<P: AsRef<[ChildNumber]>>(&mut self, script: &Script, script_type: ScriptType, path: &P) -> Result<(), Error> {
-            let deriv_path = DerivationPath::from(path.as_ref());
-            let key = MapKey::Path((Some(script_type), Some(&deriv_path))).as_map_key();
+        fn set_script_pubkey(&mut self, script: &Script, keychain: KeychainKind, path: u32) -> Result<(), Error> {
+            let key = MapKey::Path((Some(keychain), Some(path))).as_map_key();
             self.insert(key, serialize(script))$($after_insert)*;
 
             let key = MapKey::Script(Some(script)).as_map_key();
             let value = json!({
-                "t": script_type,
-                "p": deriv_path,
+                "t": keychain,
+                "p": path,
             });
             self.insert(key, serde_json::to_vec(&value)?)$($after_insert)*;
 
             Ok(())
         }
 
-        fn set_utxo(&mut self, utxo: &UTXO) -> Result<(), Error> {
-            let key = MapKey::UTXO(Some(&utxo.outpoint)).as_map_key();
-            let value = serialize(&utxo.txout);
-            self.insert(key, value)$($after_insert)*;
+        fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error> {
+            let key = MapKey::Utxo(Some(&utxo.outpoint)).as_map_key();
+            let value = json!({
+                "t": utxo.txout,
+                "i": utxo.keychain,
+                "s": utxo.is_spent,
+            });
+            self.insert(key, serde_json::to_vec(&value)?)$($after_insert)*;
 
             Ok(())
         }
@@ -63,23 +76,29 @@ macro_rules! impl_batch_operations {
             Ok(())
         }
 
-        fn set_last_index(&mut self, script_type: ScriptType, value: u32) -> Result<(), Error> {
-            let key = MapKey::LastIndex(script_type).as_map_key();
+        fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error> {
+            let key = MapKey::LastIndex(keychain).as_map_key();
             self.insert(key, &value.to_be_bytes())$($after_insert)*;
 
             Ok(())
         }
 
-        fn del_script_pubkey_from_path<P: AsRef<[ChildNumber]>>(&mut self, script_type: ScriptType, path: &P) -> Result<Option<Script>, Error> {
-            let deriv_path = DerivationPath::from(path.as_ref());
-            let key = MapKey::Path((Some(script_type), Some(&deriv_path))).as_map_key();
+        fn set_sync_time(&mut self, data: SyncTime) -> Result<(), Error> {
+            let key = MapKey::SyncTime.as_map_key();
+            self.insert(key, serde_json::to_vec(&data)?)$($after_insert)*;
+
+            Ok(())
+        }
+
+        fn del_script_pubkey_from_path(&mut self, keychain: KeychainKind, path: u32) -> Result<Option<ScriptBuf>, Error> {
+            let key = MapKey::Path((Some(keychain), Some(path))).as_map_key();
             let res = self.remove(key);
             let res = $process_delete!(res);
 
             Ok(res.map_or(Ok(None), |x| Some(deserialize(&x)).transpose())?)
         }
 
-        fn del_path_from_script_pubkey(&mut self, script: &Script) -> Result<Option<(ScriptType, DerivationPath)>, Error> {
+        fn del_path_from_script_pubkey(&mut self, script: &Script) -> Result<Option<(KeychainKind, u32)>, Error> {
             let key = MapKey::Script(Some(script)).as_map_key();
             let res = self.remove(key);
             let res = $process_delete!(res);
@@ -96,16 +115,20 @@ macro_rules! impl_batch_operations {
             }
         }
 
-        fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
-            let key = MapKey::UTXO(Some(outpoint)).as_map_key();
+        fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+            let key = MapKey::Utxo(Some(outpoint)).as_map_key();
             let res = self.remove(key);
             let res = $process_delete!(res);
 
             match res {
                 None => Ok(None),
                 Some(b) => {
-                    let txout = deserialize(&b)?;
-                    Ok(Some(UTXO { outpoint: outpoint.clone(), txout }))
+                    let mut val: serde_json::Value = serde_json::from_slice(&b)?;
+                    let txout = serde_json::from_value(val["t"].take())?;
+                    let keychain = serde_json::from_value(val["i"].take())?;
+                    let is_spent = val.get_mut("s").and_then(|s| s.take().as_bool()).unwrap_or(false);
+
+                    Ok(Some(LocalUtxo { outpoint: outpoint.clone(), txout, keychain, is_spent, }))
                 }
             }
         }
@@ -140,19 +163,20 @@ macro_rules! impl_batch_operations {
             }
         }
 
-        fn del_last_index(&mut self, script_type: ScriptType) -> Result<Option<u32>, Error> {
-            let key = MapKey::LastIndex(script_type).as_map_key();
+        fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
+            let key = MapKey::LastIndex(keychain).as_map_key();
+            let res = self.remove(key);
+            $process_delete!(res)
+            .map(ivec_to_u32)
+            .transpose()
+        }
+
+        fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+            let key = MapKey::SyncTime.as_map_key();
             let res = self.remove(key);
             let res = $process_delete!(res);
 
-            match res {
-                None => Ok(None),
-                Some(b) => {
-                    let array: [u8; 4] = b.as_ref().try_into().map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
-                    let val = u32::from_be_bytes(array);
-                    Ok(Some(val))
-                }
-            }
+            Ok(res.map(|b| serde_json::from_slice(&b)).transpose()?)
         }
     }
 }
@@ -179,10 +203,10 @@ impl BatchOperations for Batch {
 impl Database for Tree {
     fn check_descriptor_checksum<B: AsRef<[u8]>>(
         &mut self,
-        script_type: ScriptType,
+        keychain: KeychainKind,
         bytes: B,
     ) -> Result<(), Error> {
-        let key = MapKey::DescriptorChecksum(script_type).as_map_key();
+        let key = MapKey::DescriptorChecksum(keychain).as_map_key();
 
         let prev = self.get(&key)?.map(|x| x.to_vec());
         if let Some(val) = prev {
@@ -197,8 +221,8 @@ impl Database for Tree {
         }
     }
 
-    fn iter_script_pubkeys(&self, script_type: Option<ScriptType>) -> Result<Vec<Script>, Error> {
-        let key = MapKey::Path((script_type, None)).as_map_key();
+    fn iter_script_pubkeys(&self, keychain: Option<KeychainKind>) -> Result<Vec<ScriptBuf>, Error> {
+        let key = MapKey::Path((keychain, None)).as_map_key();
         self.scan_prefix(key)
             .map(|x| -> Result<_, Error> {
                 let (_, v) = x?;
@@ -207,14 +231,27 @@ impl Database for Tree {
             .collect()
     }
 
-    fn iter_utxos(&self) -> Result<Vec<UTXO>, Error> {
-        let key = MapKey::UTXO(None).as_map_key();
+    fn iter_utxos(&self) -> Result<Vec<LocalUtxo>, Error> {
+        let key = MapKey::Utxo(None).as_map_key();
         self.scan_prefix(key)
             .map(|x| -> Result<_, Error> {
                 let (k, v) = x?;
                 let outpoint = deserialize(&k[1..])?;
-                let txout = deserialize(&v)?;
-                Ok(UTXO { outpoint, txout })
+
+                let mut val: serde_json::Value = serde_json::from_slice(&v)?;
+                let txout = serde_json::from_value(val["t"].take())?;
+                let keychain = serde_json::from_value(val["i"].take())?;
+                let is_spent = val
+                    .get_mut("s")
+                    .and_then(|s| s.take().as_bool())
+                    .unwrap_or(false);
+
+                Ok(LocalUtxo {
+                    outpoint,
+                    txout,
+                    keychain,
+                    is_spent,
+                })
             })
             .collect()
     }
@@ -245,20 +282,19 @@ impl Database for Tree {
             .collect()
     }
 
-    fn get_script_pubkey_from_path<P: AsRef<[ChildNumber]>>(
+    fn get_script_pubkey_from_path(
         &self,
-        script_type: ScriptType,
-        path: &P,
-    ) -> Result<Option<Script>, Error> {
-        let deriv_path = DerivationPath::from(path.as_ref());
-        let key = MapKey::Path((Some(script_type), Some(&deriv_path))).as_map_key();
+        keychain: KeychainKind,
+        path: u32,
+    ) -> Result<Option<ScriptBuf>, Error> {
+        let key = MapKey::Path((Some(keychain), Some(path))).as_map_key();
         Ok(self.get(key)?.map(|b| deserialize(&b)).transpose()?)
     }
 
     fn get_path_from_script_pubkey(
         &self,
         script: &Script,
-    ) -> Result<Option<(ScriptType, DerivationPath)>, Error> {
+    ) -> Result<Option<(KeychainKind, u32)>, Error> {
         let key = MapKey::Script(Some(script)).as_map_key();
         self.get(key)?
             .map(|b| -> Result<_, Error> {
@@ -271,14 +307,23 @@ impl Database for Tree {
             .transpose()
     }
 
-    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
-        let key = MapKey::UTXO(Some(outpoint)).as_map_key();
+    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+        let key = MapKey::Utxo(Some(outpoint)).as_map_key();
         self.get(key)?
             .map(|b| -> Result<_, Error> {
-                let txout = deserialize(&b)?;
-                Ok(UTXO {
-                    outpoint: outpoint.clone(),
+                let mut val: serde_json::Value = serde_json::from_slice(&b)?;
+                let txout = serde_json::from_value(val["t"].take())?;
+                let keychain = serde_json::from_value(val["i"].take())?;
+                let is_spent = val
+                    .get_mut("s")
+                    .and_then(|s| s.take().as_bool())
+                    .unwrap_or(false);
+
+                Ok(LocalUtxo {
+                    outpoint: *outpoint,
                     txout,
+                    keychain,
+                    is_spent,
                 })
             })
             .transpose()
@@ -295,7 +340,7 @@ impl Database for Tree {
             .map(|b| -> Result<_, Error> {
                 let mut txdetails: TransactionDetails = serde_json::from_slice(&b)?;
                 if include_raw {
-                    txdetails.transaction = self.get_raw_tx(&txid)?;
+                    txdetails.transaction = self.get_raw_tx(txid)?;
                 }
 
                 Ok(txdetails)
@@ -303,23 +348,22 @@ impl Database for Tree {
             .transpose()
     }
 
-    fn get_last_index(&self, script_type: ScriptType) -> Result<Option<u32>, Error> {
-        let key = MapKey::LastIndex(script_type).as_map_key();
-        self.get(key)?
-            .map(|b| -> Result<_, Error> {
-                let array: [u8; 4] = b
-                    .as_ref()
-                    .try_into()
-                    .map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
-                let val = u32::from_be_bytes(array);
-                Ok(val)
-            })
-            .transpose()
+    fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
+        let key = MapKey::LastIndex(keychain).as_map_key();
+        self.get(key)?.map(ivec_to_u32).transpose()
+    }
+
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        Ok(self
+            .get(key)?
+            .map(|b| serde_json::from_slice(&b))
+            .transpose()?)
     }
 
     // inserts 0 if not present
-    fn increment_last_index(&mut self, script_type: ScriptType) -> Result<u32, Error> {
-        let key = MapKey::LastIndex(script_type).as_map_key();
+    fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
+        let key = MapKey::LastIndex(keychain).as_map_key();
         self.update_and_fetch(key, |prev| {
             let new = match prev {
                 Some(b) => {
@@ -333,17 +377,19 @@ impl Database for Tree {
 
             Some(new.to_be_bytes().to_vec())
         })?
-        .map_or(Ok(0), |b| -> Result<_, Error> {
-            let array: [u8; 4] = b
-                .as_ref()
-                .try_into()
-                .map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
-            let val = u32::from_be_bytes(array);
-            Ok(val)
-        })
+        .map_or(Ok(0), ivec_to_u32)
     }
 }
 
+fn ivec_to_u32(b: sled::IVec) -> Result<u32, Error> {
+    let array: [u8; 4] = b
+        .as_ref()
+        .try_into()
+        .map_err(|_| Error::InvalidU32Bytes(b.to_vec()))?;
+    let val = u32::from_be_bytes(array);
+    Ok(val)
+}
+
 impl BatchDatabase for Tree {
     type Batch = sled::Batch;
 
@@ -358,18 +404,12 @@ impl BatchDatabase for Tree {
 
 #[cfg(test)]
 mod test {
-    use std::str::FromStr;
+    use lazy_static::lazy_static;
     use std::sync::{Arc, Condvar, Mutex, Once};
     use std::time::{SystemTime, UNIX_EPOCH};
 
     use sled::{Db, Tree};
 
-    use bitcoin::consensus::encode::deserialize;
-    use bitcoin::hashes::hex::*;
-    use bitcoin::*;
-
-    use crate::database::*;
-
     static mut COUNT: usize = 0;
 
     lazy_static! {
@@ -410,191 +450,86 @@ mod test {
 
     #[test]
     fn test_script_pubkey() {
-        let mut tree = get_tree();
-
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let path = DerivationPath::from_str("m/0/1/2/3").unwrap();
-        let script_type = ScriptType::External;
-
-        tree.set_script_pubkey(&script, script_type, &path).unwrap();
-
-        assert_eq!(
-            tree.get_script_pubkey_from_path(script_type, &path)
-                .unwrap(),
-            Some(script.clone())
-        );
-        assert_eq!(
-            tree.get_path_from_script_pubkey(&script).unwrap(),
-            Some((script_type, path.clone()))
-        );
+        crate::database::test::test_script_pubkey(get_tree());
     }
 
     #[test]
     fn test_batch_script_pubkey() {
-        let mut tree = get_tree();
-        let mut batch = tree.begin_batch();
-
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let path = DerivationPath::from_str("m/0/1/2/3").unwrap();
-        let script_type = ScriptType::External;
-
-        batch
-            .set_script_pubkey(&script, script_type, &path)
-            .unwrap();
-
-        assert_eq!(
-            tree.get_script_pubkey_from_path(script_type, &path)
-                .unwrap(),
-            None
-        );
-        assert_eq!(tree.get_path_from_script_pubkey(&script).unwrap(), None);
-
-        tree.commit_batch(batch).unwrap();
-
-        assert_eq!(
-            tree.get_script_pubkey_from_path(script_type, &path)
-                .unwrap(),
-            Some(script.clone())
-        );
-        assert_eq!(
-            tree.get_path_from_script_pubkey(&script).unwrap(),
-            Some((script_type, path.clone()))
-        );
+        crate::database::test::test_batch_script_pubkey(get_tree());
     }
 
     #[test]
     fn test_iter_script_pubkey() {
-        let mut tree = get_tree();
-
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let path = DerivationPath::from_str("m/0/1/2/3").unwrap();
-        let script_type = ScriptType::External;
-
-        tree.set_script_pubkey(&script, script_type, &path).unwrap();
-
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 1);
+        crate::database::test::test_iter_script_pubkey(get_tree());
     }
 
     #[test]
     fn test_del_script_pubkey() {
-        let mut tree = get_tree();
-
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let path = DerivationPath::from_str("m/0/1/2/3").unwrap();
-        let script_type = ScriptType::External;
-
-        tree.set_script_pubkey(&script, script_type, &path).unwrap();
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 1);
-
-        tree.del_script_pubkey_from_path(script_type, &path)
-            .unwrap();
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 0);
+        crate::database::test::test_del_script_pubkey(get_tree());
     }
 
     #[test]
     fn test_utxo() {
-        let mut tree = get_tree();
-
-        let outpoint = OutPoint::from_str(
-            "5df6e0e2761359d30a8275058e299fcc0381534545f55cf43e41983f5d4c9456:0",
-        )
-        .unwrap();
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let txout = TxOut {
-            value: 133742,
-            script_pubkey: script,
-        };
-        let utxo = UTXO { txout, outpoint };
-
-        tree.set_utxo(&utxo).unwrap();
-
-        assert_eq!(tree.get_utxo(&outpoint).unwrap(), Some(utxo));
+        crate::database::test::test_utxo(get_tree());
     }
 
     #[test]
     fn test_raw_tx() {
-        let mut tree = get_tree();
-
-        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
-        let tx: Transaction = deserialize(&hex_tx).unwrap();
-
-        tree.set_raw_tx(&tx).unwrap();
-
-        let txid = tx.txid();
-
-        assert_eq!(tree.get_raw_tx(&txid).unwrap(), Some(tx));
+        crate::database::test::test_raw_tx(get_tree());
     }
 
     #[test]
     fn test_tx() {
-        let mut tree = get_tree();
-
-        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
-        let tx: Transaction = deserialize(&hex_tx).unwrap();
-        let txid = tx.txid();
-        let mut tx_details = TransactionDetails {
-            transaction: Some(tx),
-            txid,
-            timestamp: 123456,
-            received: 1337,
-            sent: 420420,
-            height: Some(1000),
-        };
-
-        tree.set_tx(&tx_details).unwrap();
-
-        // get with raw tx too
-        assert_eq!(
-            tree.get_tx(&tx_details.txid, true).unwrap(),
-            Some(tx_details.clone())
-        );
-        // get only raw_tx
-        assert_eq!(
-            tree.get_raw_tx(&tx_details.txid).unwrap(),
-            tx_details.transaction
-        );
-
-        // now get without raw_tx
-        tx_details.transaction = None;
-        assert_eq!(
-            tree.get_tx(&tx_details.txid, false).unwrap(),
-            Some(tx_details)
-        );
+        crate::database::test::test_tx(get_tree());
     }
 
     #[test]
     fn test_last_index() {
-        let mut tree = get_tree();
-
-        tree.set_last_index(ScriptType::External, 1337).unwrap();
-
-        assert_eq!(
-            tree.get_last_index(ScriptType::External).unwrap(),
-            Some(1337)
-        );
-        assert_eq!(tree.get_last_index(ScriptType::Internal).unwrap(), None);
-
-        let res = tree.increment_last_index(ScriptType::External).unwrap();
-        assert_eq!(res, 1338);
-        let res = tree.increment_last_index(ScriptType::Internal).unwrap();
-        assert_eq!(res, 0);
-
-        assert_eq!(
-            tree.get_last_index(ScriptType::External).unwrap(),
-            Some(1338)
-        );
-        assert_eq!(tree.get_last_index(ScriptType::Internal).unwrap(), Some(0));
+        crate::database::test::test_last_index(get_tree());
     }
 
-    // TODO: more tests...
+    #[test]
+    fn test_sync_time() {
+        crate::database::test::test_sync_time(get_tree());
+    }
+
+    #[test]
+    fn test_iter_raw_txs() {
+        crate::database::test::test_iter_raw_txs(get_tree());
+    }
+
+    #[test]
+    fn test_del_path_from_script_pubkey() {
+        crate::database::test::test_del_path_from_script_pubkey(get_tree());
+    }
+
+    #[test]
+    fn test_iter_script_pubkeys() {
+        crate::database::test::test_iter_script_pubkeys(get_tree());
+    }
+
+    #[test]
+    fn test_del_utxo() {
+        crate::database::test::test_del_utxo(get_tree());
+    }
+
+    #[test]
+    fn test_del_raw_tx() {
+        crate::database::test::test_del_raw_tx(get_tree());
+    }
+
+    #[test]
+    fn test_del_tx() {
+        crate::database::test::test_del_tx(get_tree());
+    }
+
+    #[test]
+    fn test_del_last_index() {
+        crate::database::test::test_del_last_index(get_tree());
+    }
+
+    #[test]
+    fn test_check_descriptor_checksum() {
+        crate::database::test::test_check_descriptor_checksum(get_tree());
+    }
 }

src/database/memory.rs

@@ -1,12 +1,28 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! In-memory ephemeral database
+//!
+//! This module defines an in-memory database type called [`MemoryDatabase`] that is based on a
+//! [`BTreeMap`].
+
+use std::any::Any;
 use std::collections::BTreeMap;
 use std::ops::Bound::{Excluded, Included};
 
 use bitcoin::consensus::encode::{deserialize, serialize};
 use bitcoin::hash_types::Txid;
-use bitcoin::util::bip32::{ChildNumber, DerivationPath};
-use bitcoin::{OutPoint, Script, Transaction};
+use bitcoin::{OutPoint, Script, ScriptBuf, Transaction};
 
-use crate::database::{BatchDatabase, BatchOperations, Database};
+use crate::database::{BatchDatabase, BatchOperations, ConfigurableDatabase, Database, SyncTime};
 use crate::error::Error;
 use crate::types::*;
 
@@ -17,19 +33,21 @@ use crate::types::*;
 // transactions         t<txid> -> tx details
 // deriv indexes        c{i,e} -> u32
 // descriptor checksum  d{i,e} -> vec<u8>
+// last sync time       l -> { height, timestamp }
 
 pub(crate) enum MapKey<'a> {
-    Path((Option<ScriptType>, Option<&'a DerivationPath>)),
+    Path((Option<KeychainKind>, Option<u32>)),
     Script(Option<&'a Script>),
-    UTXO(Option<&'a OutPoint>),
+    Utxo(Option<&'a OutPoint>),
     RawTx(Option<&'a Txid>),
     Transaction(Option<&'a Txid>),
-    LastIndex(ScriptType),
-    DescriptorChecksum(ScriptType),
+    LastIndex(KeychainKind),
+    SyncTime,
+    DescriptorChecksum(KeychainKind),
 }
 
 impl MapKey<'_> {
-    pub fn as_prefix(&self) -> Vec<u8> {
+    fn as_prefix(&self) -> Vec<u8> {
         match self {
             MapKey::Path((st, _)) => {
                 let mut v = b"p".to_vec();
@@ -39,25 +57,20 @@ impl MapKey<'_> {
                 v
             }
             MapKey::Script(_) => b"s".to_vec(),
-            MapKey::UTXO(_) => b"u".to_vec(),
+            MapKey::Utxo(_) => b"u".to_vec(),
             MapKey::RawTx(_) => b"r".to_vec(),
             MapKey::Transaction(_) => b"t".to_vec(),
             MapKey::LastIndex(st) => [b"c", st.as_ref()].concat(),
+            MapKey::SyncTime => b"l".to_vec(),
             MapKey::DescriptorChecksum(st) => [b"d", st.as_ref()].concat(),
         }
     }
 
     fn serialize_content(&self) -> Vec<u8> {
         match self {
-            MapKey::Path((_, Some(path))) => {
-                let mut res = vec![];
-                for val in *path {
-                    res.extend(&u32::from(*val).to_be_bytes());
-                }
-                res
-            }
+            MapKey::Path((_, Some(child))) => child.to_be_bytes().to_vec(),
             MapKey::Script(Some(s)) => serialize(*s),
-            MapKey::UTXO(Some(s)) => serialize(*s),
+            MapKey::Utxo(Some(s)) => serialize(*s),
             MapKey::RawTx(Some(s)) => serialize(*s),
             MapKey::Transaction(Some(s)) => serialize(*s),
             _ => vec![],
@@ -72,24 +85,41 @@ impl MapKey<'_> {
     }
 }
 
-fn after(key: &Vec<u8>) -> Vec<u8> {
-    let mut key = key.clone();
-    let len = key.len();
-    if len > 0 {
-        // TODO i guess it could break if the value is 0xFF, but it's fine for now
-        key[len - 1] += 1;
+fn after(key: &[u8]) -> Vec<u8> {
+    let mut key = key.to_owned();
+    let mut idx = key.len();
+    while idx > 0 {
+        if key[idx - 1] == 0xFF {
+            idx -= 1;
+            continue;
+        } else {
+            key[idx - 1] += 1;
+            break;
+        }
     }
 
     key
 }
 
-#[derive(Debug)]
+/// In-memory ephemeral database
+///
+/// This database can be used as a temporary storage for wallets that are not kept permanently on
+/// a device, or on platforms that don't provide a filesystem, like `wasm32`.
+///
+/// Once it's dropped its content will be lost.
+///
+/// If you are looking for a permanent storage solution, you can try with the default key-value
+/// database called [`sled`]. See the [`database`] module documentation for more details.
+///
+/// [`database`]: crate::database
+#[derive(Debug, Default)]
 pub struct MemoryDatabase {
-    map: BTreeMap<Vec<u8>, Box<dyn std::any::Any>>,
+    map: BTreeMap<Vec<u8>, Box<dyn Any + Send + Sync>>,
     deleted_keys: Vec<Vec<u8>>,
 }
 
 impl MemoryDatabase {
+    /// Create a new empty database
     pub fn new() -> Self {
         MemoryDatabase {
             map: BTreeMap::new(),
@@ -99,29 +129,31 @@ impl MemoryDatabase {
 }
 
 impl BatchOperations for MemoryDatabase {
-    fn set_script_pubkey<P: AsRef<[ChildNumber]>>(
+    fn set_script_pubkey(
         &mut self,
         script: &Script,
-        script_type: ScriptType,
-        path: &P,
+        keychain: KeychainKind,
+        path: u32,
     ) -> Result<(), Error> {
-        let deriv_path = DerivationPath::from(path.as_ref());
-        let key = MapKey::Path((Some(script_type), Some(&deriv_path))).as_map_key();
-        self.map.insert(key, Box::new(script.clone()));
+        let key = MapKey::Path((Some(keychain), Some(path))).as_map_key();
+        self.map.insert(key, Box::new(ScriptBuf::from(script)));
 
         let key = MapKey::Script(Some(script)).as_map_key();
         let value = json!({
-            "t": script_type,
-            "p": deriv_path,
+            "t": keychain,
+            "p": path,
         });
         self.map.insert(key, Box::new(value));
 
         Ok(())
     }
 
-    fn set_utxo(&mut self, utxo: &UTXO) -> Result<(), Error> {
-        let key = MapKey::UTXO(Some(&utxo.outpoint)).as_map_key();
-        self.map.insert(key, Box::new(utxo.txout.clone()));
+    fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error> {
+        let key = MapKey::Utxo(Some(&utxo.outpoint)).as_map_key();
+        self.map.insert(
+            key,
+            Box::new((utxo.txout.clone(), utxo.keychain, utxo.is_spent)),
+        );
 
         Ok(())
     }
@@ -147,20 +179,25 @@ impl BatchOperations for MemoryDatabase {
 
         Ok(())
     }
-    fn set_last_index(&mut self, script_type: ScriptType, value: u32) -> Result<(), Error> {
-        let key = MapKey::LastIndex(script_type).as_map_key();
+    fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error> {
+        let key = MapKey::LastIndex(keychain).as_map_key();
         self.map.insert(key, Box::new(value));
 
         Ok(())
     }
+    fn set_sync_time(&mut self, data: SyncTime) -> Result<(), Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        self.map.insert(key, Box::new(data));
 
-    fn del_script_pubkey_from_path<P: AsRef<[ChildNumber]>>(
+        Ok(())
+    }
+
+    fn del_script_pubkey_from_path(
         &mut self,
-        script_type: ScriptType,
-        path: &P,
-    ) -> Result<Option<Script>, Error> {
-        let deriv_path = DerivationPath::from(path.as_ref());
-        let key = MapKey::Path((Some(script_type), Some(&deriv_path))).as_map_key();
+        keychain: KeychainKind,
+        path: u32,
+    ) -> Result<Option<ScriptBuf>, Error> {
+        let key = MapKey::Path((Some(keychain), Some(path))).as_map_key();
         let res = self.map.remove(&key);
         self.deleted_keys.push(key);
 
@@ -169,7 +206,7 @@ impl BatchOperations for MemoryDatabase {
     fn del_path_from_script_pubkey(
         &mut self,
         script: &Script,
-    ) -> Result<Option<(ScriptType, DerivationPath)>, Error> {
+    ) -> Result<Option<(KeychainKind, u32)>, Error> {
         let key = MapKey::Script(Some(script)).as_map_key();
         let res = self.map.remove(&key);
         self.deleted_keys.push(key);
@@ -185,18 +222,20 @@ impl BatchOperations for MemoryDatabase {
             }
         }
     }
-    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
-        let key = MapKey::UTXO(Some(outpoint)).as_map_key();
+    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+        let key = MapKey::Utxo(Some(outpoint)).as_map_key();
         let res = self.map.remove(&key);
         self.deleted_keys.push(key);
 
         match res {
             None => Ok(None),
             Some(b) => {
-                let txout = b.downcast_ref().cloned().unwrap();
-                Ok(Some(UTXO {
-                    outpoint: outpoint.clone(),
+                let (txout, keychain, is_spent) = b.downcast_ref().cloned().unwrap();
+                Ok(Some(LocalUtxo {
+                    outpoint: *outpoint,
                     txout,
+                    keychain,
+                    is_spent,
                 }))
             }
         }
@@ -233,8 +272,8 @@ impl BatchOperations for MemoryDatabase {
             }
         }
     }
-    fn del_last_index(&mut self, script_type: ScriptType) -> Result<Option<u32>, Error> {
-        let key = MapKey::LastIndex(script_type).as_map_key();
+    fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
+        let key = MapKey::LastIndex(keychain).as_map_key();
         let res = self.map.remove(&key);
         self.deleted_keys.push(key);
 
@@ -243,15 +282,22 @@ impl BatchOperations for MemoryDatabase {
             Some(b) => Ok(Some(*b.downcast_ref().unwrap())),
         }
     }
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        let res = self.map.remove(&key);
+        self.deleted_keys.push(key);
+
+        Ok(res.map(|b| b.downcast_ref().cloned().unwrap()))
+    }
 }
 
 impl Database for MemoryDatabase {
     fn check_descriptor_checksum<B: AsRef<[u8]>>(
         &mut self,
-        script_type: ScriptType,
+        keychain: KeychainKind,
         bytes: B,
     ) -> Result<(), Error> {
-        let key = MapKey::DescriptorChecksum(script_type).as_map_key();
+        let key = MapKey::DescriptorChecksum(keychain).as_map_key();
 
         let prev = self
             .map
@@ -269,22 +315,27 @@ impl Database for MemoryDatabase {
         }
     }
 
-    fn iter_script_pubkeys(&self, script_type: Option<ScriptType>) -> Result<Vec<Script>, Error> {
-        let key = MapKey::Path((script_type, None)).as_map_key();
+    fn iter_script_pubkeys(&self, keychain: Option<KeychainKind>) -> Result<Vec<ScriptBuf>, Error> {
+        let key = MapKey::Path((keychain, None)).as_map_key();
         self.map
             .range::<Vec<u8>, _>((Included(&key), Excluded(&after(&key))))
             .map(|(_, v)| Ok(v.downcast_ref().cloned().unwrap()))
             .collect()
     }
 
-    fn iter_utxos(&self) -> Result<Vec<UTXO>, Error> {
-        let key = MapKey::UTXO(None).as_map_key();
+    fn iter_utxos(&self) -> Result<Vec<LocalUtxo>, Error> {
+        let key = MapKey::Utxo(None).as_map_key();
         self.map
             .range::<Vec<u8>, _>((Included(&key), Excluded(&after(&key))))
             .map(|(k, v)| {
                 let outpoint = deserialize(&k[1..]).unwrap();
-                let txout = v.downcast_ref().cloned().unwrap();
-                Ok(UTXO { outpoint, txout })
+                let (txout, keychain, is_spent) = v.downcast_ref().cloned().unwrap();
+                Ok(LocalUtxo {
+                    outpoint,
+                    txout,
+                    keychain,
+                    is_spent,
+                })
             })
             .collect()
     }
@@ -313,13 +364,12 @@ impl Database for MemoryDatabase {
             .collect()
     }
 
-    fn get_script_pubkey_from_path<P: AsRef<[ChildNumber]>>(
+    fn get_script_pubkey_from_path(
         &self,
-        script_type: ScriptType,
-        path: &P,
-    ) -> Result<Option<Script>, Error> {
-        let deriv_path = DerivationPath::from(path.as_ref());
-        let key = MapKey::Path((Some(script_type), Some(&deriv_path))).as_map_key();
+        keychain: KeychainKind,
+        path: u32,
+    ) -> Result<Option<ScriptBuf>, Error> {
+        let key = MapKey::Path((Some(keychain), Some(path))).as_map_key();
         Ok(self
             .map
             .get(&key)
@@ -329,7 +379,7 @@ impl Database for MemoryDatabase {
     fn get_path_from_script_pubkey(
         &self,
         script: &Script,
-    ) -> Result<Option<(ScriptType, DerivationPath)>, Error> {
+    ) -> Result<Option<(KeychainKind, u32)>, Error> {
         let key = MapKey::Script(Some(script)).as_map_key();
         Ok(self.map.get(&key).map(|b| {
             let mut val: serde_json::Value = b.downcast_ref().cloned().unwrap();
@@ -340,13 +390,15 @@ impl Database for MemoryDatabase {
         }))
     }
 
-    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error> {
-        let key = MapKey::UTXO(Some(outpoint)).as_map_key();
+    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error> {
+        let key = MapKey::Utxo(Some(outpoint)).as_map_key();
         Ok(self.map.get(&key).map(|b| {
-            let txout = b.downcast_ref().cloned().unwrap();
-            UTXO {
-                outpoint: outpoint.clone(),
+            let (txout, keychain, is_spent) = b.downcast_ref().cloned().unwrap();
+            LocalUtxo {
+                outpoint: *outpoint,
                 txout,
+                keychain,
+                is_spent,
             }
         }))
     }
@@ -364,26 +416,34 @@ impl Database for MemoryDatabase {
         Ok(self.map.get(&key).map(|b| {
             let mut txdetails: TransactionDetails = b.downcast_ref().cloned().unwrap();
             if include_raw {
-                txdetails.transaction = self.get_raw_tx(&txid).unwrap();
+                txdetails.transaction = self.get_raw_tx(txid).unwrap();
             }
 
             txdetails
         }))
     }
 
-    fn get_last_index(&self, script_type: ScriptType) -> Result<Option<u32>, Error> {
-        let key = MapKey::LastIndex(script_type).as_map_key();
+    fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
+        let key = MapKey::LastIndex(keychain).as_map_key();
         Ok(self.map.get(&key).map(|b| *b.downcast_ref().unwrap()))
     }
 
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
+        let key = MapKey::SyncTime.as_map_key();
+        Ok(self
+            .map
+            .get(&key)
+            .map(|b| b.downcast_ref().cloned().unwrap()))
+    }
+
     // inserts 0 if not present
-    fn increment_last_index(&mut self, script_type: ScriptType) -> Result<u32, Error> {
-        let key = MapKey::LastIndex(script_type).as_map_key();
+    fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
+        let key = MapKey::LastIndex(keychain).as_map_key();
         let value = self
             .map
-            .entry(key.clone())
+            .entry(key)
             .and_modify(|x| *x.downcast_mut::<u32>().unwrap() += 1)
-            .or_insert(Box::<u32>::new(0))
+            .or_insert_with(|| Box::<u32>::new(0))
             .downcast_mut()
             .unwrap();
 
@@ -399,26 +459,145 @@ impl BatchDatabase for MemoryDatabase {
     }
 
     fn commit_batch(&mut self, mut batch: Self::Batch) -> Result<(), Error> {
-        for key in batch.deleted_keys {
-            self.map.remove(&key);
+        for key in batch.deleted_keys.iter() {
+            self.map.remove(key);
+        }
+        self.map.append(&mut batch.map);
+        Ok(())
+    }
+}
+
+impl ConfigurableDatabase for MemoryDatabase {
+    type Config = ();
+
+    fn from_config(_config: &Self::Config) -> Result<Self, Error> {
+        Ok(MemoryDatabase::default())
+    }
+}
+
+#[macro_export]
+#[doc(hidden)]
+/// Artificially insert a tx in the database, as if we had found it with a `sync`. This is a hidden
+/// macro and not a `[cfg(test)]` function so it can be called within the context of doctests which
+/// don't have `test` set.
+macro_rules! populate_test_db {
+    ($db:expr, $tx_meta:expr, $current_height:expr$(,)?) => {{
+        $crate::populate_test_db!($db, $tx_meta, $current_height, (@coinbase false))
+    }};
+    ($db:expr, $tx_meta:expr, $current_height:expr, (@coinbase $is_coinbase:expr)$(,)?) => {{
+        use $crate::database::SyncTime;
+        use $crate::database::{BatchOperations, Database};
+        let mut db = $db;
+        let tx_meta = $tx_meta;
+        let current_height: Option<u32> = $current_height;
+        let mut input = vec![$crate::bitcoin::TxIn::default()];
+        if !$is_coinbase {
+            input[0].previous_output.vout = 0;
+        }
+        let tx = $crate::bitcoin::Transaction {
+            version: 1,
+            lock_time: bitcoin::absolute::LockTime::ZERO,
+            input,
+            output: tx_meta
+                .output
+                .iter()
+                .map(|out_meta| $crate::bitcoin::TxOut {
+                    value: out_meta.value,
+                    script_pubkey: $crate::bitcoin::Address::from_str(&out_meta.to_address)
+                        .unwrap()
+                        .assume_checked()
+                        .script_pubkey(),
+                })
+                .collect(),
+        };
+
+        let txid = tx.txid();
+        // Set Confirmation time only if current height is provided.
+        // panics if `tx_meta.min_confirmation` is Some, and current_height is None.
+        let confirmation_time = tx_meta
+            .min_confirmations
+            .and_then(|v| if v == 0 { None } else { Some(v) })
+            .map(|conf| $crate::BlockTime {
+                height: current_height.expect("Current height is needed for testing transaction with min-confirmation values").checked_sub(conf as u32).unwrap() + 1,
+                timestamp: 0,
+            });
+
+        // Set the database sync_time.
+        // Check if the current_height is less than already known sync height, apply the max
+        // If any of them is None, the other will be applied instead.
+        // If both are None, this will not be set.
+        if let Some(height) = db.get_sync_time().unwrap()
+                                .map(|sync_time| sync_time.block_time.height)
+                                .max(current_height) {
+            let sync_time = SyncTime {
+                block_time: BlockTime {
+                    height,
+                    timestamp: 0
+                }
+            };
+            db.set_sync_time(sync_time).unwrap();
         }
 
-        Ok(self.map.append(&mut batch.map))
-    }
+        let tx_details = $crate::TransactionDetails {
+            transaction: Some(tx.clone()),
+            txid,
+            fee: Some(0),
+            received: 0,
+            sent: 0,
+            confirmation_time,
+        };
+
+        db.set_tx(&tx_details).unwrap();
+        for (vout, out) in tx.output.iter().enumerate() {
+            db.set_utxo(&$crate::LocalUtxo {
+                txout: out.clone(),
+                outpoint: $crate::bitcoin::OutPoint {
+                    txid,
+                    vout: vout as u32,
+                },
+                keychain: $crate::KeychainKind::External,
+                is_spent: false,
+            })
+            .unwrap();
+        }
+
+        txid
+    }};
+}
+
+#[macro_export]
+#[doc(hidden)]
+/// Macro for getting a wallet for use in a doctest
+macro_rules! doctest_wallet {
+    () => {{
+        use $crate::bitcoin::Network;
+        use $crate::database::MemoryDatabase;
+        use $crate::testutils;
+        let descriptor = "wpkh(cVpPVruEDdmutPzisEsYvtST1usBR3ntr8pXSyt6D2YYqXRyPcFW)";
+        let descriptors = testutils!(@descriptors (descriptor) (descriptor));
+
+        let mut db = MemoryDatabase::new();
+        let txid = populate_test_db!(
+            &mut db,
+            testutils! {
+                @tx ( (@external descriptors, 0) => 500_000 ) (@confirmations 1)
+            },
+            Some(100),
+        );
+
+        $crate::Wallet::new(
+            &descriptors.0,
+            descriptors.1.as_ref(),
+            Network::Regtest,
+            db
+        )
+        .unwrap()
+    }}
 }
 
 #[cfg(test)]
 mod test {
-    use std::str::FromStr;
-    use std::sync::{Arc, Condvar, Mutex, Once};
-    use std::time::{SystemTime, UNIX_EPOCH};
-
-    use bitcoin::consensus::encode::deserialize;
-    use bitcoin::hashes::hex::*;
-    use bitcoin::*;
-
-    use super::*;
-    use crate::database::*;
+    use super::MemoryDatabase;
 
     fn get_tree() -> MemoryDatabase {
         MemoryDatabase::new()
@@ -426,215 +605,86 @@ mod test {
 
     #[test]
     fn test_script_pubkey() {
-        let mut tree = get_tree();
-
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let path = DerivationPath::from_str("m/0/1/2/3").unwrap();
-        let script_type = ScriptType::External;
-
-        tree.set_script_pubkey(&script, script_type, &path).unwrap();
-
-        assert_eq!(
-            tree.get_script_pubkey_from_path(script_type, &path)
-                .unwrap(),
-            Some(script.clone())
-        );
-        assert_eq!(
-            tree.get_path_from_script_pubkey(&script).unwrap(),
-            Some((script_type, path.clone()))
-        );
+        crate::database::test::test_script_pubkey(get_tree());
     }
 
     #[test]
     fn test_batch_script_pubkey() {
-        let mut tree = get_tree();
-        let mut batch = tree.begin_batch();
-
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let path = DerivationPath::from_str("m/0/1/2/3").unwrap();
-        let script_type = ScriptType::External;
-
-        batch
-            .set_script_pubkey(&script, script_type, &path)
-            .unwrap();
-
-        assert_eq!(
-            tree.get_script_pubkey_from_path(script_type, &path)
-                .unwrap(),
-            None
-        );
-        assert_eq!(tree.get_path_from_script_pubkey(&script).unwrap(), None);
-
-        tree.commit_batch(batch).unwrap();
-
-        assert_eq!(
-            tree.get_script_pubkey_from_path(script_type, &path)
-                .unwrap(),
-            Some(script.clone())
-        );
-        assert_eq!(
-            tree.get_path_from_script_pubkey(&script).unwrap(),
-            Some((script_type, path.clone()))
-        );
+        crate::database::test::test_batch_script_pubkey(get_tree());
     }
 
     #[test]
     fn test_iter_script_pubkey() {
-        let mut tree = get_tree();
-
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let path = DerivationPath::from_str("m/0/1/2/3").unwrap();
-        let script_type = ScriptType::External;
-
-        tree.set_script_pubkey(&script, script_type, &path).unwrap();
-
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 1);
+        crate::database::test::test_iter_script_pubkey(get_tree());
     }
 
     #[test]
     fn test_del_script_pubkey() {
-        let mut tree = get_tree();
-
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let path = DerivationPath::from_str("m/0/1/2/3").unwrap();
-        let script_type = ScriptType::External;
-
-        tree.set_script_pubkey(&script, script_type, &path).unwrap();
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 1);
-
-        tree.del_script_pubkey_from_path(script_type, &path)
-            .unwrap();
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 0);
-    }
-
-    #[test]
-    fn test_del_script_pubkey_batch() {
-        let mut tree = get_tree();
-
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let path = DerivationPath::from_str("m/0/1/2/3").unwrap();
-        let script_type = ScriptType::External;
-
-        tree.set_script_pubkey(&script, script_type, &path).unwrap();
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 1);
-
-        let mut batch = tree.begin_batch();
-        batch
-            .del_script_pubkey_from_path(script_type, &path)
-            .unwrap();
-
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 1);
-
-        tree.commit_batch(batch);
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 0);
+        crate::database::test::test_del_script_pubkey(get_tree());
     }
 
     #[test]
     fn test_utxo() {
-        let mut tree = get_tree();
-
-        let outpoint = OutPoint::from_str(
-            "5df6e0e2761359d30a8275058e299fcc0381534545f55cf43e41983f5d4c9456:0",
-        )
-        .unwrap();
-        let script = Script::from(
-            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
-        );
-        let txout = TxOut {
-            value: 133742,
-            script_pubkey: script,
-        };
-        let utxo = UTXO { txout, outpoint };
-
-        tree.set_utxo(&utxo).unwrap();
-
-        assert_eq!(tree.get_utxo(&outpoint).unwrap(), Some(utxo));
+        crate::database::test::test_utxo(get_tree());
     }
 
     #[test]
     fn test_raw_tx() {
-        let mut tree = get_tree();
-
-        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
-        let tx: Transaction = deserialize(&hex_tx).unwrap();
-
-        tree.set_raw_tx(&tx).unwrap();
-
-        let txid = tx.txid();
-
-        assert_eq!(tree.get_raw_tx(&txid).unwrap(), Some(tx));
+        crate::database::test::test_raw_tx(get_tree());
     }
 
     #[test]
     fn test_tx() {
-        let mut tree = get_tree();
-
-        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
-        let tx: Transaction = deserialize(&hex_tx).unwrap();
-        let txid = tx.txid();
-        let mut tx_details = TransactionDetails {
-            transaction: Some(tx),
-            txid,
-            timestamp: 123456,
-            received: 1337,
-            sent: 420420,
-            height: Some(1000),
-        };
-
-        tree.set_tx(&tx_details).unwrap();
-
-        // get with raw tx too
-        assert_eq!(
-            tree.get_tx(&tx_details.txid, true).unwrap(),
-            Some(tx_details.clone())
-        );
-        // get only raw_tx
-        assert_eq!(
-            tree.get_raw_tx(&tx_details.txid).unwrap(),
-            tx_details.transaction
-        );
-
-        // now get without raw_tx
-        tx_details.transaction = None;
-        assert_eq!(
-            tree.get_tx(&tx_details.txid, false).unwrap(),
-            Some(tx_details)
-        );
+        crate::database::test::test_tx(get_tree());
     }
 
     #[test]
     fn test_last_index() {
-        let mut tree = get_tree();
-
-        tree.set_last_index(ScriptType::External, 1337).unwrap();
-
-        assert_eq!(
-            tree.get_last_index(ScriptType::External).unwrap(),
-            Some(1337)
-        );
-        assert_eq!(tree.get_last_index(ScriptType::Internal).unwrap(), None);
-
-        let res = tree.increment_last_index(ScriptType::External).unwrap();
-        assert_eq!(res, 1338);
-        let res = tree.increment_last_index(ScriptType::Internal).unwrap();
-        assert_eq!(res, 0);
-
-        assert_eq!(
-            tree.get_last_index(ScriptType::External).unwrap(),
-            Some(1338)
-        );
-        assert_eq!(tree.get_last_index(ScriptType::Internal).unwrap(), Some(0));
+        crate::database::test::test_last_index(get_tree());
     }
 
-    // TODO: more tests...
+    #[test]
+    fn test_sync_time() {
+        crate::database::test::test_sync_time(get_tree());
+    }
+
+    #[test]
+    fn test_iter_raw_txs() {
+        crate::database::test::test_iter_raw_txs(get_tree());
+    }
+
+    #[test]
+    fn test_del_path_from_script_pubkey() {
+        crate::database::test::test_del_path_from_script_pubkey(get_tree());
+    }
+
+    #[test]
+    fn test_iter_script_pubkeys() {
+        crate::database::test::test_iter_script_pubkeys(get_tree());
+    }
+
+    #[test]
+    fn test_del_utxo() {
+        crate::database::test::test_del_utxo(get_tree());
+    }
+
+    #[test]
+    fn test_del_raw_tx() {
+        crate::database::test::test_del_raw_tx(get_tree());
+    }
+
+    #[test]
+    fn test_del_tx() {
+        crate::database::test::test_del_tx(get_tree());
+    }
+
+    #[test]
+    fn test_del_last_index() {
+        crate::database::test::test_del_last_index(get_tree());
+    }
+
+    #[test]
+    fn test_check_descriptor_checksum() {
+        crate::database::test::test_check_descriptor_checksum(get_tree());
+    }
 }

src/database/mod.rs

@@ -1,105 +1,209 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Database types
+//!
+//! This module provides the implementation of some defaults database types, along with traits that
+//! can be implemented externally to let [`Wallet`]s use customized databases.
+//!
+//! It's important to note that the databases defined here only contains "blockchain-related" data.
+//! They can be seen more as a cache than a critical piece of storage that contains secrets and
+//! keys.
+//!
+//! The currently recommended database is [`sled`], which is a pretty simple key-value embedded
+//! database written in Rust. If the `key-value-db` feature is enabled (which by default is),
+//! this library automatically implements all the required traits for [`sled::Tree`].
+//!
+//! [`Wallet`]: crate::wallet::Wallet
+
+use serde::{Deserialize, Serialize};
+
 use bitcoin::hash_types::Txid;
-use bitcoin::util::bip32::{ChildNumber, DerivationPath};
-use bitcoin::{OutPoint, Script, Transaction, TxOut};
+use bitcoin::{OutPoint, Script, ScriptBuf, Transaction, TxOut};
 
 use crate::error::Error;
 use crate::types::*;
 
-#[cfg(any(feature = "key-value-db", feature = "default"))]
-pub mod keyvalue;
-pub mod memory;
+pub mod any;
+pub use any::{AnyDatabase, AnyDatabaseConfig};
 
+#[cfg(feature = "key-value-db")]
+pub(crate) mod keyvalue;
+
+#[cfg(feature = "sqlite")]
+pub(crate) mod sqlite;
+#[cfg(feature = "sqlite")]
+pub use sqlite::SqliteDatabase;
+
+pub mod memory;
+pub use memory::MemoryDatabase;
+
+/// Blockchain state at the time of syncing
+///
+/// Contains only the block time and height at the moment
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub struct SyncTime {
+    /// Block timestamp and height at the time of sync
+    pub block_time: BlockTime,
+}
+
+/// Trait for operations that can be batched
+///
+/// This trait defines the list of operations that must be implemented on the [`Database`] type and
+/// the [`BatchDatabase::Batch`] type.
 pub trait BatchOperations {
-    fn set_script_pubkey<P: AsRef<[ChildNumber]>>(
+    /// Store a script_pubkey along with its keychain and child number.
+    fn set_script_pubkey(
         &mut self,
         script: &Script,
-        script_type: ScriptType,
-        path: &P,
+        keychain: KeychainKind,
+        child: u32,
     ) -> Result<(), Error>;
-    fn set_utxo(&mut self, utxo: &UTXO) -> Result<(), Error>;
+    /// Store a [`LocalUtxo`]
+    fn set_utxo(&mut self, utxo: &LocalUtxo) -> Result<(), Error>;
+    /// Store a raw transaction
     fn set_raw_tx(&mut self, transaction: &Transaction) -> Result<(), Error>;
+    /// Store the metadata of a transaction
     fn set_tx(&mut self, transaction: &TransactionDetails) -> Result<(), Error>;
-    fn set_last_index(&mut self, script_type: ScriptType, value: u32) -> Result<(), Error>;
+    /// Store the last derivation index for a given keychain.
+    fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error>;
+    /// Store the sync time
+    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error>;
 
-    fn del_script_pubkey_from_path<P: AsRef<[ChildNumber]>>(
+    /// Delete a script_pubkey given the keychain and its child number.
+    fn del_script_pubkey_from_path(
         &mut self,
-        script_type: ScriptType,
-        path: &P,
-    ) -> Result<Option<Script>, Error>;
+        keychain: KeychainKind,
+        child: u32,
+    ) -> Result<Option<ScriptBuf>, Error>;
+    /// Delete the data related to a specific script_pubkey, meaning the keychain and the child
+    /// number.
     fn del_path_from_script_pubkey(
         &mut self,
         script: &Script,
-    ) -> Result<Option<(ScriptType, DerivationPath)>, Error>;
-    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error>;
+    ) -> Result<Option<(KeychainKind, u32)>, Error>;
+    /// Delete a [`LocalUtxo`] given its [`OutPoint`]
+    fn del_utxo(&mut self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error>;
+    /// Delete a raw transaction given its [`Txid`]
     fn del_raw_tx(&mut self, txid: &Txid) -> Result<Option<Transaction>, Error>;
+    /// Delete the metadata of a transaction and optionally the raw transaction itself
     fn del_tx(
         &mut self,
         txid: &Txid,
         include_raw: bool,
     ) -> Result<Option<TransactionDetails>, Error>;
-    fn del_last_index(&mut self, script_type: ScriptType) -> Result<Option<u32>, Error>;
+    /// Delete the last derivation index for a keychain.
+    fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error>;
+    /// Reset the sync time to `None`
+    ///
+    /// Returns the removed value
+    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error>;
 }
 
+/// Trait for reading data from a database
+///
+/// This traits defines the operations that can be used to read data out of a database
 pub trait Database: BatchOperations {
+    /// Read and checks the descriptor checksum for a given keychain.
+    ///
+    /// Should return [`Error::ChecksumMismatch`](crate::error::Error::ChecksumMismatch) if the
+    /// checksum doesn't match. If there's no checksum in the database, simply store it for the
+    /// next time.
     fn check_descriptor_checksum<B: AsRef<[u8]>>(
         &mut self,
-        script_type: ScriptType,
+        keychain: KeychainKind,
         bytes: B,
     ) -> Result<(), Error>;
 
-    fn iter_script_pubkeys(&self, script_type: Option<ScriptType>) -> Result<Vec<Script>, Error>;
-    fn iter_utxos(&self) -> Result<Vec<UTXO>, Error>;
+    /// Return the list of script_pubkeys
+    fn iter_script_pubkeys(&self, keychain: Option<KeychainKind>) -> Result<Vec<ScriptBuf>, Error>;
+    /// Return the list of [`LocalUtxo`]s
+    fn iter_utxos(&self) -> Result<Vec<LocalUtxo>, Error>;
+    /// Return the list of raw transactions
     fn iter_raw_txs(&self) -> Result<Vec<Transaction>, Error>;
+    /// Return the list of transactions metadata
     fn iter_txs(&self, include_raw: bool) -> Result<Vec<TransactionDetails>, Error>;
 
-    fn get_script_pubkey_from_path<P: AsRef<[ChildNumber]>>(
+    /// Fetch a script_pubkey given the child number of a keychain.
+    fn get_script_pubkey_from_path(
         &self,
-        script_type: ScriptType,
-        path: &P,
-    ) -> Result<Option<Script>, Error>;
+        keychain: KeychainKind,
+        child: u32,
+    ) -> Result<Option<ScriptBuf>, Error>;
+    /// Fetch the keychain and child number of a given script_pubkey
     fn get_path_from_script_pubkey(
         &self,
         script: &Script,
-    ) -> Result<Option<(ScriptType, DerivationPath)>, Error>;
-    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<UTXO>, Error>;
+    ) -> Result<Option<(KeychainKind, u32)>, Error>;
+    /// Fetch a [`LocalUtxo`] given its [`OutPoint`]
+    fn get_utxo(&self, outpoint: &OutPoint) -> Result<Option<LocalUtxo>, Error>;
+    /// Fetch a raw transaction given its [`Txid`]
     fn get_raw_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
+    /// Fetch the transaction metadata and optionally also the raw transaction
     fn get_tx(&self, txid: &Txid, include_raw: bool) -> Result<Option<TransactionDetails>, Error>;
-    fn get_last_index(&self, script_type: ScriptType) -> Result<Option<u32>, Error>;
+    /// Return the last derivation index for a keychain.
+    fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error>;
+    /// Return the sync time, if present
+    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error>;
 
-    // inserts 0 if not present
-    fn increment_last_index(&mut self, script_type: ScriptType) -> Result<u32, Error>;
+    /// Increment the last derivation index for a keychain and return it
+    ///
+    /// It should insert and return `0` if not present in the database
+    fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error>;
 }
 
+/// Trait for a database that supports batch operations
+///
+/// This trait defines the methods to start and apply a batch of operations.
 pub trait BatchDatabase: Database {
+    /// Container for the operations
     type Batch: BatchOperations;
 
+    /// Create a new batch container
     fn begin_batch(&self) -> Self::Batch;
+    /// Consume and apply a batch of operations
     fn commit_batch(&mut self, batch: Self::Batch) -> Result<(), Error>;
 }
 
-pub trait DatabaseUtils: Database {
+/// Trait for [`Database`] types that can be created given a configuration
+pub trait ConfigurableDatabase: Database + Sized {
+    /// Type that contains the configuration
+    type Config: std::fmt::Debug;
+
+    /// Create a new instance given a configuration
+    fn from_config(config: &Self::Config) -> Result<Self, Error>;
+}
+
+pub(crate) trait DatabaseUtils: Database {
     fn is_mine(&self, script: &Script) -> Result<bool, Error> {
         self.get_path_from_script_pubkey(script)
             .map(|o| o.is_some())
     }
 
-    fn get_raw_tx_or<F>(&self, txid: &Txid, f: F) -> Result<Option<Transaction>, Error>
+    fn get_raw_tx_or<D>(&self, txid: &Txid, default: D) -> Result<Option<Transaction>, Error>
     where
-        F: FnOnce() -> Result<Option<Transaction>, Error>,
+        D: FnOnce() -> Result<Option<Transaction>, Error>,
     {
         self.get_tx(txid, true)?
-            .map(|t| t.transaction)
-            .flatten()
-            .map_or_else(f, |t| Ok(Some(t)))
+            .and_then(|t| t.transaction)
+            .map_or_else(default, |t| Ok(Some(t)))
     }
 
     fn get_previous_output(&self, outpoint: &OutPoint) -> Result<Option<TxOut>, Error> {
         self.get_raw_tx(&outpoint.txid)?
-            .and_then(|previous_tx| {
+            .map(|previous_tx| {
                 if outpoint.vout as usize >= previous_tx.output.len() {
-                    Some(Err(Error::InvalidOutpoint(outpoint.clone())))
+                    Err(Error::InvalidOutpoint(*outpoint))
                 } else {
-                    Some(Ok(previous_tx.output[outpoint.vout as usize].clone()))
+                    Ok(previous_tx.output[outpoint.vout as usize].clone())
                 }
             })
             .transpose()
@@ -107,3 +211,447 @@ pub trait DatabaseUtils: Database {
 }
 
 impl<T: Database> DatabaseUtils for T {}
+
+#[cfg(test)]
+pub mod test {
+    use bitcoin::consensus::encode::deserialize;
+    use bitcoin::consensus::serialize;
+    use bitcoin::hashes::hex::*;
+    use bitcoin::Witness;
+    use bitcoin::*;
+    use std::str::FromStr;
+
+    use super::*;
+
+    pub fn test_script_pubkey<D: Database>(mut db: D) {
+        let script = ScriptBuf::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let path = 42;
+        let keychain = KeychainKind::External;
+
+        db.set_script_pubkey(&script, keychain, path).unwrap();
+
+        assert_eq!(
+            db.get_script_pubkey_from_path(keychain, path).unwrap(),
+            Some(script.clone())
+        );
+        assert_eq!(
+            db.get_path_from_script_pubkey(&script).unwrap(),
+            Some((keychain, path))
+        );
+    }
+
+    pub fn test_batch_script_pubkey<D: BatchDatabase>(mut db: D) {
+        let mut batch = db.begin_batch();
+
+        let script = ScriptBuf::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let path = 42;
+        let keychain = KeychainKind::External;
+
+        batch.set_script_pubkey(&script, keychain, path).unwrap();
+
+        assert_eq!(
+            db.get_script_pubkey_from_path(keychain, path).unwrap(),
+            None
+        );
+        assert_eq!(db.get_path_from_script_pubkey(&script).unwrap(), None);
+
+        db.commit_batch(batch).unwrap();
+
+        assert_eq!(
+            db.get_script_pubkey_from_path(keychain, path).unwrap(),
+            Some(script.clone())
+        );
+        assert_eq!(
+            db.get_path_from_script_pubkey(&script).unwrap(),
+            Some((keychain, path))
+        );
+    }
+
+    pub fn test_iter_script_pubkey<D: Database>(mut db: D) {
+        let script = ScriptBuf::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let path = 42;
+        let keychain = KeychainKind::External;
+
+        db.set_script_pubkey(&script, keychain, path).unwrap();
+
+        assert_eq!(db.iter_script_pubkeys(None).unwrap().len(), 1);
+    }
+
+    pub fn test_del_script_pubkey<D: Database>(mut db: D) {
+        let script = ScriptBuf::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let path = 42;
+        let keychain = KeychainKind::External;
+
+        db.set_script_pubkey(&script, keychain, path).unwrap();
+        assert_eq!(db.iter_script_pubkeys(None).unwrap().len(), 1);
+
+        db.del_script_pubkey_from_path(keychain, path).unwrap();
+        assert_eq!(db.iter_script_pubkeys(None).unwrap().len(), 0);
+    }
+
+    pub fn test_utxo<D: Database>(mut db: D) {
+        let outpoint = OutPoint::from_str(
+            "5df6e0e2761359d30a8275058e299fcc0381534545f55cf43e41983f5d4c9456:0",
+        )
+        .unwrap();
+        let script = ScriptBuf::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let txout = TxOut {
+            value: 133742,
+            script_pubkey: script,
+        };
+        let utxo = LocalUtxo {
+            txout,
+            outpoint,
+            keychain: KeychainKind::External,
+            is_spent: true,
+        };
+
+        db.set_utxo(&utxo).unwrap();
+        db.set_utxo(&utxo).unwrap();
+        assert_eq!(db.iter_utxos().unwrap().len(), 1);
+        assert_eq!(db.get_utxo(&outpoint).unwrap(), Some(utxo));
+    }
+
+    pub fn test_raw_tx<D: Database>(mut db: D) {
+        let hex_tx = Vec::<u8>::from_hex("02000000000101f58c18a90d7a76b30c7e47d4e817adfdd79a6a589a615ef36e360f913adce2cd0000000000feffffff0210270000000000001600145c9a1816d38db5cbdd4b067b689dc19eb7d930e2cf70aa2b080000001600140f48b63160043047f4f60f7f8f551f80458f693f024730440220413f42b7bc979945489a38f5221e5527d4b8e3aa63eae2099e01945896ad6c10022024ceec492d685c31d8adb64e935a06933877c5ae0e21f32efe029850914c5bad012102361caae96f0e9f3a453d354bb37a5c3244422fb22819bf0166c0647a38de39f21fca2300").unwrap();
+        let mut tx: Transaction = deserialize(&hex_tx).unwrap();
+
+        db.set_raw_tx(&tx).unwrap();
+
+        let txid = tx.txid();
+
+        assert_eq!(db.get_raw_tx(&txid).unwrap(), Some(tx.clone()));
+
+        // mutate transaction's witnesses
+        for tx_in in tx.input.iter_mut() {
+            tx_in.witness = Witness::new();
+        }
+
+        let updated_hex_tx = serialize(&tx);
+
+        // verify that mutation was successful
+        assert_ne!(hex_tx, updated_hex_tx);
+
+        db.set_raw_tx(&tx).unwrap();
+
+        let txid = tx.txid();
+
+        assert_eq!(db.get_raw_tx(&txid).unwrap(), Some(tx));
+    }
+
+    pub fn test_tx<D: Database>(mut db: D) {
+        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
+        let tx: Transaction = deserialize(&hex_tx).unwrap();
+        let txid = tx.txid();
+        let mut tx_details = TransactionDetails {
+            transaction: Some(tx),
+            txid,
+            received: 1337,
+            sent: 420420,
+            fee: Some(140),
+            confirmation_time: Some(BlockTime {
+                timestamp: 123456,
+                height: 1000,
+            }),
+        };
+
+        db.set_tx(&tx_details).unwrap();
+
+        // get with raw tx too
+        assert_eq!(
+            db.get_tx(&tx_details.txid, true).unwrap(),
+            Some(tx_details.clone())
+        );
+        // get only raw_tx
+        assert_eq!(
+            db.get_raw_tx(&tx_details.txid).unwrap(),
+            tx_details.transaction
+        );
+
+        // now get without raw_tx
+        tx_details.transaction = None;
+        assert_eq!(
+            db.get_tx(&tx_details.txid, false).unwrap(),
+            Some(tx_details)
+        );
+    }
+
+    pub fn test_list_transaction<D: Database>(mut db: D) {
+        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
+        let tx: Transaction = deserialize(&hex_tx).unwrap();
+        let txid = tx.txid();
+        let mut tx_details = TransactionDetails {
+            transaction: Some(tx),
+            txid,
+            received: 1337,
+            sent: 420420,
+            fee: Some(140),
+            confirmation_time: Some(BlockTime {
+                timestamp: 123456,
+                height: 1000,
+            }),
+        };
+
+        db.set_tx(&tx_details).unwrap();
+
+        // get raw tx
+        assert_eq!(db.iter_txs(true).unwrap(), vec![tx_details.clone()]);
+
+        // now get without raw tx
+        tx_details.transaction = None;
+
+        // get not raw tx
+        assert_eq!(db.iter_txs(false).unwrap(), vec![tx_details.clone()]);
+    }
+
+    pub fn test_last_index<D: Database>(mut db: D) {
+        db.set_last_index(KeychainKind::External, 1337).unwrap();
+
+        assert_eq!(
+            db.get_last_index(KeychainKind::External).unwrap(),
+            Some(1337)
+        );
+        assert_eq!(db.get_last_index(KeychainKind::Internal).unwrap(), None);
+
+        let res = db.increment_last_index(KeychainKind::External).unwrap();
+        assert_eq!(res, 1338);
+        let res = db.increment_last_index(KeychainKind::Internal).unwrap();
+        assert_eq!(res, 0);
+
+        assert_eq!(
+            db.get_last_index(KeychainKind::External).unwrap(),
+            Some(1338)
+        );
+        assert_eq!(db.get_last_index(KeychainKind::Internal).unwrap(), Some(0));
+    }
+
+    pub fn test_sync_time<D: Database>(mut db: D) {
+        assert!(db.get_sync_time().unwrap().is_none());
+
+        db.set_sync_time(SyncTime {
+            block_time: BlockTime {
+                height: 100,
+                timestamp: 1000,
+            },
+        })
+        .unwrap();
+
+        let extracted = db.get_sync_time().unwrap();
+        assert!(extracted.is_some());
+        assert_eq!(extracted.as_ref().unwrap().block_time.height, 100);
+        assert_eq!(extracted.as_ref().unwrap().block_time.timestamp, 1000);
+
+        db.del_sync_time().unwrap();
+        assert!(db.get_sync_time().unwrap().is_none());
+    }
+
+    pub fn test_iter_raw_txs<D: Database>(mut db: D) {
+        let txs = db.iter_raw_txs().unwrap();
+        assert!(txs.is_empty());
+
+        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
+        let first_tx: Transaction = deserialize(&hex_tx).unwrap();
+
+        let hex_tx = Vec::<u8>::from_hex("02000000000101f58c18a90d7a76b30c7e47d4e817adfdd79a6a589a615ef36e360f913adce2cd0000000000feffffff0210270000000000001600145c9a1816d38db5cbdd4b067b689dc19eb7d930e2cf70aa2b080000001600140f48b63160043047f4f60f7f8f551f80458f693f024730440220413f42b7bc979945489a38f5221e5527d4b8e3aa63eae2099e01945896ad6c10022024ceec492d685c31d8adb64e935a06933877c5ae0e21f32efe029850914c5bad012102361caae96f0e9f3a453d354bb37a5c3244422fb22819bf0166c0647a38de39f21fca2300").unwrap();
+        let second_tx: Transaction = deserialize(&hex_tx).unwrap();
+
+        db.set_raw_tx(&first_tx).unwrap();
+        db.set_raw_tx(&second_tx).unwrap();
+
+        let txs = db.iter_raw_txs().unwrap();
+
+        assert!(txs.contains(&first_tx));
+        assert!(txs.contains(&second_tx));
+        assert_eq!(txs.len(), 2);
+    }
+
+    pub fn test_del_path_from_script_pubkey<D: Database>(mut db: D) {
+        let keychain = KeychainKind::External;
+
+        let script = ScriptBuf::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let path = 42;
+
+        let res = db.del_path_from_script_pubkey(&script).unwrap();
+
+        assert!(res.is_none());
+
+        let _res = db.set_script_pubkey(&script, keychain, path);
+        let (chain, child) = db.del_path_from_script_pubkey(&script).unwrap().unwrap();
+
+        assert_eq!(chain, keychain);
+        assert_eq!(child, path);
+
+        let res = db.get_path_from_script_pubkey(&script).unwrap();
+        assert!(res.is_none());
+    }
+
+    pub fn test_iter_script_pubkeys<D: Database>(mut db: D) {
+        let keychain = KeychainKind::External;
+        let scripts = db.iter_script_pubkeys(Some(keychain)).unwrap();
+        assert!(scripts.is_empty());
+
+        let first_script = ScriptBuf::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let path = 42;
+
+        db.set_script_pubkey(&first_script, keychain, path).unwrap();
+
+        let second_script = ScriptBuf::from(
+            Vec::<u8>::from_hex("00145c9a1816d38db5cbdd4b067b689dc19eb7d930e2").unwrap(),
+        );
+        let path = 57;
+
+        db.set_script_pubkey(&second_script, keychain, path)
+            .unwrap();
+        let scripts = db.iter_script_pubkeys(Some(keychain)).unwrap();
+
+        assert!(scripts.contains(&first_script));
+        assert!(scripts.contains(&second_script));
+        assert_eq!(scripts.len(), 2);
+    }
+
+    pub fn test_del_utxo<D: Database>(mut db: D) {
+        let outpoint = OutPoint::from_str(
+            "5df6e0e2761359d30a8275058e299fcc0381534545f55cf43e41983f5d4c9456:0",
+        )
+        .unwrap();
+        let script = ScriptBuf::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let txout = TxOut {
+            value: 133742,
+            script_pubkey: script,
+        };
+        let utxo = LocalUtxo {
+            txout,
+            outpoint,
+            keychain: KeychainKind::External,
+            is_spent: true,
+        };
+
+        let res = db.del_utxo(&outpoint).unwrap();
+        assert!(res.is_none());
+
+        db.set_utxo(&utxo).unwrap();
+
+        let res = db.del_utxo(&outpoint).unwrap();
+
+        assert_eq!(res.unwrap(), utxo);
+
+        let res = db.get_utxo(&outpoint).unwrap();
+        assert!(res.is_none());
+    }
+
+    pub fn test_del_raw_tx<D: Database>(mut db: D) {
+        let hex_tx = Vec::<u8>::from_hex("02000000000101f58c18a90d7a76b30c7e47d4e817adfdd79a6a589a615ef36e360f913adce2cd0000000000feffffff0210270000000000001600145c9a1816d38db5cbdd4b067b689dc19eb7d930e2cf70aa2b080000001600140f48b63160043047f4f60f7f8f551f80458f693f024730440220413f42b7bc979945489a38f5221e5527d4b8e3aa63eae2099e01945896ad6c10022024ceec492d685c31d8adb64e935a06933877c5ae0e21f32efe029850914c5bad012102361caae96f0e9f3a453d354bb37a5c3244422fb22819bf0166c0647a38de39f21fca2300").unwrap();
+        let tx: Transaction = deserialize(&hex_tx).unwrap();
+
+        let res = db.del_raw_tx(&tx.txid()).unwrap();
+
+        assert!(res.is_none());
+
+        db.set_raw_tx(&tx).unwrap();
+
+        let res = db.del_raw_tx(&tx.txid()).unwrap();
+
+        assert_eq!(res.unwrap(), tx);
+
+        let res = db.get_raw_tx(&tx.txid()).unwrap();
+        assert!(res.is_none());
+    }
+
+    pub fn test_del_tx<D: Database>(mut db: D) {
+        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
+        let tx: Transaction = deserialize(&hex_tx).unwrap();
+        let txid = tx.txid();
+        let mut tx_details = TransactionDetails {
+            transaction: Some(tx.clone()),
+            txid,
+            received: 1337,
+            sent: 420420,
+            fee: Some(140),
+            confirmation_time: Some(BlockTime {
+                timestamp: 123456,
+                height: 1000,
+            }),
+        };
+
+        let res = db.del_tx(&tx.txid(), true).unwrap();
+
+        assert!(res.is_none());
+
+        db.set_tx(&tx_details).unwrap();
+
+        let res = db.del_tx(&tx.txid(), false).unwrap();
+        tx_details.transaction = None;
+        assert_eq!(res.unwrap(), tx_details);
+
+        let res = db.get_tx(&tx.txid(), true).unwrap();
+        assert!(res.is_none());
+
+        let res = db.get_raw_tx(&tx.txid()).unwrap();
+        assert_eq!(res.unwrap(), tx);
+
+        db.set_tx(&tx_details).unwrap();
+        let res = db.del_tx(&tx.txid(), true).unwrap();
+        tx_details.transaction = Some(tx.clone());
+        assert_eq!(res.unwrap(), tx_details);
+
+        let res = db.get_tx(&tx.txid(), true).unwrap();
+        assert!(res.is_none());
+
+        let res = db.get_raw_tx(&tx.txid()).unwrap();
+        assert!(res.is_none());
+    }
+
+    pub fn test_del_last_index<D: Database>(mut db: D) {
+        let keychain = KeychainKind::External;
+
+        let _res = db.increment_last_index(keychain);
+
+        let res = db.get_last_index(keychain).unwrap().unwrap();
+
+        assert_eq!(res, 0);
+
+        let _res = db.increment_last_index(keychain);
+
+        let res = db.del_last_index(keychain).unwrap().unwrap();
+
+        assert_eq!(res, 1);
+
+        let res = db.get_last_index(keychain).unwrap();
+        assert!(res.is_none());
+    }
+
+    pub fn test_check_descriptor_checksum<D: Database>(mut db: D) {
+        // insert checksum associated to keychain
+        let checksum = "1cead456".as_bytes();
+        let keychain = KeychainKind::External;
+        let _res = db.check_descriptor_checksum(keychain, checksum);
+
+        // check if `check_descriptor_checksum` throws
+        // `Error::ChecksumMismatch` error if the
+        // function is passed a checksum that does
+        // not match the one initially inserted
+        let checksum = "1cead454".as_bytes();
+        let keychain = KeychainKind::External;
+        let res = db.check_descriptor_checksum(keychain, checksum);
+
+        assert!(res.is_err());
+    }
+
+    // TODO: more tests...
+}

src/descriptor/checksum.rs

@@ -1,9 +1,23 @@
-use std::iter::FromIterator;
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
-use crate::descriptor::Error;
+//! Descriptor checksum
+//!
+//! This module contains a re-implementation of the function used by Bitcoin Core to calculate the
+//! checksum of a descriptor
 
-const INPUT_CHARSET: &str =  "0123456789()[],'/*abcdefgh@:$%{}IJKLMNOPQRSTUVWXYZ&+-.;<=>?!^_|~ijklmnopqrstuvwxyzABCDEFGH`#\"\\ ";
-const CHECKSUM_CHARSET: &str = "qpzry9x8gf2tvdw0s3jn54khce6mua7l";
+use crate::descriptor::DescriptorError;
+
+const INPUT_CHARSET: &[u8] = b"0123456789()[],'/*abcdefgh@:$%{}IJKLMNOPQRSTUVWXYZ&+-.;<=>?!^_|~ijklmnopqrstuvwxyzABCDEFGH`#\"\\ ";
+const CHECKSUM_CHARSET: &[u8] = b"qpzry9x8gf2tvdw0s3jn54khce6mua7l";
 
 fn poly_mod(mut c: u64, val: u64) -> u64 {
     let c0 = c >> 35;
@@ -27,14 +41,29 @@ fn poly_mod(mut c: u64, val: u64) -> u64 {
     c
 }
 
-pub fn get_checksum(desc: &str) -> Result<String, Error> {
+/// Computes the checksum bytes of a descriptor.
+/// `exclude_hash = true` ignores all data after the first '#' (inclusive).
+pub(crate) fn calc_checksum_bytes_internal(
+    mut desc: &str,
+    exclude_hash: bool,
+) -> Result<[u8; 8], DescriptorError> {
     let mut c = 1;
     let mut cls = 0;
     let mut clscount = 0;
-    for ch in desc.chars() {
+
+    let mut original_checksum = None;
+    if exclude_hash {
+        if let Some(split) = desc.split_once('#') {
+            desc = split.0;
+            original_checksum = Some(split.1);
+        }
+    }
+
+    for ch in desc.as_bytes() {
         let pos = INPUT_CHARSET
-            .find(ch)
-            .ok_or(Error::InvalidDescriptorCharacter(ch))? as u64;
+            .iter()
+            .position(|b| b == ch)
+            .ok_or(DescriptorError::InvalidDescriptorCharacter(*ch))? as u64;
         c = poly_mod(c, pos & 31);
         cls = cls * 3 + (pos >> 5);
         clscount += 1;
@@ -50,15 +79,103 @@ pub fn get_checksum(desc: &str) -> Result<String, Error> {
     (0..8).for_each(|_| c = poly_mod(c, 0));
     c ^= 1;
 
-    let mut chars = Vec::with_capacity(8);
+    let mut checksum = [0_u8; 8];
     for j in 0..8 {
-        chars.push(
-            CHECKSUM_CHARSET
-                .chars()
-                .nth(((c >> (5 * (7 - j))) & 31) as usize)
-                .unwrap(),
+        checksum[j] = CHECKSUM_CHARSET[((c >> (5 * (7 - j))) & 31) as usize];
+    }
+
+    // if input data already had a checksum, check calculated checksum against original checksum
+    if let Some(original_checksum) = original_checksum {
+        if original_checksum.as_bytes() != checksum {
+            return Err(DescriptorError::InvalidDescriptorChecksum);
+        }
+    }
+
+    Ok(checksum)
+}
+
+/// Compute the checksum bytes of a descriptor, excludes any existing checksum in the descriptor string from the calculation
+pub fn calc_checksum_bytes(desc: &str) -> Result<[u8; 8], DescriptorError> {
+    calc_checksum_bytes_internal(desc, true)
+}
+
+/// Compute the checksum of a descriptor, excludes any existing checksum in the descriptor string from the calculation
+pub fn calc_checksum(desc: &str) -> Result<String, DescriptorError> {
+    // unsafe is okay here as the checksum only uses bytes in `CHECKSUM_CHARSET`
+    calc_checksum_bytes_internal(desc, true)
+        .map(|b| unsafe { String::from_utf8_unchecked(b.to_vec()) })
+}
+
+// TODO in release 0.25.0, remove get_checksum_bytes and get_checksum
+// TODO in release 0.25.0, consolidate calc_checksum_bytes_internal into calc_checksum_bytes
+
+/// Compute the checksum bytes of a descriptor
+#[deprecated(
+    since = "0.24.0",
+    note = "Use new `calc_checksum_bytes` function which excludes any existing checksum in the descriptor string before calculating the checksum hash bytes. See https://github.com/bitcoindevkit/bdk/pull/765."
+)]
+pub fn get_checksum_bytes(desc: &str) -> Result<[u8; 8], DescriptorError> {
+    calc_checksum_bytes_internal(desc, false)
+}
+
+/// Compute the checksum of a descriptor
+#[deprecated(
+    since = "0.24.0",
+    note = "Use new `calc_checksum` function which excludes any existing checksum in the descriptor string before calculating the checksum hash. See https://github.com/bitcoindevkit/bdk/pull/765."
+)]
+pub fn get_checksum(desc: &str) -> Result<String, DescriptorError> {
+    // unsafe is okay here as the checksum only uses bytes in `CHECKSUM_CHARSET`
+    calc_checksum_bytes_internal(desc, false)
+        .map(|b| unsafe { String::from_utf8_unchecked(b.to_vec()) })
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::descriptor::calc_checksum;
+    use assert_matches::assert_matches;
+
+    // test calc_checksum() function; it should return the same value as Bitcoin Core
+    #[test]
+    fn test_calc_checksum() {
+        let desc = "wpkh(tprv8ZgxMBicQKsPdpkqS7Eair4YxjcuuvDPNYmKX3sCniCf16tHEVrjjiSXEkFRnUH77yXc6ZcwHHcLNfjdi5qUvw3VDfgYiH5mNsj5izuiu2N/1/2/*)";
+        assert_eq!(calc_checksum(desc).unwrap(), "tqz0nc62");
+
+        let desc = "pkh(tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/44'/1'/0'/0/*)";
+        assert_eq!(calc_checksum(desc).unwrap(), "lasegmfs");
+    }
+
+    // test calc_checksum() function; it should return the same value as Bitcoin Core even if the
+    // descriptor string includes a checksum hash
+    #[test]
+    fn test_calc_checksum_with_checksum_hash() {
+        let desc = "wpkh(tprv8ZgxMBicQKsPdpkqS7Eair4YxjcuuvDPNYmKX3sCniCf16tHEVrjjiSXEkFRnUH77yXc6ZcwHHcLNfjdi5qUvw3VDfgYiH5mNsj5izuiu2N/1/2/*)#tqz0nc62";
+        assert_eq!(calc_checksum(desc).unwrap(), "tqz0nc62");
+
+        let desc = "pkh(tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/44'/1'/0'/0/*)#lasegmfs";
+        assert_eq!(calc_checksum(desc).unwrap(), "lasegmfs");
+
+        let desc = "wpkh(tprv8ZgxMBicQKsPdpkqS7Eair4YxjcuuvDPNYmKX3sCniCf16tHEVrjjiSXEkFRnUH77yXc6ZcwHHcLNfjdi5qUvw3VDfgYiH5mNsj5izuiu2N/1/2/*)#tqz0nc26";
+        assert_matches!(
+            calc_checksum(desc),
+            Err(DescriptorError::InvalidDescriptorChecksum)
+        );
+
+        let desc = "pkh(tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/44'/1'/0'/0/*)#lasegmsf";
+        assert_matches!(
+            calc_checksum(desc),
+            Err(DescriptorError::InvalidDescriptorChecksum)
         );
     }
 
-    Ok(String::from_iter(chars))
+    #[test]
+    fn test_calc_checksum_invalid_character() {
+        let sparkle_heart = unsafe { std::str::from_utf8_unchecked(&[240, 159, 146, 150]) };
+        let invalid_desc = format!("wpkh(tprv8ZgxMBicQKsPdpkqS7Eair4YxjcuuvDPNYmKX3sCniCf16tHEVrjjiSXEkFRnUH77yXc6ZcwHHcL{}fjdi5qUvw3VDfgYiH5mNsj5izuiu2N/1/2/*)", sparkle_heart);
+
+        assert_matches!(
+            calc_checksum(&invalid_desc),
+            Err(DescriptorError::InvalidDescriptorCharacter(invalid_char)) if invalid_char == sparkle_heart.as_bytes()[0]
+        );
+    }
 }

src/descriptor/error.rs

@@ -1,43 +1,92 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Descriptor errors
+
+/// Errors related to the parsing and usage of descriptors
 #[derive(Debug)]
 pub enum Error {
-    InternalError,
-    InvalidPrefix(Vec<u8>),
-    HardenedDerivationOnXpub,
-    MalformedInput,
-    KeyParsingError(String),
-
-    AliasAsPublicKey,
-    KeyHasSecret,
-    Incomplete,
-    MissingAlias(String),
-    InvalidAlias(String),
+    /// Invalid HD Key path, such as having a wildcard but a length != 1
+    InvalidHdKeyPath,
+    /// The provided descriptor doesn't match its checksum
+    InvalidDescriptorChecksum,
+    /// The descriptor contains hardened derivation steps on public extended keys
+    HardenedDerivationXpub,
+    /// The descriptor contains multipath keys
+    MultiPath,
 
+    /// Error thrown while working with [`keys`](crate::keys)
+    Key(crate::keys::KeyError),
+    /// Error while extracting and manipulating policies
     Policy(crate::descriptor::policy::PolicyError),
 
-    InputIndexDoesntExist,
-    MissingPublicKey,
-    MissingDetails,
+    /// Invalid byte found in the descriptor checksum
+    InvalidDescriptorCharacter(u8),
 
-    InvalidDescriptorCharacter(char),
-
-    CantDeriveWithMiniscript,
-
-    BIP32(bitcoin::util::bip32::Error),
-    Base58(bitcoin::util::base58::Error),
-    PK(bitcoin::util::key::Error),
+    /// BIP32 error
+    Bip32(bitcoin::bip32::Error),
+    /// Error during base58 decoding
+    Base58(bitcoin::base58::Error),
+    /// Key-related error
+    Pk(bitcoin::key::Error),
+    /// Miniscript error
     Miniscript(miniscript::Error),
+    /// Hex decoding error
     Hex(bitcoin::hashes::hex::Error),
 }
 
+impl From<crate::keys::KeyError> for Error {
+    fn from(key_error: crate::keys::KeyError) -> Error {
+        match key_error {
+            crate::keys::KeyError::Miniscript(inner) => Error::Miniscript(inner),
+            crate::keys::KeyError::Bip32(inner) => Error::Bip32(inner),
+            e => Error::Key(e),
+        }
+    }
+}
+
 impl std::fmt::Display for Error {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        write!(f, "{:?}", self)
+        match self {
+            Self::InvalidHdKeyPath => write!(f, "Invalid HD key path"),
+            Self::InvalidDescriptorChecksum => {
+                write!(f, "The provided descriptor doesn't match its checksum")
+            }
+            Self::HardenedDerivationXpub => write!(
+                f,
+                "The descriptor contains hardened derivation steps on public extended keys"
+            ),
+            Self::MultiPath => write!(
+                f,
+                "The descriptor contains multipath keys, which are not supported yet"
+            ),
+            Self::Key(err) => write!(f, "Key error: {}", err),
+            Self::Policy(err) => write!(f, "Policy error: {}", err),
+            Self::InvalidDescriptorCharacter(char) => {
+                write!(f, "Invalid descriptor character: {}", char)
+            }
+            Self::Bip32(err) => write!(f, "BIP32 error: {}", err),
+            Self::Base58(err) => write!(f, "Base58 error: {}", err),
+            Self::Pk(err) => write!(f, "Key-related error: {}", err),
+            Self::Miniscript(err) => write!(f, "Miniscript error: {}", err),
+            Self::Hex(err) => write!(f, "Hex decoding error: {}", err),
+        }
     }
 }
 
-impl_error!(bitcoin::util::bip32::Error, BIP32);
-impl_error!(bitcoin::util::base58::Error, Base58);
-impl_error!(bitcoin::util::key::Error, PK);
+impl std::error::Error for Error {}
+
+impl_error!(bitcoin::bip32::Error, Bip32);
+impl_error!(bitcoin::base58::Error, Base58);
+impl_error!(bitcoin::key::Error, Pk);
 impl_error!(miniscript::Error, Miniscript);
 impl_error!(bitcoin::hashes::hex::Error, Hex);
 impl_error!(crate::descriptor::policy::PolicyError, Policy);

src/descriptor/extended_key.rs

@@ -1,372 +0,0 @@
-use std::fmt::{self, Display};
-use std::str::FromStr;
-
-use bitcoin::hashes::hex::{FromHex, ToHex};
-use bitcoin::secp256k1;
-use bitcoin::util::base58;
-use bitcoin::util::bip32::{
-    ChildNumber, DerivationPath, ExtendedPrivKey, ExtendedPubKey, Fingerprint,
-};
-use bitcoin::PublicKey;
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
-
-#[derive(Copy, Clone, PartialEq, Eq, Debug)]
-pub enum DerivationIndex {
-    Fixed,
-    Normal,
-    Hardened,
-}
-
-impl DerivationIndex {
-    fn as_path(&self, index: u32) -> DerivationPath {
-        match self {
-            DerivationIndex::Fixed => vec![],
-            DerivationIndex::Normal => vec![ChildNumber::Normal { index }],
-            DerivationIndex::Hardened => vec![ChildNumber::Hardened { index }],
-        }
-        .into()
-    }
-}
-
-impl Display for DerivationIndex {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        let chars = match *self {
-            Self::Fixed => "",
-            Self::Normal => "/*",
-            Self::Hardened => "/*'",
-        };
-
-        write!(f, "{}", chars)
-    }
-}
-
-#[derive(Clone, Debug)]
-pub struct DescriptorExtendedKey {
-    pub master_fingerprint: Option<Fingerprint>,
-    pub master_derivation: Option<DerivationPath>,
-    pub pubkey: ExtendedPubKey,
-    pub secret: Option<ExtendedPrivKey>,
-    pub path: DerivationPath,
-    pub final_index: DerivationIndex,
-}
-
-impl DescriptorExtendedKey {
-    pub fn full_path(&self, index: u32) -> DerivationPath {
-        let mut final_path: Vec<ChildNumber> = Vec::new();
-        if let Some(path) = &self.master_derivation {
-            let path_as_vec: Vec<ChildNumber> = path.clone().into();
-            final_path.extend_from_slice(&path_as_vec);
-        }
-        let our_path: Vec<ChildNumber> = self.path_with_index(index).into();
-        final_path.extend_from_slice(&our_path);
-
-        final_path.into()
-    }
-
-    pub fn path_with_index(&self, index: u32) -> DerivationPath {
-        let mut final_path: Vec<ChildNumber> = Vec::new();
-        let our_path: Vec<ChildNumber> = self.path.clone().into();
-        final_path.extend_from_slice(&our_path);
-        let other_path: Vec<ChildNumber> = self.final_index.as_path(index).into();
-        final_path.extend_from_slice(&other_path);
-
-        final_path.into()
-    }
-
-    pub fn derive<C: secp256k1::Verification + secp256k1::Signing>(
-        &self,
-        ctx: &secp256k1::Secp256k1<C>,
-        index: u32,
-    ) -> Result<PublicKey, super::Error> {
-        Ok(self.derive_xpub(ctx, index)?.public_key)
-    }
-
-    pub fn derive_xpub<C: secp256k1::Verification + secp256k1::Signing>(
-        &self,
-        ctx: &secp256k1::Secp256k1<C>,
-        index: u32,
-    ) -> Result<ExtendedPubKey, super::Error> {
-        if let Some(xprv) = self.secret {
-            let derive_priv = xprv.derive_priv(ctx, &self.path_with_index(index))?;
-            Ok(ExtendedPubKey::from_private(ctx, &derive_priv))
-        } else {
-            Ok(self.pubkey.derive_pub(ctx, &self.path_with_index(index))?)
-        }
-    }
-
-    pub fn root_xpub<C: secp256k1::Verification + secp256k1::Signing>(
-        &self,
-        ctx: &secp256k1::Secp256k1<C>,
-    ) -> ExtendedPubKey {
-        if let Some(ref xprv) = self.secret {
-            ExtendedPubKey::from_private(ctx, xprv)
-        } else {
-            self.pubkey
-        }
-    }
-}
-
-impl Display for DescriptorExtendedKey {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        if let Some(ref fingerprint) = self.master_fingerprint {
-            write!(f, "[{}", fingerprint.to_hex())?;
-            if let Some(ref path) = self.master_derivation {
-                write!(f, "{}", &path.to_string()[1..])?;
-            }
-            write!(f, "]")?;
-        }
-
-        if let Some(xprv) = self.secret {
-            write!(f, "{}", xprv)?
-        } else {
-            write!(f, "{}", self.pubkey)?
-        }
-
-        write!(f, "{}{}", &self.path.to_string()[1..], self.final_index)
-    }
-}
-
-impl FromStr for DescriptorExtendedKey {
-    type Err = super::Error;
-
-    fn from_str(inp: &str) -> Result<DescriptorExtendedKey, Self::Err> {
-        let len = inp.len();
-
-        let (master_fingerprint, master_derivation, offset) = match inp.starts_with("[") {
-            false => (None, None, 0),
-            true => {
-                if inp.len() < 9 {
-                    return Err(super::Error::MalformedInput);
-                }
-
-                let master_fingerprint = &inp[1..9];
-                let close_bracket_index =
-                    &inp[9..].find("]").ok_or(super::Error::MalformedInput)?;
-                let path = if *close_bracket_index > 0 {
-                    Some(DerivationPath::from_str(&format!(
-                        "m{}",
-                        &inp[9..9 + *close_bracket_index]
-                    ))?)
-                } else {
-                    None
-                };
-
-                (
-                    Some(Fingerprint::from_hex(master_fingerprint)?),
-                    path,
-                    9 + *close_bracket_index + 1,
-                )
-            }
-        };
-
-        let (key_range, offset) = match &inp[offset..].find("/") {
-            Some(index) => (offset..offset + *index, offset + *index),
-            None => (offset..len, len),
-        };
-        let data = base58::from_check(&inp[key_range.clone()])?;
-        let secp = secp256k1::Secp256k1::new();
-        let (pubkey, secret) = match &data[0..4] {
-            [0x04u8, 0x88, 0xB2, 0x1E] | [0x04u8, 0x35, 0x87, 0xCF] => {
-                (ExtendedPubKey::from_str(&inp[key_range])?, None)
-            }
-            [0x04u8, 0x88, 0xAD, 0xE4] | [0x04u8, 0x35, 0x83, 0x94] => {
-                let private = ExtendedPrivKey::from_str(&inp[key_range])?;
-                (ExtendedPubKey::from_private(&secp, &private), Some(private))
-            }
-            data => return Err(super::Error::InvalidPrefix(data.into())),
-        };
-
-        let (path, final_index, _) = match &inp[offset..].starts_with("/") {
-            false => (DerivationPath::from(vec![]), DerivationIndex::Fixed, offset),
-            true => {
-                let (all, skip) = match &inp[len - 2..len] {
-                    "/*" => (DerivationIndex::Normal, 2),
-                    "*'" | "*h" => (DerivationIndex::Hardened, 3),
-                    _ => (DerivationIndex::Fixed, 0),
-                };
-
-                if all == DerivationIndex::Hardened && secret.is_none() {
-                    return Err(super::Error::HardenedDerivationOnXpub);
-                }
-
-                (
-                    DerivationPath::from_str(&format!("m{}", &inp[offset..len - skip]))?,
-                    all,
-                    len,
-                )
-            }
-        };
-
-        if secret.is_none()
-            && path.into_iter().any(|child| match child {
-                ChildNumber::Hardened { .. } => true,
-                _ => false,
-            })
-        {
-            return Err(super::Error::HardenedDerivationOnXpub);
-        }
-
-        Ok(DescriptorExtendedKey {
-            master_fingerprint,
-            master_derivation,
-            pubkey,
-            secret,
-            path,
-            final_index,
-        })
-    }
-}
-
-#[cfg(test)]
-mod test {
-    use std::str::FromStr;
-
-    use bitcoin::hashes::hex::FromHex;
-    use bitcoin::util::bip32::{ChildNumber, DerivationPath};
-
-    use crate::descriptor::*;
-
-    macro_rules! hex_fingerprint {
-        ($hex:expr) => {
-            Fingerprint::from_hex($hex).unwrap()
-        };
-    }
-
-    macro_rules! deriv_path {
-        ($str:expr) => {
-            DerivationPath::from_str($str).unwrap()
-        };
-
-        () => {
-            DerivationPath::from(vec![])
-        };
-    }
-
-    #[test]
-    fn test_derivation_index_fixed() {
-        let index = DerivationIndex::Fixed;
-        assert_eq!(index.as_path(1337), DerivationPath::from(vec![]));
-        assert_eq!(format!("{}", index), "");
-    }
-
-    #[test]
-    fn test_derivation_index_normal() {
-        let index = DerivationIndex::Normal;
-        assert_eq!(
-            index.as_path(1337),
-            DerivationPath::from(vec![ChildNumber::Normal { index: 1337 }])
-        );
-        assert_eq!(format!("{}", index), "/*");
-    }
-
-    #[test]
-    fn test_derivation_index_hardened() {
-        let index = DerivationIndex::Hardened;
-        assert_eq!(
-            index.as_path(1337),
-            DerivationPath::from(vec![ChildNumber::Hardened { index: 1337 }])
-        );
-        assert_eq!(format!("{}", index), "/*'");
-    }
-
-    #[test]
-    fn test_parse_xpub_no_path_fixed() {
-        let key = "xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL";
-        let ek = DescriptorExtendedKey::from_str(key).unwrap();
-        assert_eq!(ek.pubkey.fingerprint(), hex_fingerprint!("31a507b8"));
-        assert_eq!(ek.path, deriv_path!());
-        assert_eq!(ek.final_index, DerivationIndex::Fixed);
-    }
-
-    #[test]
-    fn test_parse_xpub_with_path_fixed() {
-        let key = "xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/1/2/3";
-        let ek = DescriptorExtendedKey::from_str(key).unwrap();
-        assert_eq!(ek.pubkey.fingerprint(), hex_fingerprint!("31a507b8"));
-        assert_eq!(ek.path, deriv_path!("m/1/2/3"));
-        assert_eq!(ek.final_index, DerivationIndex::Fixed);
-    }
-
-    #[test]
-    fn test_parse_xpub_with_path_normal() {
-        let key = "xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/1/2/3/*";
-        let ek = DescriptorExtendedKey::from_str(key).unwrap();
-        assert_eq!(ek.pubkey.fingerprint(), hex_fingerprint!("31a507b8"));
-        assert_eq!(ek.path, deriv_path!("m/1/2/3"));
-        assert_eq!(ek.final_index, DerivationIndex::Normal);
-    }
-
-    #[test]
-    #[should_panic(expected = "HardenedDerivationOnXpub")]
-    fn test_parse_xpub_with_path_hardened() {
-        let key = "xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/*'";
-        let ek = DescriptorExtendedKey::from_str(key).unwrap();
-        assert_eq!(ek.pubkey.fingerprint(), hex_fingerprint!("31a507b8"));
-        assert_eq!(ek.path, deriv_path!("m/1/2/3"));
-        assert_eq!(ek.final_index, DerivationIndex::Fixed);
-    }
-
-    #[test]
-    fn test_parse_tprv_with_path_hardened() {
-        let key = "tprv8ZgxMBicQKsPduL5QnGihpprdHyypMGi4DhimjtzYemu7se5YQNcZfAPLqXRuGHb5ZX2eTQj62oNqMnyxJ7B7wz54Uzswqw8fFqMVdcmVF7/1/2/3/*'";
-        let ek = DescriptorExtendedKey::from_str(key).unwrap();
-        assert!(ek.secret.is_some());
-        assert_eq!(ek.pubkey.fingerprint(), hex_fingerprint!("5ea4190e"));
-        assert_eq!(ek.path, deriv_path!("m/1/2/3"));
-        assert_eq!(ek.final_index, DerivationIndex::Hardened);
-    }
-
-    #[test]
-    fn test_parse_xpub_master_details() {
-        let key = "[d34db33f/44'/0'/0']xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL";
-        let ek = DescriptorExtendedKey::from_str(key).unwrap();
-        assert_eq!(ek.master_fingerprint, Some(hex_fingerprint!("d34db33f")));
-        assert_eq!(ek.master_derivation, Some(deriv_path!("m/44'/0'/0'")));
-    }
-
-    #[test]
-    fn test_parse_xpub_master_details_empty_derivation() {
-        let key = "[d34db33f]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL";
-        let ek = DescriptorExtendedKey::from_str(key).unwrap();
-        assert_eq!(ek.master_fingerprint, Some(hex_fingerprint!("d34db33f")));
-        assert_eq!(ek.master_derivation, None);
-    }
-
-    #[test]
-    #[should_panic(expected = "MalformedInput")]
-    fn test_parse_xpub_short_input() {
-        let key = "[d34d";
-        DescriptorExtendedKey::from_str(key).unwrap();
-    }
-
-    #[test]
-    #[should_panic(expected = "MalformedInput")]
-    fn test_parse_xpub_missing_closing_bracket() {
-        let key = "[d34db33fxpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL";
-        DescriptorExtendedKey::from_str(key).unwrap();
-    }
-
-    #[test]
-    #[should_panic(expected = "InvalidChar")]
-    fn test_parse_xpub_invalid_fingerprint() {
-        let key = "[d34db33z]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL";
-        DescriptorExtendedKey::from_str(key).unwrap();
-    }
-
-    #[test]
-    fn test_xpub_normal_full_path() {
-        let key = "xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/1/2/*";
-        let ek = DescriptorExtendedKey::from_str(key).unwrap();
-        assert_eq!(ek.full_path(42), deriv_path!("m/1/2/42"));
-    }
-
-    #[test]
-    fn test_xpub_fixed_full_path() {
-        let key = "xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/1/2";
-        let ek = DescriptorExtendedKey::from_str(key).unwrap();
-        assert_eq!(ek.full_path(42), deriv_path!("m/1/2"));
-        assert_eq!(ek.full_path(1337), deriv_path!("m/1/2"));
-    }
-}

src/descriptor/keys.rs

@@ -1,280 +0,0 @@
-use std::fmt;
-use std::str::FromStr;
-
-use bitcoin::secp256k1::{All, Secp256k1};
-use bitcoin::{PrivateKey, PublicKey};
-
-use bitcoin::util::bip32::{
-    ChildNumber, DerivationPath, ExtendedPrivKey, ExtendedPubKey, Fingerprint,
-};
-
-use super::error::Error;
-use super::extended_key::DerivationIndex;
-use super::DescriptorExtendedKey;
-
-#[derive(Debug, Clone)]
-pub struct KeyAlias {
-    alias: String,
-    has_secret: bool,
-}
-
-impl KeyAlias {
-    pub(crate) fn new_boxed(alias: &str, has_secret: bool) -> Box<dyn Key> {
-        Box::new(KeyAlias {
-            alias: alias.into(),
-            has_secret,
-        })
-    }
-}
-
-pub(crate) fn parse_key(string: &str) -> Result<(String, Box<dyn RealKey>), Error> {
-    if let Ok(pk) = PublicKey::from_str(string) {
-        return Ok((string.to_string(), Box::new(pk)));
-    } else if let Ok(sk) = PrivateKey::from_wif(string) {
-        return Ok((string.to_string(), Box::new(sk)));
-    } else if let Ok(ext_key) = DescriptorExtendedKey::from_str(string) {
-        return Ok((string.to_string(), Box::new(ext_key)));
-    }
-
-    return Err(Error::KeyParsingError(string.to_string()));
-}
-
-pub trait Key: std::fmt::Debug + std::fmt::Display {
-    fn as_public_key(&self, secp: &Secp256k1<All>, index: Option<u32>) -> Result<PublicKey, Error>;
-    fn is_fixed(&self) -> bool;
-
-    fn alias(&self) -> Option<&str> {
-        None
-    }
-
-    fn as_secret_key(&self) -> Option<PrivateKey> {
-        None
-    }
-
-    fn xprv(&self) -> Option<ExtendedPrivKey> {
-        None
-    }
-
-    fn full_path(&self, _index: u32) -> Option<DerivationPath> {
-        None
-    }
-
-    fn fingerprint(&self, _secp: &Secp256k1<All>) -> Option<Fingerprint> {
-        None
-    }
-
-    fn has_secret(&self) -> bool {
-        self.xprv().is_some() || self.as_secret_key().is_some()
-    }
-
-    fn public(&self, secp: &Secp256k1<All>) -> Result<Box<dyn RealKey>, Error> {
-        Ok(Box::new(self.as_public_key(secp, None)?))
-    }
-}
-
-pub trait RealKey: Key {
-    fn into_key(&self) -> Box<dyn Key>;
-}
-
-impl<T: RealKey + 'static> From<T> for Box<dyn RealKey> {
-    fn from(key: T) -> Self {
-        Box::new(key)
-    }
-}
-
-impl Key for PublicKey {
-    fn as_public_key(
-        &self,
-        _secp: &Secp256k1<All>,
-        _index: Option<u32>,
-    ) -> Result<PublicKey, Error> {
-        Ok(PublicKey::clone(self))
-    }
-
-    fn is_fixed(&self) -> bool {
-        true
-    }
-}
-
-impl RealKey for PublicKey {
-    fn into_key(&self) -> Box<dyn Key> {
-        Box::new(self.clone())
-    }
-}
-
-impl Key for PrivateKey {
-    fn as_public_key(
-        &self,
-        secp: &Secp256k1<All>,
-        _index: Option<u32>,
-    ) -> Result<PublicKey, Error> {
-        Ok(self.public_key(secp))
-    }
-
-    fn as_secret_key(&self) -> Option<PrivateKey> {
-        Some(PrivateKey::clone(self))
-    }
-
-    fn is_fixed(&self) -> bool {
-        true
-    }
-}
-impl RealKey for PrivateKey {
-    fn into_key(&self) -> Box<dyn Key> {
-        Box::new(self.clone())
-    }
-}
-
-impl Key for DescriptorExtendedKey {
-    fn fingerprint(&self, secp: &Secp256k1<All>) -> Option<Fingerprint> {
-        if let Some(fing) = self.master_fingerprint {
-            Some(fing.clone())
-        } else {
-            Some(self.root_xpub(secp).fingerprint())
-        }
-    }
-
-    fn as_public_key(&self, secp: &Secp256k1<All>, index: Option<u32>) -> Result<PublicKey, Error> {
-        Ok(self.derive_xpub(secp, index.unwrap_or(0))?.public_key)
-    }
-
-    fn public(&self, secp: &Secp256k1<All>) -> Result<Box<dyn RealKey>, Error> {
-        if self.final_index == DerivationIndex::Hardened {
-            return Err(Error::HardenedDerivationOnXpub);
-        }
-
-        if self.xprv().is_none() {
-            return Ok(Box::new(self.clone()));
-        }
-
-        // copy the part of the path that can be derived on the xpub
-        let path = self
-            .path
-            .into_iter()
-            .rev()
-            .take_while(|child| match child {
-                ChildNumber::Normal { .. } => true,
-                _ => false,
-            })
-            .cloned()
-            .collect::<Vec<_>>();
-        // take the prefix that has to be derived on the xprv
-        let master_derivation_add = self
-            .path
-            .into_iter()
-            .take(self.path.as_ref().len() - path.len())
-            .cloned()
-            .collect::<Vec<_>>();
-        let has_derived = !master_derivation_add.is_empty();
-
-        let derived_xprv = self
-            .secret
-            .as_ref()
-            .unwrap()
-            .derive_priv(secp, &master_derivation_add)?;
-        let pubkey = ExtendedPubKey::from_private(secp, &derived_xprv);
-
-        let master_derivation = self
-            .master_derivation
-            .as_ref()
-            .map_or(vec![], |path| path.as_ref().to_vec())
-            .into_iter()
-            .chain(master_derivation_add.into_iter())
-            .collect::<Vec<_>>();
-        let master_derivation = match &master_derivation[..] {
-            &[] => None,
-            child_vec => Some(child_vec.into()),
-        };
-
-        let master_fingerprint = match self.master_fingerprint {
-            Some(desc) => Some(desc.clone()),
-            None if has_derived => Some(self.fingerprint(secp).unwrap()),
-            _ => None,
-        };
-
-        Ok(Box::new(DescriptorExtendedKey {
-            master_fingerprint,
-            master_derivation,
-            pubkey,
-            secret: None,
-            path: path.into(),
-            final_index: self.final_index,
-        }))
-    }
-
-    fn xprv(&self) -> Option<ExtendedPrivKey> {
-        self.secret
-    }
-
-    fn full_path(&self, index: u32) -> Option<DerivationPath> {
-        Some(self.full_path(index))
-    }
-
-    fn is_fixed(&self) -> bool {
-        self.final_index == DerivationIndex::Fixed
-    }
-}
-impl RealKey for DescriptorExtendedKey {
-    fn into_key(&self) -> Box<dyn Key> {
-        Box::new(self.clone())
-    }
-}
-
-impl std::fmt::Display for KeyAlias {
-    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        let flag = if self.has_secret { "#" } else { "" };
-
-        write!(f, "{}{}", flag, self.alias)
-    }
-}
-
-impl Key for KeyAlias {
-    fn as_public_key(
-        &self,
-        _secp: &Secp256k1<All>,
-        _index: Option<u32>,
-    ) -> Result<PublicKey, Error> {
-        Err(Error::AliasAsPublicKey)
-    }
-
-    fn is_fixed(&self) -> bool {
-        true
-    }
-
-    fn alias(&self) -> Option<&str> {
-        Some(self.alias.as_str())
-    }
-
-    fn has_secret(&self) -> bool {
-        self.has_secret
-    }
-
-    fn public(&self, _secp: &Secp256k1<All>) -> Result<Box<dyn RealKey>, Error> {
-        Err(Error::AliasAsPublicKey)
-    }
-}
-
-#[derive(Debug, Clone, Hash, PartialEq, PartialOrd, Eq, Ord, Default)]
-pub(crate) struct DummyKey();
-
-impl fmt::Display for DummyKey {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        write!(f, "DummyKey")
-    }
-}
-
-impl std::str::FromStr for DummyKey {
-    type Err = ();
-
-    fn from_str(_: &str) -> Result<Self, Self::Err> {
-        Ok(DummyKey::default())
-    }
-}
-
-impl miniscript::MiniscriptKey for DummyKey {
-    type Hash = DummyKey;
-
-    fn to_pubkeyhash(&self) -> DummyKey {
-        DummyKey::default()
-    }
-}

src/doctest.rs

@@ -0,0 +1,14 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+#[doc = include_str!("../README.md")]
+#[cfg(doctest)]
+pub struct ReadmeDoctests;

src/error.rs

@@ -1,83 +1,351 @@
-use bitcoin::{OutPoint, Script, Txid};
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
+use std::fmt;
+
+use crate::bitcoin::Network;
+use crate::{descriptor, wallet};
+use bitcoin::{OutPoint, Txid};
+
+/// Errors that can be thrown by the [`Wallet`](crate::wallet::Wallet)
 #[derive(Debug)]
 pub enum Error {
-    KeyMismatch(bitcoin::secp256k1::PublicKey, bitcoin::secp256k1::PublicKey),
-    MissingInputUTXO(usize),
+    /// Wrong number of bytes found when trying to convert to u32
     InvalidU32Bytes(Vec<u8>),
+    /// Generic error
     Generic(String),
+    /// This error is thrown when trying to convert Bare and Public key script to address
     ScriptDoesntHaveAddressForm,
-    SendAllMultipleOutputs,
+    /// Cannot build a tx without recipients
+    NoRecipients,
+    /// `manually_selected_only` option is selected but no utxo has been passed
+    NoUtxosSelected,
+    /// Output created is under the dust limit, 546 satoshis
     OutputBelowDustLimit(usize),
-    InsufficientFunds,
-    UnknownUTXO,
-    DifferentTransactions,
-
+    /// Wallet's UTXO set is not enough to cover recipient's requested plus fee
+    InsufficientFunds {
+        /// Sats needed for some transaction
+        needed: u64,
+        /// Sats available for spending
+        available: u64,
+    },
+    /// Branch and bound coin selection possible attempts with sufficiently big UTXO set could grow
+    /// exponentially, thus a limit is set, and when hit, this error is thrown
+    BnBTotalTriesExceeded,
+    /// Branch and bound coin selection tries to avoid needing a change by finding the right inputs for
+    /// the desired outputs plus fee, if there is not such combination this error is thrown
+    BnBNoExactMatch,
+    /// Happens when trying to spend an UTXO that is not in the internal database
+    UnknownUtxo,
+    /// Thrown when a tx is not found in the internal database
+    TransactionNotFound,
+    /// Happens when trying to bump a transaction that is already confirmed
+    TransactionConfirmed,
+    /// Trying to replace a tx that has a sequence >= `0xFFFFFFFE`
+    IrreplaceableTransaction,
+    /// When bumping a tx the fee rate requested is lower than required
+    FeeRateTooLow {
+        /// Required fee rate (satoshi/vbyte)
+        required: crate::types::FeeRate,
+    },
+    /// When bumping a tx the absolute fee requested is lower than replaced tx absolute fee
+    FeeTooLow {
+        /// Required fee absolute value (satoshi)
+        required: u64,
+    },
+    /// Node doesn't have data to estimate a fee rate
+    FeeRateUnavailable,
+    /// In order to use the [`TxBuilder::add_global_xpubs`] option every extended
+    /// key in the descriptor must either be a master key itself (having depth = 0) or have an
+    /// explicit origin provided
+    ///
+    /// [`TxBuilder::add_global_xpubs`]: crate::wallet::tx_builder::TxBuilder::add_global_xpubs
+    MissingKeyOrigin(String),
+    /// Error while working with [`keys`](crate::keys)
+    Key(crate::keys::KeyError),
+    /// Descriptor checksum mismatch
     ChecksumMismatch,
-    DifferentDescriptorStructure,
-
-    SpendingPolicyRequired,
+    /// Spending policy is not compatible with this [`KeychainKind`](crate::types::KeychainKind)
+    SpendingPolicyRequired(crate::types::KeychainKind),
+    /// Error while extracting and manipulating policies
     InvalidPolicyPathError(crate::descriptor::policy::PolicyError),
+    /// Signing error
+    Signer(crate::wallet::signer::SignerError),
+    /// Invalid network
+    InvalidNetwork {
+        /// requested network, for example what is given as bdk-cli option
+        requested: Network,
+        /// found network, for example the network of the bitcoin node
+        found: Network,
+    },
+    /// The address requested comes from an hardened index
+    HardenedIndex,
+    #[cfg(feature = "verify")]
+    /// Transaction verification error
+    Verification(crate::wallet::verify::VerifyError),
 
-    // Signing errors (expected, received)
-    InputTxidMismatch((Txid, OutPoint)),
-    InputRedeemScriptMismatch((Script, Script)), // scriptPubKey, redeemScript
-    InputWitnessScriptMismatch((Script, Script)), // scriptPubKey, redeemScript
-    InputUnknownSegwitScript(Script),
-    InputMissingWitnessScript(usize),
-    MissingUTXO,
-
-    // Blockchain interface errors
-    Uncapable(crate::blockchain::Capability),
-    OfflineClient,
+    /// Progress value must be between `0.0` (included) and `100.0` (included)
     InvalidProgressValue(f32),
+    /// Progress update error (maybe the channel has been closed)
     ProgressUpdateError,
-    MissingCachedAddresses,
+    /// Requested outpoint doesn't exist in the tx (vout greater than available outputs)
     InvalidOutpoint(OutPoint),
 
+    /// Error related to the parsing and usage of descriptors
     Descriptor(crate::descriptor::error::Error),
-
+    /// Encoding error
     Encode(bitcoin::consensus::encode::Error),
-    BIP32(bitcoin::util::bip32::Error),
+    /// Miniscript error
+    Miniscript(miniscript::Error),
+    /// Miniscript PSBT error
+    MiniscriptPsbt(MiniscriptPsbtError),
+    /// BIP32 error
+    Bip32(bitcoin::bip32::Error),
+    /// A secp256k1 error
     Secp256k1(bitcoin::secp256k1::Error),
-    JSON(serde_json::Error),
+    /// Error serializing or deserializing JSON data
+    Json(serde_json::Error),
+    /// Hex decoding error
     Hex(bitcoin::hashes::hex::Error),
-    PSBT(bitcoin::util::psbt::Error),
+    /// Partially signed bitcoin transaction error
+    Psbt(bitcoin::psbt::Error),
+    /// Partially signed bitcoin transaction parse error
+    PsbtParse(bitcoin::psbt::PsbtParseError),
+
+    //KeyMismatch(bitcoin::secp256k1::PublicKey, bitcoin::secp256k1::PublicKey),
+    //MissingInputUTXO(usize),
+    //InvalidAddressNetwork(Address),
+    //DifferentTransactions,
+    //DifferentDescriptorStructure,
+    //Uncapable(crate::blockchain::Capability),
+    //MissingCachedAddresses,
+    /// [`crate::blockchain::WalletSync`] sync attempt failed due to missing scripts in cache which
+    /// are needed to satisfy `stop_gap`.
+    MissingCachedScripts(MissingCachedScripts),
 
     #[cfg(feature = "electrum")]
+    /// Electrum client error
     Electrum(electrum_client::Error),
     #[cfg(feature = "esplora")]
-    Esplora(crate::blockchain::esplora::EsploraError),
+    /// Esplora client error
+    Esplora(Box<crate::blockchain::esplora::EsploraError>),
+    #[cfg(feature = "compact_filters")]
+    /// Compact filters client error)
+    CompactFilters(crate::blockchain::compact_filters::CompactFiltersError),
     #[cfg(feature = "key-value-db")]
+    /// Sled database error
     Sled(sled::Error),
+    #[cfg(feature = "rpc")]
+    /// Rpc client error
+    Rpc(bitcoincore_rpc::Error),
+    #[cfg(feature = "sqlite")]
+    /// Rusqlite client error
+    Rusqlite(rusqlite::Error),
 }
 
+/// Errors returned by miniscript when updating inconsistent PSBTs
+#[derive(Debug, Clone)]
+pub enum MiniscriptPsbtError {
+    Conversion(miniscript::descriptor::ConversionError),
+    UtxoUpdate(miniscript::psbt::UtxoUpdateError),
+    OutputUpdate(miniscript::psbt::OutputUpdateError),
+}
+
+impl fmt::Display for MiniscriptPsbtError {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            Self::Conversion(err) => write!(f, "Conversion error: {}", err),
+            Self::UtxoUpdate(err) => write!(f, "UTXO update error: {}", err),
+            Self::OutputUpdate(err) => write!(f, "Output update error: {}", err),
+        }
+    }
+}
+
+impl std::error::Error for MiniscriptPsbtError {}
+
+/// Represents the last failed [`crate::blockchain::WalletSync`] sync attempt in which we were short
+/// on cached `scriptPubKey`s.
+#[derive(Debug)]
+pub struct MissingCachedScripts {
+    /// Number of scripts in which txs were requested during last request.
+    pub last_count: usize,
+    /// Minimum number of scripts to cache more of in order to satisfy `stop_gap`.
+    pub missing_count: usize,
+}
+
+impl fmt::Display for Error {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            Self::InvalidU32Bytes(_) => write!(
+                f,
+                "Wrong number of bytes found when trying to convert to u32"
+            ),
+            Self::Generic(err) => write!(f, "Generic error: {}", err),
+            Self::ScriptDoesntHaveAddressForm => write!(f, "Script doesn't have address form"),
+            Self::NoRecipients => write!(f, "Cannot build tx without recipients"),
+            Self::NoUtxosSelected => write!(f, "No UTXO selected"),
+            Self::OutputBelowDustLimit(limit) => {
+                write!(f, "Output below the dust limit: {}", limit)
+            }
+            Self::InsufficientFunds { needed, available } => write!(
+                f,
+                "Insufficient funds: {} sat available of {} sat needed",
+                available, needed
+            ),
+            Self::BnBTotalTriesExceeded => {
+                write!(f, "Branch and bound coin selection: total tries exceeded")
+            }
+            Self::BnBNoExactMatch => write!(f, "Branch and bound coin selection: not exact match"),
+            Self::UnknownUtxo => write!(f, "UTXO not found in the internal database"),
+            Self::TransactionNotFound => {
+                write!(f, "Transaction not found in the internal database")
+            }
+            Self::TransactionConfirmed => write!(f, "Transaction already confirmed"),
+            Self::IrreplaceableTransaction => write!(f, "Transaction can't be replaced"),
+            Self::FeeRateTooLow { required } => write!(
+                f,
+                "Fee rate too low: required {} sat/vbyte",
+                required.as_sat_per_vb()
+            ),
+            Self::FeeTooLow { required } => write!(f, "Fee to low: required {} sat", required),
+            Self::FeeRateUnavailable => write!(f, "Fee rate unavailable"),
+            Self::MissingKeyOrigin(err) => write!(f, "Missing key origin: {}", err),
+            Self::Key(err) => write!(f, "Key error: {}", err),
+            Self::ChecksumMismatch => write!(f, "Descriptor checksum mismatch"),
+            Self::SpendingPolicyRequired(keychain_kind) => {
+                write!(f, "Spending policy required: {:?}", keychain_kind)
+            }
+            Self::InvalidPolicyPathError(err) => write!(f, "Invalid policy path: {}", err),
+            Self::Signer(err) => write!(f, "Signer error: {}", err),
+            Self::InvalidNetwork { requested, found } => write!(
+                f,
+                "Invalid network: requested {} but found {}",
+                requested, found
+            ),
+            Self::HardenedIndex => write!(f, "Requested address from an hardened index"),
+            #[cfg(feature = "verify")]
+            Self::Verification(err) => write!(f, "Transaction verification error: {}", err),
+            Self::InvalidProgressValue(progress) => {
+                write!(f, "Invalid progress value: {}", progress)
+            }
+            Self::ProgressUpdateError => write!(
+                f,
+                "Progress update error (maybe the channel has been closed)"
+            ),
+            Self::InvalidOutpoint(outpoint) => write!(
+                f,
+                "Requested outpoint doesn't exist in the tx: {}",
+                outpoint
+            ),
+            Self::Descriptor(err) => write!(f, "Descriptor error: {}", err),
+            Self::Encode(err) => write!(f, "Encoding error: {}", err),
+            Self::Miniscript(err) => write!(f, "Miniscript error: {}", err),
+            Self::MiniscriptPsbt(err) => write!(f, "Miniscript PSBT error: {}", err),
+            Self::Bip32(err) => write!(f, "BIP32 error: {}", err),
+            Self::Secp256k1(err) => write!(f, "Secp256k1 error: {}", err),
+            Self::Json(err) => write!(f, "Serialize/Deserialize JSON error: {}", err),
+            Self::Hex(err) => write!(f, "Hex decoding error: {}", err),
+            Self::Psbt(err) => write!(f, "PSBT error: {}", err),
+            Self::PsbtParse(err) => write!(f, "Impossible to parse PSBT: {}", err),
+            Self::MissingCachedScripts(missing_cached_scripts) => {
+                write!(f, "Missing cached scripts: {:?}", missing_cached_scripts)
+            }
+            #[cfg(feature = "electrum")]
+            Self::Electrum(err) => write!(f, "Electrum client error: {}", err),
+            #[cfg(feature = "esplora")]
+            Self::Esplora(err) => write!(f, "Esplora client error: {}", err),
+            #[cfg(feature = "compact_filters")]
+            Self::CompactFilters(err) => write!(f, "Compact filters client error: {}", err),
+            #[cfg(feature = "key-value-db")]
+            Self::Sled(err) => write!(f, "Sled database error: {}", err),
+            #[cfg(feature = "rpc")]
+            Self::Rpc(err) => write!(f, "RPC client error: {}", err),
+            #[cfg(feature = "sqlite")]
+            Self::Rusqlite(err) => write!(f, "SQLite error: {}", err),
+        }
+    }
+}
+
+impl std::error::Error for Error {}
+
 macro_rules! impl_error {
     ( $from:ty, $to:ident ) => {
-        impl std::convert::From<$from> for Error {
+        impl_error!($from, $to, Error);
+    };
+    ( $from:ty, $to:ident, $impl_for:ty ) => {
+        impl std::convert::From<$from> for $impl_for {
             fn from(err: $from) -> Self {
-                Error::$to(err)
+                <$impl_for>::$to(err)
             }
         }
     };
 }
 
-impl_error!(crate::descriptor::error::Error, Descriptor);
-impl_error!(
-    crate::descriptor::policy::PolicyError,
-    InvalidPolicyPathError
-);
+impl_error!(descriptor::error::Error, Descriptor);
+impl_error!(descriptor::policy::PolicyError, InvalidPolicyPathError);
+impl_error!(wallet::signer::SignerError, Signer);
+
+impl From<crate::keys::KeyError> for Error {
+    fn from(key_error: crate::keys::KeyError) -> Error {
+        match key_error {
+            crate::keys::KeyError::Miniscript(inner) => Error::Miniscript(inner),
+            crate::keys::KeyError::Bip32(inner) => Error::Bip32(inner),
+            crate::keys::KeyError::InvalidChecksum => Error::ChecksumMismatch,
+            e => Error::Key(e),
+        }
+    }
+}
 
 impl_error!(bitcoin::consensus::encode::Error, Encode);
-impl_error!(bitcoin::util::bip32::Error, BIP32);
+impl_error!(miniscript::Error, Miniscript);
+impl_error!(MiniscriptPsbtError, MiniscriptPsbt);
+impl_error!(bitcoin::bip32::Error, Bip32);
 impl_error!(bitcoin::secp256k1::Error, Secp256k1);
-impl_error!(serde_json::Error, JSON);
+impl_error!(serde_json::Error, Json);
 impl_error!(bitcoin::hashes::hex::Error, Hex);
-impl_error!(bitcoin::util::psbt::Error, PSBT);
+impl_error!(bitcoin::psbt::Error, Psbt);
+impl_error!(bitcoin::psbt::PsbtParseError, PsbtParse);
 
 #[cfg(feature = "electrum")]
 impl_error!(electrum_client::Error, Electrum);
-#[cfg(feature = "esplora")]
-impl_error!(crate::blockchain::esplora::EsploraError, Esplora);
 #[cfg(feature = "key-value-db")]
 impl_error!(sled::Error, Sled);
+#[cfg(feature = "rpc")]
+impl_error!(bitcoincore_rpc::Error, Rpc);
+#[cfg(feature = "sqlite")]
+impl_error!(rusqlite::Error, Rusqlite);
+
+#[cfg(feature = "compact_filters")]
+impl From<crate::blockchain::compact_filters::CompactFiltersError> for Error {
+    fn from(other: crate::blockchain::compact_filters::CompactFiltersError) -> Self {
+        match other {
+            crate::blockchain::compact_filters::CompactFiltersError::Global(e) => *e,
+            err => Error::CompactFilters(err),
+        }
+    }
+}
+
+#[cfg(feature = "verify")]
+impl From<crate::wallet::verify::VerifyError> for Error {
+    fn from(other: crate::wallet::verify::VerifyError) -> Self {
+        match other {
+            crate::wallet::verify::VerifyError::Global(inner) => *inner,
+            err => Error::Verification(err),
+        }
+    }
+}
+
+#[cfg(feature = "esplora")]
+impl From<crate::blockchain::esplora::EsploraError> for Error {
+    fn from(other: crate::blockchain::esplora::EsploraError) -> Self {
+        Error::Esplora(Box::new(other))
+    }
+}

src/keys/bip39.rs

@@ -0,0 +1,225 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! BIP-0039
+
+// TODO: maybe write our own implementation of bip39? Seems stupid to have an extra dependency for
+// something that should be fairly simple to re-implement.
+
+use bitcoin::bip32;
+use bitcoin::Network;
+
+use miniscript::ScriptContext;
+
+pub use bip39::{Error, Language, Mnemonic};
+
+type Seed = [u8; 64];
+
+/// Type describing entropy length (aka word count) in the mnemonic
+pub enum WordCount {
+    /// 12 words mnemonic (128 bits entropy)
+    Words12 = 128,
+    /// 15 words mnemonic (160 bits entropy)
+    Words15 = 160,
+    /// 18 words mnemonic (192 bits entropy)
+    Words18 = 192,
+    /// 21 words mnemonic (224 bits entropy)
+    Words21 = 224,
+    /// 24 words mnemonic (256 bits entropy)
+    Words24 = 256,
+}
+
+use super::{
+    any_network, DerivableKey, DescriptorKey, ExtendedKey, GeneratableKey, GeneratedKey, KeyError,
+};
+
+fn set_valid_on_any_network<Ctx: ScriptContext>(
+    descriptor_key: DescriptorKey<Ctx>,
+) -> DescriptorKey<Ctx> {
+    // We have to pick one network to build the xprv, but since the bip39 standard doesn't
+    // encode the network, the xprv we create is actually valid everywhere. So we override the
+    // valid networks with `any_network()`.
+    descriptor_key.override_valid_networks(any_network())
+}
+
+/// Type for a BIP39 mnemonic with an optional passphrase
+pub type MnemonicWithPassphrase = (Mnemonic, Option<String>);
+
+#[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
+impl<Ctx: ScriptContext> DerivableKey<Ctx> for Seed {
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        Ok(bip32::ExtendedPrivKey::new_master(Network::Bitcoin, &self[..])?.into())
+    }
+
+    fn into_descriptor_key(
+        self,
+        source: Option<bip32::KeySource>,
+        derivation_path: bip32::DerivationPath,
+    ) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let descriptor_key = self
+            .into_extended_key()?
+            .into_descriptor_key(source, derivation_path)?;
+
+        Ok(set_valid_on_any_network(descriptor_key))
+    }
+}
+
+#[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
+impl<Ctx: ScriptContext> DerivableKey<Ctx> for MnemonicWithPassphrase {
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        let (mnemonic, passphrase) = self;
+        let seed: Seed = mnemonic.to_seed(passphrase.as_deref().unwrap_or(""));
+
+        seed.into_extended_key()
+    }
+
+    fn into_descriptor_key(
+        self,
+        source: Option<bip32::KeySource>,
+        derivation_path: bip32::DerivationPath,
+    ) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let descriptor_key = self
+            .into_extended_key()?
+            .into_descriptor_key(source, derivation_path)?;
+
+        Ok(set_valid_on_any_network(descriptor_key))
+    }
+}
+
+#[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
+impl<Ctx: ScriptContext> DerivableKey<Ctx> for (GeneratedKey<Mnemonic, Ctx>, Option<String>) {
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        let (mnemonic, passphrase) = self;
+        (mnemonic.into_key(), passphrase).into_extended_key()
+    }
+
+    fn into_descriptor_key(
+        self,
+        source: Option<bip32::KeySource>,
+        derivation_path: bip32::DerivationPath,
+    ) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let (mnemonic, passphrase) = self;
+        (mnemonic.into_key(), passphrase).into_descriptor_key(source, derivation_path)
+    }
+}
+
+#[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
+impl<Ctx: ScriptContext> DerivableKey<Ctx> for Mnemonic {
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        (self, None).into_extended_key()
+    }
+
+    fn into_descriptor_key(
+        self,
+        source: Option<bip32::KeySource>,
+        derivation_path: bip32::DerivationPath,
+    ) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let descriptor_key = self
+            .into_extended_key()?
+            .into_descriptor_key(source, derivation_path)?;
+
+        Ok(set_valid_on_any_network(descriptor_key))
+    }
+}
+
+#[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
+impl<Ctx: ScriptContext> GeneratableKey<Ctx> for Mnemonic {
+    type Entropy = [u8; 32];
+
+    type Options = (WordCount, Language);
+    type Error = Option<bip39::Error>;
+
+    fn generate_with_entropy(
+        (word_count, language): Self::Options,
+        entropy: Self::Entropy,
+    ) -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
+        let entropy = &entropy[..(word_count as usize / 8)];
+        let mnemonic = Mnemonic::from_entropy_in(language, entropy)?;
+
+        Ok(GeneratedKey::new(mnemonic, any_network()))
+    }
+}
+
+#[cfg(test)]
+mod test {
+    use std::str::FromStr;
+
+    use bitcoin::bip32;
+
+    use bip39::{Language, Mnemonic};
+
+    use crate::keys::{any_network, GeneratableKey, GeneratedKey};
+
+    use super::WordCount;
+
+    #[test]
+    fn test_keys_bip39_mnemonic() {
+        let mnemonic =
+            "aim bunker wash balance finish force paper analyst cabin spoon stable organ";
+        let mnemonic = Mnemonic::parse_in(Language::English, mnemonic).unwrap();
+        let path = bip32::DerivationPath::from_str("m/44'/0'/0'/0").unwrap();
+
+        let key = (mnemonic, path);
+        let (desc, keys, networks) = crate::descriptor!(wpkh(key)).unwrap();
+        assert_eq!(desc.to_string(), "wpkh([be83839f/44'/0'/0']xpub6DCQ1YcqvZtSwGWMrwHELPehjWV3f2MGZ69yBADTxFEUAoLwb5Mp5GniQK6tTp3AgbngVz9zEFbBJUPVnkG7LFYt8QMTfbrNqs6FNEwAPKA/0/*)#0r8v4nkv");
+        assert_eq!(keys.len(), 1);
+        assert_eq!(networks.len(), 4);
+    }
+
+    #[test]
+    fn test_keys_bip39_mnemonic_passphrase() {
+        let mnemonic =
+            "aim bunker wash balance finish force paper analyst cabin spoon stable organ";
+        let mnemonic = Mnemonic::parse_in(Language::English, mnemonic).unwrap();
+        let path = bip32::DerivationPath::from_str("m/44'/0'/0'/0").unwrap();
+
+        let key = ((mnemonic, Some("passphrase".into())), path);
+        let (desc, keys, networks) = crate::descriptor!(wpkh(key)).unwrap();
+        assert_eq!(desc.to_string(), "wpkh([8f6cb80c/44'/0'/0']xpub6DWYS8bbihFevy29M4cbw4ZR3P5E12jB8R88gBDWCTCNpYiDHhYWNywrCF9VZQYagzPmsZpxXpytzSoxynyeFr4ZyzheVjnpLKuse4fiwZw/0/*)#h0j0tg5m");
+        assert_eq!(keys.len(), 1);
+        assert_eq!(networks.len(), 4);
+    }
+
+    #[test]
+    fn test_keys_generate_bip39() {
+        let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
+            Mnemonic::generate_with_entropy(
+                (WordCount::Words12, Language::English),
+                crate::keys::test::TEST_ENTROPY,
+            )
+            .unwrap();
+        assert_eq!(generated_mnemonic.valid_networks, any_network());
+        assert_eq!(
+            generated_mnemonic.to_string(),
+            "primary fetch primary fetch primary fetch primary fetch primary fetch primary fever"
+        );
+
+        let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
+            Mnemonic::generate_with_entropy(
+                (WordCount::Words24, Language::English),
+                crate::keys::test::TEST_ENTROPY,
+            )
+            .unwrap();
+        assert_eq!(generated_mnemonic.valid_networks, any_network());
+        assert_eq!(generated_mnemonic.to_string(), "primary fetch primary fetch primary fetch primary fetch primary fetch primary fetch primary fetch primary fetch primary fetch primary fetch primary fetch primary foster");
+    }
+
+    #[test]
+    fn test_keys_generate_bip39_random() {
+        let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
+            Mnemonic::generate((WordCount::Words12, Language::English)).unwrap();
+        assert_eq!(generated_mnemonic.valid_networks, any_network());
+
+        let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
+            Mnemonic::generate((WordCount::Words24, Language::English)).unwrap();
+        assert_eq!(generated_mnemonic.valid_networks, any_network());
+    }
+}

src/keys/mod.rs

@@ -0,0 +1,994 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Key formats
+
+use std::any::TypeId;
+use std::collections::HashSet;
+use std::marker::PhantomData;
+use std::ops::Deref;
+use std::str::FromStr;
+
+use bitcoin::secp256k1::{self, Secp256k1, Signing};
+
+use bitcoin::bip32;
+use bitcoin::{key::XOnlyPublicKey, Network, PrivateKey, PublicKey};
+
+use miniscript::descriptor::{Descriptor, DescriptorXKey, Wildcard};
+pub use miniscript::descriptor::{
+    DescriptorPublicKey, DescriptorSecretKey, KeyMap, SinglePriv, SinglePub, SinglePubKey,
+    SortedMultiVec,
+};
+pub use miniscript::ScriptContext;
+use miniscript::{Miniscript, Terminal};
+
+use crate::descriptor::{CheckMiniscript, DescriptorError};
+use crate::wallet::utils::SecpCtx;
+
+#[cfg(feature = "keys-bip39")]
+#[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
+pub mod bip39;
+
+/// Set of valid networks for a key
+pub type ValidNetworks = HashSet<Network>;
+
+/// Create a set containing mainnet, testnet, signet, and regtest
+pub fn any_network() -> ValidNetworks {
+    vec![
+        Network::Bitcoin,
+        Network::Testnet,
+        Network::Regtest,
+        Network::Signet,
+    ]
+    .into_iter()
+    .collect()
+}
+/// Create a set only containing mainnet
+pub fn mainnet_network() -> ValidNetworks {
+    vec![Network::Bitcoin].into_iter().collect()
+}
+/// Create a set containing testnet and regtest
+pub fn test_networks() -> ValidNetworks {
+    vec![Network::Testnet, Network::Regtest, Network::Signet]
+        .into_iter()
+        .collect()
+}
+/// Compute the intersection of two sets
+pub fn merge_networks(a: &ValidNetworks, b: &ValidNetworks) -> ValidNetworks {
+    a.intersection(b).cloned().collect()
+}
+
+/// Container for public or secret keys
+#[derive(Debug)]
+pub enum DescriptorKey<Ctx: ScriptContext> {
+    #[doc(hidden)]
+    Public(DescriptorPublicKey, ValidNetworks, PhantomData<Ctx>),
+    #[doc(hidden)]
+    Secret(DescriptorSecretKey, ValidNetworks, PhantomData<Ctx>),
+}
+
+impl<Ctx: ScriptContext> DescriptorKey<Ctx> {
+    /// Create an instance given a public key and a set of valid networks
+    pub fn from_public(public: DescriptorPublicKey, networks: ValidNetworks) -> Self {
+        DescriptorKey::Public(public, networks, PhantomData)
+    }
+
+    /// Create an instance given a secret key and a set of valid networks
+    pub fn from_secret(secret: DescriptorSecretKey, networks: ValidNetworks) -> Self {
+        DescriptorKey::Secret(secret, networks, PhantomData)
+    }
+
+    /// Override the computed set of valid networks
+    pub fn override_valid_networks(self, networks: ValidNetworks) -> Self {
+        match self {
+            DescriptorKey::Public(key, _, _) => DescriptorKey::Public(key, networks, PhantomData),
+            DescriptorKey::Secret(key, _, _) => DescriptorKey::Secret(key, networks, PhantomData),
+        }
+    }
+
+    // This method is used internally by `bdk::fragment!` and `bdk::descriptor!`. It has to be
+    // public because it is effectively called by external crates once the macros are expanded,
+    // but since it is not meant to be part of the public api we hide it from the docs.
+    #[doc(hidden)]
+    pub fn extract(
+        self,
+        secp: &SecpCtx,
+    ) -> Result<(DescriptorPublicKey, KeyMap, ValidNetworks), KeyError> {
+        match self {
+            DescriptorKey::Public(public, valid_networks, _) => {
+                Ok((public, KeyMap::default(), valid_networks))
+            }
+            DescriptorKey::Secret(secret, valid_networks, _) => {
+                let mut key_map = KeyMap::with_capacity(1);
+
+                let public = secret
+                    .to_public(secp)
+                    .map_err(|e| miniscript::Error::Unexpected(e.to_string()))?;
+                key_map.insert(public.clone(), secret);
+
+                Ok((public, key_map, valid_networks))
+            }
+        }
+    }
+}
+
+/// Enum representation of the known valid [`ScriptContext`]s
+#[derive(Debug, Eq, PartialEq, Copy, Clone)]
+pub enum ScriptContextEnum {
+    /// Legacy scripts
+    Legacy,
+    /// Segwitv0 scripts
+    Segwitv0,
+    /// Taproot scripts
+    Tap,
+}
+
+impl ScriptContextEnum {
+    /// Returns whether the script context is [`ScriptContextEnum::Legacy`]
+    pub fn is_legacy(&self) -> bool {
+        self == &ScriptContextEnum::Legacy
+    }
+
+    /// Returns whether the script context is [`ScriptContextEnum::Segwitv0`]
+    pub fn is_segwit_v0(&self) -> bool {
+        self == &ScriptContextEnum::Segwitv0
+    }
+
+    /// Returns whether the script context is [`ScriptContextEnum::Tap`]
+    pub fn is_taproot(&self) -> bool {
+        self == &ScriptContextEnum::Tap
+    }
+}
+
+/// Trait that adds extra useful methods to [`ScriptContext`]s
+pub trait ExtScriptContext: ScriptContext {
+    /// Returns the [`ScriptContext`] as a [`ScriptContextEnum`]
+    fn as_enum() -> ScriptContextEnum;
+
+    /// Returns whether the script context is [`Legacy`](miniscript::Legacy)
+    fn is_legacy() -> bool {
+        Self::as_enum().is_legacy()
+    }
+
+    /// Returns whether the script context is [`Segwitv0`](miniscript::Segwitv0)
+    fn is_segwit_v0() -> bool {
+        Self::as_enum().is_segwit_v0()
+    }
+
+    /// Returns whether the script context is [`Tap`](miniscript::Tap), aka Taproot or Segwit V1
+    fn is_taproot() -> bool {
+        Self::as_enum().is_taproot()
+    }
+}
+
+impl<Ctx: ScriptContext + 'static> ExtScriptContext for Ctx {
+    fn as_enum() -> ScriptContextEnum {
+        match TypeId::of::<Ctx>() {
+            t if t == TypeId::of::<miniscript::Legacy>() => ScriptContextEnum::Legacy,
+            t if t == TypeId::of::<miniscript::Segwitv0>() => ScriptContextEnum::Segwitv0,
+            t if t == TypeId::of::<miniscript::Tap>() => ScriptContextEnum::Tap,
+            _ => unimplemented!("Unknown ScriptContext type"),
+        }
+    }
+}
+
+/// Trait for objects that can be turned into a public or secret [`DescriptorKey`]
+///
+/// The generic type `Ctx` is used to define the context in which the key is valid: some key
+/// formats, like the mnemonics used by Electrum wallets, encode internally whether the wallet is
+/// legacy or segwit. Thus, trying to turn a valid legacy mnemonic into a `DescriptorKey`
+/// that would become part of a segwit descriptor should fail.
+///
+/// For key types that do care about this, the [`ExtScriptContext`] trait provides some useful
+/// methods that can be used to check at runtime which `Ctx` is being used.
+///
+/// For key types that can do this check statically (because they can only work within a
+/// single `Ctx`), the "specialized" trait can be implemented to make the compiler handle the type
+/// checking.
+///
+/// Keys also have control over the networks they support: constructing the return object with
+/// [`DescriptorKey::from_public`] or [`DescriptorKey::from_secret`] allows to specify a set of
+/// [`ValidNetworks`].
+///
+/// ## Examples
+///
+/// Key type valid in any context:
+///
+/// ```
+/// use bdk::bitcoin::PublicKey;
+///
+/// use bdk::keys::{DescriptorKey, IntoDescriptorKey, KeyError, ScriptContext};
+///
+/// pub struct MyKeyType {
+///     pubkey: PublicKey,
+/// }
+///
+/// impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for MyKeyType {
+///     fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+///         self.pubkey.into_descriptor_key()
+///     }
+/// }
+/// ```
+///
+/// Key type that is only valid on mainnet:
+///
+/// ```
+/// use bdk::bitcoin::PublicKey;
+///
+/// use bdk::keys::{
+///     mainnet_network, DescriptorKey, DescriptorPublicKey, IntoDescriptorKey, KeyError,
+///     ScriptContext, SinglePub, SinglePubKey,
+/// };
+///
+/// pub struct MyKeyType {
+///     pubkey: PublicKey,
+/// }
+///
+/// impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for MyKeyType {
+///     fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+///         Ok(DescriptorKey::from_public(
+///             DescriptorPublicKey::Single(SinglePub {
+///                 origin: None,
+///                 key: SinglePubKey::FullKey(self.pubkey),
+///             }),
+///             mainnet_network(),
+///         ))
+///     }
+/// }
+/// ```
+///
+/// Key type that internally encodes in which context it's valid. The context is checked at runtime:
+///
+/// ```
+/// use bdk::bitcoin::PublicKey;
+///
+/// use bdk::keys::{DescriptorKey, ExtScriptContext, IntoDescriptorKey, KeyError, ScriptContext};
+///
+/// pub struct MyKeyType {
+///     is_legacy: bool,
+///     pubkey: PublicKey,
+/// }
+///
+/// impl<Ctx: ScriptContext + 'static> IntoDescriptorKey<Ctx> for MyKeyType {
+///     fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+///         if Ctx::is_legacy() == self.is_legacy {
+///             self.pubkey.into_descriptor_key()
+///         } else {
+///             Err(KeyError::InvalidScriptContext)
+///         }
+///     }
+/// }
+/// ```
+///
+/// Key type that can only work within [`miniscript::Segwitv0`] context. Only the specialized version
+/// of the trait is implemented.
+///
+/// This example deliberately fails to compile, to demonstrate how the compiler can catch when keys
+/// are misused. In this case, the "segwit-only" key is used to build a `pkh()` descriptor, which
+/// makes the compiler (correctly) fail.
+///
+/// ```compile_fail
+/// use bdk::bitcoin::PublicKey;
+/// use std::str::FromStr;
+///
+/// use bdk::keys::{DescriptorKey, IntoDescriptorKey, KeyError};
+///
+/// pub struct MySegwitOnlyKeyType {
+///     pubkey: PublicKey,
+/// }
+///
+/// impl IntoDescriptorKey<bdk::miniscript::Segwitv0> for MySegwitOnlyKeyType {
+///     fn into_descriptor_key(self) -> Result<DescriptorKey<bdk::miniscript::Segwitv0>, KeyError> {
+///         self.pubkey.into_descriptor_key()
+///     }
+/// }
+///
+/// let key = MySegwitOnlyKeyType {
+///     pubkey: PublicKey::from_str("...")?,
+/// };
+/// let (descriptor, _, _) = bdk::descriptor!(pkh(key))?;
+/// //                                       ^^^^^ changing this to `wpkh` would make it compile
+///
+/// # Ok::<_, Box<dyn std::error::Error>>(())
+/// ```
+pub trait IntoDescriptorKey<Ctx: ScriptContext>: Sized {
+    /// Turn the key into a [`DescriptorKey`] within the requested [`ScriptContext`]
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError>;
+}
+
+/// Enum for extended keys that can be either `xprv` or `xpub`
+///
+/// An instance of [`ExtendedKey`] can be constructed from an [`ExtendedPrivKey`](bip32::ExtendedPrivKey)
+/// or an [`ExtendedPubKey`](bip32::ExtendedPubKey) by using the `From` trait.
+///
+/// Defaults to the [`Legacy`](miniscript::Legacy) context.
+pub enum ExtendedKey<Ctx: ScriptContext = miniscript::Legacy> {
+    /// A private extended key, aka an `xprv`
+    Private((bip32::ExtendedPrivKey, PhantomData<Ctx>)),
+    /// A public extended key, aka an `xpub`
+    Public((bip32::ExtendedPubKey, PhantomData<Ctx>)),
+}
+
+impl<Ctx: ScriptContext> ExtendedKey<Ctx> {
+    /// Return whether or not the key contains the private data
+    pub fn has_secret(&self) -> bool {
+        match self {
+            ExtendedKey::Private(_) => true,
+            ExtendedKey::Public(_) => false,
+        }
+    }
+
+    /// Transform the [`ExtendedKey`] into an [`ExtendedPrivKey`](bip32::ExtendedPrivKey) for the
+    /// given [`Network`], if the key contains the private data
+    pub fn into_xprv(self, network: Network) -> Option<bip32::ExtendedPrivKey> {
+        match self {
+            ExtendedKey::Private((mut xprv, _)) => {
+                xprv.network = network;
+                Some(xprv)
+            }
+            ExtendedKey::Public(_) => None,
+        }
+    }
+
+    /// Transform the [`ExtendedKey`] into an [`ExtendedPubKey`](bip32::ExtendedPubKey) for the
+    /// given [`Network`]
+    pub fn into_xpub<C: Signing>(
+        self,
+        network: bitcoin::Network,
+        secp: &Secp256k1<C>,
+    ) -> bip32::ExtendedPubKey {
+        let mut xpub = match self {
+            ExtendedKey::Private((xprv, _)) => bip32::ExtendedPubKey::from_priv(secp, &xprv),
+            ExtendedKey::Public((xpub, _)) => xpub,
+        };
+
+        xpub.network = network;
+        xpub
+    }
+}
+
+impl<Ctx: ScriptContext> From<bip32::ExtendedPubKey> for ExtendedKey<Ctx> {
+    fn from(xpub: bip32::ExtendedPubKey) -> Self {
+        ExtendedKey::Public((xpub, PhantomData))
+    }
+}
+
+impl<Ctx: ScriptContext> From<bip32::ExtendedPrivKey> for ExtendedKey<Ctx> {
+    fn from(xprv: bip32::ExtendedPrivKey) -> Self {
+        ExtendedKey::Private((xprv, PhantomData))
+    }
+}
+
+/// Trait for keys that can be derived.
+///
+/// When extra metadata are provided, a [`DerivableKey`] can be transformed into a
+/// [`DescriptorKey`]: the trait [`IntoDescriptorKey`] is automatically implemented
+/// for `(DerivableKey, DerivationPath)` and
+/// `(DerivableKey, KeySource, DerivationPath)` tuples.
+///
+/// For key types that don't encode any indication about the path to use (like bip39), it's
+/// generally recommended to implement this trait instead of [`IntoDescriptorKey`]. The same
+/// rules regarding script context and valid networks apply.
+///
+/// ## Examples
+///
+/// Key types that can be directly converted into an [`ExtendedPrivKey`] or
+/// an [`ExtendedPubKey`] can implement only the required `into_extended_key()` method.
+///
+/// ```
+/// use bdk::bitcoin;
+/// use bdk::bitcoin::bip32;
+/// use bdk::keys::{DerivableKey, ExtendedKey, KeyError, ScriptContext};
+///
+/// struct MyCustomKeyType {
+///     key_data: bitcoin::PrivateKey,
+///     chain_code: [u8; 32],
+///     network: bitcoin::Network,
+/// }
+///
+/// impl<Ctx: ScriptContext> DerivableKey<Ctx> for MyCustomKeyType {
+///     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+///         let xprv = bip32::ExtendedPrivKey {
+///             network: self.network,
+///             depth: 0,
+///             parent_fingerprint: bip32::Fingerprint::default(),
+///             private_key: self.key_data.inner,
+///             chain_code: bip32::ChainCode::from(&self.chain_code),
+///             child_number: bip32::ChildNumber::Normal { index: 0 },
+///         };
+///
+///         xprv.into_extended_key()
+///     }
+/// }
+/// ```
+///
+/// Types that don't internally encode the [`Network`](bitcoin::Network) in which they are valid need some extra
+/// steps to override the set of valid networks, otherwise only the network specified in the
+/// [`ExtendedPrivKey`] or [`ExtendedPubKey`] will be considered valid.
+///
+/// ```
+/// use bdk::bitcoin;
+/// use bdk::bitcoin::bip32;
+/// use bdk::keys::{
+///     any_network, DerivableKey, DescriptorKey, ExtendedKey, KeyError, ScriptContext,
+/// };
+///
+/// struct MyCustomKeyType {
+///     key_data: bitcoin::PrivateKey,
+///     chain_code: [u8; 32],
+/// }
+///
+/// impl<Ctx: ScriptContext> DerivableKey<Ctx> for MyCustomKeyType {
+///     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+///         let xprv = bip32::ExtendedPrivKey {
+///             network: bitcoin::Network::Bitcoin, // pick an arbitrary network here
+///             depth: 0,
+///             parent_fingerprint: bip32::Fingerprint::default(),
+///             private_key: self.key_data.inner,
+///             chain_code: bip32::ChainCode::from(&self.chain_code),
+///             child_number: bip32::ChildNumber::Normal { index: 0 },
+///         };
+///
+///         xprv.into_extended_key()
+///     }
+///
+///     fn into_descriptor_key(
+///         self,
+///         source: Option<bip32::KeySource>,
+///         derivation_path: bip32::DerivationPath,
+///     ) -> Result<DescriptorKey<Ctx>, KeyError> {
+///         let descriptor_key = self
+///             .into_extended_key()?
+///             .into_descriptor_key(source, derivation_path)?;
+///
+///         // Override the set of valid networks here
+///         Ok(descriptor_key.override_valid_networks(any_network()))
+///     }
+/// }
+/// ```
+///
+/// [`DerivationPath`]: (bip32::DerivationPath)
+/// [`ExtendedPrivKey`]: (bip32::ExtendedPrivKey)
+/// [`ExtendedPubKey`]: (bip32::ExtendedPubKey)
+pub trait DerivableKey<Ctx: ScriptContext = miniscript::Legacy>: Sized {
+    /// Consume `self` and turn it into an [`ExtendedKey`]
+    ///
+    /// This can be used to get direct access to `xprv`s and `xpub`s for types that implement this trait,
+    /// like [`Mnemonic`](bip39::Mnemonic) when the `keys-bip39` feature is enabled.
+    #[cfg_attr(
+        feature = "keys-bip39",
+        doc = r##"
+```rust
+use bdk::bitcoin::Network;
+use bdk::keys::{DerivableKey, ExtendedKey};
+use bdk::keys::bip39::{Mnemonic, Language};
+
+# fn main() -> Result<(), Box<dyn std::error::Error>> {
+let xkey: ExtendedKey =
+    Mnemonic::parse_in(
+        Language::English,
+        "jelly crash boy whisper mouse ecology tuna soccer memory million news short",
+    )?
+    .into_extended_key()?;
+let xprv = xkey.into_xprv(Network::Bitcoin).unwrap();
+# Ok(()) }
+```
+"##
+    )]
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError>;
+
+    /// Consume `self` and turn it into a [`DescriptorKey`] by adding the extra metadata, such as
+    /// key origin and derivation path
+    fn into_descriptor_key(
+        self,
+        origin: Option<bip32::KeySource>,
+        derivation_path: bip32::DerivationPath,
+    ) -> Result<DescriptorKey<Ctx>, KeyError> {
+        match self.into_extended_key()? {
+            ExtendedKey::Private((xprv, _)) => DescriptorSecretKey::XPrv(DescriptorXKey {
+                origin,
+                xkey: xprv,
+                derivation_path,
+                wildcard: Wildcard::Unhardened,
+            })
+            .into_descriptor_key(),
+            ExtendedKey::Public((xpub, _)) => DescriptorPublicKey::XPub(DescriptorXKey {
+                origin,
+                xkey: xpub,
+                derivation_path,
+                wildcard: Wildcard::Unhardened,
+            })
+            .into_descriptor_key(),
+        }
+    }
+}
+
+/// Identity conversion
+impl<Ctx: ScriptContext> DerivableKey<Ctx> for ExtendedKey<Ctx> {
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        Ok(self)
+    }
+}
+
+impl<Ctx: ScriptContext> DerivableKey<Ctx> for bip32::ExtendedPubKey {
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        Ok(self.into())
+    }
+}
+
+impl<Ctx: ScriptContext> DerivableKey<Ctx> for bip32::ExtendedPrivKey {
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        Ok(self.into())
+    }
+}
+
+/// Output of a [`GeneratableKey`] key generation
+pub struct GeneratedKey<K, Ctx: ScriptContext> {
+    key: K,
+    valid_networks: ValidNetworks,
+    phantom: PhantomData<Ctx>,
+}
+
+impl<K, Ctx: ScriptContext> GeneratedKey<K, Ctx> {
+    fn new(key: K, valid_networks: ValidNetworks) -> Self {
+        GeneratedKey {
+            key,
+            valid_networks,
+            phantom: PhantomData,
+        }
+    }
+
+    /// Consumes `self` and returns the key
+    pub fn into_key(self) -> K {
+        self.key
+    }
+}
+
+impl<K, Ctx: ScriptContext> Deref for GeneratedKey<K, Ctx> {
+    type Target = K;
+
+    fn deref(&self) -> &Self::Target {
+        &self.key
+    }
+}
+
+impl<K: Clone, Ctx: ScriptContext> Clone for GeneratedKey<K, Ctx> {
+    fn clone(&self) -> GeneratedKey<K, Ctx> {
+        GeneratedKey {
+            key: self.key.clone(),
+            valid_networks: self.valid_networks.clone(),
+            phantom: self.phantom,
+        }
+    }
+}
+
+// Make generated "derivable" keys themselves "derivable". Also make sure they are assigned the
+// right `valid_networks`.
+impl<Ctx, K> DerivableKey<Ctx> for GeneratedKey<K, Ctx>
+where
+    Ctx: ScriptContext,
+    K: DerivableKey<Ctx>,
+{
+    fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
+        self.key.into_extended_key()
+    }
+
+    fn into_descriptor_key(
+        self,
+        origin: Option<bip32::KeySource>,
+        derivation_path: bip32::DerivationPath,
+    ) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let descriptor_key = self.key.into_descriptor_key(origin, derivation_path)?;
+        Ok(descriptor_key.override_valid_networks(self.valid_networks))
+    }
+}
+
+// Make generated keys directly usable in descriptors, and make sure they get assigned the right
+// `valid_networks`.
+impl<Ctx, K> IntoDescriptorKey<Ctx> for GeneratedKey<K, Ctx>
+where
+    Ctx: ScriptContext,
+    K: IntoDescriptorKey<Ctx>,
+{
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let desc_key = self.key.into_descriptor_key()?;
+        Ok(desc_key.override_valid_networks(self.valid_networks))
+    }
+}
+
+/// Trait for keys that can be generated
+///
+/// The same rules about [`ScriptContext`] and [`ValidNetworks`] from [`IntoDescriptorKey`] apply.
+///
+/// This trait is particularly useful when combined with [`DerivableKey`]: if `Self`
+/// implements it, the returned [`GeneratedKey`] will also implement it. The same is true for
+/// [`IntoDescriptorKey`]: the generated keys can be directly used in descriptors if `Self` is also
+/// [`IntoDescriptorKey`].
+pub trait GeneratableKey<Ctx: ScriptContext>: Sized {
+    /// Type specifying the amount of entropy required e.g. `[u8;32]`
+    type Entropy: AsMut<[u8]> + Default;
+
+    /// Extra options required by the `generate_with_entropy`
+    type Options;
+    /// Returned error in case of failure
+    type Error: std::fmt::Debug;
+
+    /// Generate a key given the extra options and the entropy
+    fn generate_with_entropy(
+        options: Self::Options,
+        entropy: Self::Entropy,
+    ) -> Result<GeneratedKey<Self, Ctx>, Self::Error>;
+
+    /// Generate a key given the options with a random entropy
+    fn generate(options: Self::Options) -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
+        use rand::{thread_rng, Rng};
+
+        let mut entropy = Self::Entropy::default();
+        thread_rng().fill(entropy.as_mut());
+        Self::generate_with_entropy(options, entropy)
+    }
+}
+
+/// Trait that allows generating a key with the default options
+///
+/// This trait is automatically implemented if the [`GeneratableKey::Options`] implements [`Default`].
+pub trait GeneratableDefaultOptions<Ctx>: GeneratableKey<Ctx>
+where
+    Ctx: ScriptContext,
+    <Self as GeneratableKey<Ctx>>::Options: Default,
+{
+    /// Generate a key with the default options and a given entropy
+    fn generate_with_entropy_default(
+        entropy: Self::Entropy,
+    ) -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
+        Self::generate_with_entropy(Default::default(), entropy)
+    }
+
+    /// Generate a key with the default options and a random entropy
+    fn generate_default() -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
+        Self::generate(Default::default())
+    }
+}
+
+/// Automatic implementation of [`GeneratableDefaultOptions`] for [`GeneratableKey`]s where
+/// `Options` implements `Default`
+impl<Ctx, K> GeneratableDefaultOptions<Ctx> for K
+where
+    Ctx: ScriptContext,
+    K: GeneratableKey<Ctx>,
+    <K as GeneratableKey<Ctx>>::Options: Default,
+{
+}
+
+impl<Ctx: ScriptContext> GeneratableKey<Ctx> for bip32::ExtendedPrivKey {
+    type Entropy = [u8; 32];
+
+    type Options = ();
+    type Error = bip32::Error;
+
+    fn generate_with_entropy(
+        _: Self::Options,
+        entropy: Self::Entropy,
+    ) -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
+        // pick a arbitrary network here, but say that we support all of them
+        let xprv = bip32::ExtendedPrivKey::new_master(Network::Bitcoin, entropy.as_ref())?;
+        Ok(GeneratedKey::new(xprv, any_network()))
+    }
+}
+
+/// Options for generating a [`PrivateKey`]
+///
+/// Defaults to creating compressed keys, which save on-chain bytes and fees
+#[derive(Debug, Copy, Clone)]
+pub struct PrivateKeyGenerateOptions {
+    /// Whether the generated key should be "compressed" or not
+    pub compressed: bool,
+}
+
+impl Default for PrivateKeyGenerateOptions {
+    fn default() -> Self {
+        PrivateKeyGenerateOptions { compressed: true }
+    }
+}
+
+impl<Ctx: ScriptContext> GeneratableKey<Ctx> for PrivateKey {
+    type Entropy = [u8; secp256k1::constants::SECRET_KEY_SIZE];
+
+    type Options = PrivateKeyGenerateOptions;
+    type Error = bip32::Error;
+
+    fn generate_with_entropy(
+        options: Self::Options,
+        entropy: Self::Entropy,
+    ) -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
+        // pick a arbitrary network here, but say that we support all of them
+        let inner = secp256k1::SecretKey::from_slice(&entropy)?;
+        let private_key = PrivateKey {
+            compressed: options.compressed,
+            network: Network::Bitcoin,
+            inner,
+        };
+
+        Ok(GeneratedKey::new(private_key, any_network()))
+    }
+}
+
+impl<Ctx: ScriptContext, T: DerivableKey<Ctx>> IntoDescriptorKey<Ctx>
+    for (T, bip32::DerivationPath)
+{
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        self.0.into_descriptor_key(None, self.1)
+    }
+}
+
+impl<Ctx: ScriptContext, T: DerivableKey<Ctx>> IntoDescriptorKey<Ctx>
+    for (T, bip32::KeySource, bip32::DerivationPath)
+{
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        self.0.into_descriptor_key(Some(self.1), self.2)
+    }
+}
+
+fn expand_multi_keys<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
+    pks: Vec<Pk>,
+    secp: &SecpCtx,
+) -> Result<(Vec<DescriptorPublicKey>, KeyMap, ValidNetworks), KeyError> {
+    let (pks, key_maps_networks): (Vec<_>, Vec<_>) = pks
+        .into_iter()
+        .map(|key| key.into_descriptor_key()?.extract(secp))
+        .collect::<Result<Vec<_>, _>>()?
+        .into_iter()
+        .map(|(a, b, c)| (a, (b, c)))
+        .unzip();
+
+    let (key_map, valid_networks) = key_maps_networks.into_iter().fold(
+        (KeyMap::default(), any_network()),
+        |(mut keys_acc, net_acc), (key, net)| {
+            keys_acc.extend(key.into_iter());
+            let net_acc = merge_networks(&net_acc, &net);
+
+            (keys_acc, net_acc)
+        },
+    );
+
+    Ok((pks, key_map, valid_networks))
+}
+
+// Used internally by `bdk::fragment!` to build `pk_k()` fragments
+#[doc(hidden)]
+pub fn make_pk<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
+    descriptor_key: Pk,
+    secp: &SecpCtx,
+) -> Result<(Miniscript<DescriptorPublicKey, Ctx>, KeyMap, ValidNetworks), DescriptorError> {
+    let (key, key_map, valid_networks) = descriptor_key.into_descriptor_key()?.extract(secp)?;
+    let minisc = Miniscript::from_ast(Terminal::PkK(key))?;
+
+    minisc.check_miniscript()?;
+
+    Ok((minisc, key_map, valid_networks))
+}
+
+// Used internally by `bdk::fragment!` to build `pk_h()` fragments
+#[doc(hidden)]
+pub fn make_pkh<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
+    descriptor_key: Pk,
+    secp: &SecpCtx,
+) -> Result<(Miniscript<DescriptorPublicKey, Ctx>, KeyMap, ValidNetworks), DescriptorError> {
+    let (key, key_map, valid_networks) = descriptor_key.into_descriptor_key()?.extract(secp)?;
+    let minisc = Miniscript::from_ast(Terminal::PkH(key))?;
+
+    minisc.check_miniscript()?;
+
+    Ok((minisc, key_map, valid_networks))
+}
+
+// Used internally by `bdk::fragment!` to build `multi()` fragments
+#[doc(hidden)]
+pub fn make_multi<
+    Pk: IntoDescriptorKey<Ctx>,
+    Ctx: ScriptContext,
+    V: Fn(usize, Vec<DescriptorPublicKey>) -> Terminal<DescriptorPublicKey, Ctx>,
+>(
+    thresh: usize,
+    variant: V,
+    pks: Vec<Pk>,
+    secp: &SecpCtx,
+) -> Result<(Miniscript<DescriptorPublicKey, Ctx>, KeyMap, ValidNetworks), DescriptorError> {
+    let (pks, key_map, valid_networks) = expand_multi_keys(pks, secp)?;
+    let minisc = Miniscript::from_ast(variant(thresh, pks))?;
+
+    minisc.check_miniscript()?;
+
+    Ok((minisc, key_map, valid_networks))
+}
+
+// Used internally by `bdk::descriptor!` to build `sortedmulti()` fragments
+#[doc(hidden)]
+pub fn make_sortedmulti<Pk, Ctx, F>(
+    thresh: usize,
+    pks: Vec<Pk>,
+    build_desc: F,
+    secp: &SecpCtx,
+) -> Result<(Descriptor<DescriptorPublicKey>, KeyMap, ValidNetworks), DescriptorError>
+where
+    Pk: IntoDescriptorKey<Ctx>,
+    Ctx: ScriptContext,
+    F: Fn(
+        usize,
+        Vec<DescriptorPublicKey>,
+    ) -> Result<(Descriptor<DescriptorPublicKey>, PhantomData<Ctx>), DescriptorError>,
+{
+    let (pks, key_map, valid_networks) = expand_multi_keys(pks, secp)?;
+    let descriptor = build_desc(thresh, pks)?.0;
+
+    Ok((descriptor, key_map, valid_networks))
+}
+
+/// The "identity" conversion is used internally by some `bdk::fragment`s
+impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for DescriptorKey<Ctx> {
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        Ok(self)
+    }
+}
+
+impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for DescriptorPublicKey {
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let networks = match self {
+            DescriptorPublicKey::Single(_) => any_network(),
+            DescriptorPublicKey::XPub(DescriptorXKey { xkey, .. })
+                if xkey.network == Network::Bitcoin =>
+            {
+                mainnet_network()
+            }
+            _ => test_networks(),
+        };
+
+        Ok(DescriptorKey::from_public(self, networks))
+    }
+}
+
+impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for PublicKey {
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        DescriptorPublicKey::Single(SinglePub {
+            key: SinglePubKey::FullKey(self),
+            origin: None,
+        })
+        .into_descriptor_key()
+    }
+}
+
+impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for XOnlyPublicKey {
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        DescriptorPublicKey::Single(SinglePub {
+            key: SinglePubKey::XOnly(self),
+            origin: None,
+        })
+        .into_descriptor_key()
+    }
+}
+
+impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for DescriptorSecretKey {
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        let networks = match &self {
+            DescriptorSecretKey::Single(sk) if sk.key.network == Network::Bitcoin => {
+                mainnet_network()
+            }
+            DescriptorSecretKey::XPrv(DescriptorXKey { xkey, .. })
+                if xkey.network == Network::Bitcoin =>
+            {
+                mainnet_network()
+            }
+            _ => test_networks(),
+        };
+
+        Ok(DescriptorKey::from_secret(self, networks))
+    }
+}
+
+impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for &'_ str {
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        DescriptorSecretKey::from_str(self)
+            .map_err(|e| KeyError::Message(e.to_string()))?
+            .into_descriptor_key()
+    }
+}
+
+impl<Ctx: ScriptContext> IntoDescriptorKey<Ctx> for PrivateKey {
+    fn into_descriptor_key(self) -> Result<DescriptorKey<Ctx>, KeyError> {
+        DescriptorSecretKey::Single(SinglePriv {
+            key: self,
+            origin: None,
+        })
+        .into_descriptor_key()
+    }
+}
+
+/// Errors thrown while working with [`keys`](crate::keys)
+#[derive(Debug)]
+pub enum KeyError {
+    /// The key cannot exist in the given script context
+    InvalidScriptContext,
+    /// The key is not valid for the given network
+    InvalidNetwork,
+    /// The key has an invalid checksum
+    InvalidChecksum,
+
+    /// Custom error message
+    Message(String),
+
+    /// BIP32 error
+    Bip32(bitcoin::bip32::Error),
+    /// Miniscript error
+    Miniscript(miniscript::Error),
+}
+
+impl_error!(miniscript::Error, Miniscript, KeyError);
+impl_error!(bitcoin::bip32::Error, Bip32, KeyError);
+
+impl std::fmt::Display for KeyError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::InvalidScriptContext => write!(f, "Invalid script context"),
+            Self::InvalidNetwork => write!(f, "Invalid network"),
+            Self::InvalidChecksum => write!(f, "Invalid checksum"),
+            Self::Message(err) => write!(f, "{}", err),
+            Self::Bip32(err) => write!(f, "BIP32 error: {}", err),
+            Self::Miniscript(err) => write!(f, "Miniscript error: {}", err),
+        }
+    }
+}
+
+impl std::error::Error for KeyError {}
+
+#[cfg(test)]
+pub mod test {
+    use bitcoin::bip32;
+
+    use super::*;
+
+    pub const TEST_ENTROPY: [u8; 32] = [0xAA; 32];
+
+    #[test]
+    fn test_keys_generate_xprv() {
+        let generated_xprv: GeneratedKey<_, miniscript::Segwitv0> =
+            bip32::ExtendedPrivKey::generate_with_entropy_default(TEST_ENTROPY).unwrap();
+
+        assert_eq!(generated_xprv.valid_networks, any_network());
+        assert_eq!(generated_xprv.to_string(), "xprv9s21ZrQH143K4Xr1cJyqTvuL2FWR8eicgY9boWqMBv8MDVUZ65AXHnzBrK1nyomu6wdcabRgmGTaAKawvhAno1V5FowGpTLVx3jxzE5uk3Q");
+    }
+
+    #[test]
+    fn test_keys_generate_wif() {
+        let generated_wif: GeneratedKey<_, miniscript::Segwitv0> =
+            bitcoin::PrivateKey::generate_with_entropy_default(TEST_ENTROPY).unwrap();
+
+        assert_eq!(generated_wif.valid_networks, any_network());
+        assert_eq!(
+            generated_wif.to_string(),
+            "L2wTu6hQrnDMiFNWA5na6jB12ErGQqtXwqpSL7aWquJaZG8Ai3ch"
+        );
+    }
+
+    #[cfg(feature = "keys-bip39")]
+    #[test]
+    fn test_keys_wif_network_bip39() {
+        let xkey: ExtendedKey = bip39::Mnemonic::parse_in(
+            bip39::Language::English,
+            "jelly crash boy whisper mouse ecology tuna soccer memory million news short",
+        )
+        .unwrap()
+        .into_extended_key()
+        .unwrap();
+        let xprv = xkey.into_xprv(Network::Testnet).unwrap();
+
+        assert_eq!(xprv.network, Network::Testnet);
+    }
+}

src/lib.rs

@@ -1,44 +1,288 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+// rustdoc will warn if there are missing docs
+#![warn(missing_docs)]
+// only enables the `doc_cfg` feature when
+// the `docsrs` configuration attribute is defined
+#![cfg_attr(docsrs, feature(doc_cfg))]
+#![cfg_attr(
+    docsrs,
+    doc(html_logo_url = "https://github.com/bitcoindevkit/bdk/raw/master/static/bdk.png")
+)]
+
+//! A modern, lightweight, descriptor-based wallet library written in Rust.
+//!
+//! # About
+//!
+//! The BDK library aims to be the core building block for Bitcoin wallets of any kind.
+//!
+//! * It uses [Miniscript](https://github.com/rust-bitcoin/rust-miniscript) to support descriptors with generalized conditions. This exact same library can be used to build
+//!   single-sig wallets, multisigs, timelocked contracts and more.
+//! * It supports multiple blockchain backends and databases, allowing developers to choose exactly what's right for their projects.
+//! * It is built to be cross-platform: the core logic works on desktop, mobile, and even WebAssembly.
+//! * It is very easy to extend: developers can implement customized logic for blockchain backends, databases, signers, coin selection, and more, without having to fork and modify this library.
+//!
+//! # A Tour of BDK
+//!
+//! BDK consists of a number of modules that provide a range of functionality
+//! essential for implementing descriptor based Bitcoin wallet applications in Rust. In this
+//! section, we will take a brief tour of BDK, summarizing the major APIs and
+//! their uses.
+//!
+//! The easiest way to get started is to add bdk to your dependencies with the default features.
+//! The default features include a simple key-value database ([`sled`](sled)) to cache
+//! blockchain data and an [electrum](https://docs.rs/electrum-client/) blockchain client to
+//! interact with the bitcoin P2P network.
+//!
+//! # Examples
+#![cfg_attr(
+    feature = "electrum",
+    doc = r##"
+## Sync the balance of a descriptor
+
+```no_run
+use bdk::{Wallet, SyncOptions};
+use bdk::database::MemoryDatabase;
+use bdk::blockchain::ElectrumBlockchain;
+use bdk::electrum_client::Client;
+
+fn main() -> Result<(), bdk::Error> {
+    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let blockchain = ElectrumBlockchain::from(client);
+    let wallet = Wallet::new(
+        "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
+        Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
+        bitcoin::Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    println!("Descriptor balance: {} SAT", wallet.get_balance()?);
+
+    Ok(())
+}
+```
+"##
+)]
+//!
+//! ## Generate a few addresses
+//!
+//! ### Example
+//! ```
+//! use bdk::{Wallet};
+//! use bdk::database::MemoryDatabase;
+//! use bdk::wallet::AddressIndex::New;
+//!
+//! fn main() -> Result<(), bdk::Error> {
+//! let wallet = Wallet::new(
+//!         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
+//!         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
+//!         bitcoin::Network::Testnet,
+//!         MemoryDatabase::default(),
+//!     )?;
+//!
+//!     println!("Address #0: {}", wallet.get_address(New)?);
+//!     println!("Address #1: {}", wallet.get_address(New)?);
+//!     println!("Address #2: {}", wallet.get_address(New)?);
+//!
+//!     Ok(())
+//! }
+//! ```
+#![cfg_attr(
+    feature = "electrum",
+    doc = r##"
+## Create a transaction
+
+```no_run
+use bdk::{FeeRate, Wallet, SyncOptions};
+use bdk::database::MemoryDatabase;
+use bdk::blockchain::ElectrumBlockchain;
+use bdk::electrum_client::Client;
+
+use bitcoin::consensus::serialize;
+use bdk::wallet::AddressIndex::New;
+
+fn main() -> Result<(), bdk::Error> {
+    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
+    let wallet = Wallet::new(
+        "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
+        Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
+        bitcoin::Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+    let blockchain = ElectrumBlockchain::from(client);
+
+    wallet.sync(&blockchain, SyncOptions::default())?;
+
+    let send_to = wallet.get_address(New)?;
+    let (psbt, details) = {
+        let mut builder =  wallet.build_tx();
+        builder
+            .add_recipient(send_to.script_pubkey(), 50_000)
+            .enable_rbf()
+            .do_not_spend_change()
+            .fee_rate(FeeRate::from_sat_per_vb(5.0));
+        builder.finish()?
+    };
+
+    println!("Transaction details: {:#?}", details);
+    println!("Unsigned PSBT: {}", &psbt);
+
+    Ok(())
+}
+```
+"##
+)]
+//!
+//! ## Sign a transaction
+//!
+//! ```no_run
+//! use std::str::FromStr;
+//!
+//! use bitcoin::psbt::PartiallySignedTransaction as Psbt;
+//!
+//! use bdk::{Wallet, SignOptions};
+//! use bdk::database::MemoryDatabase;
+//!
+//! fn main() -> Result<(), bdk::Error> {
+//!     let wallet = Wallet::new(
+//!         "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
+//!         Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
+//!         bitcoin::Network::Testnet,
+//!         MemoryDatabase::default(),
+//!     )?;
+//!
+//!     let psbt = "...";
+//!     let mut psbt = Psbt::from_str(psbt)?;
+//!
+//!     let finalized = wallet.sign(&mut psbt, SignOptions::default())?;
+//!
+//!     Ok(())
+//! }
+//! ```
+//!
+//! # Feature flags
+//!
+//! BDK uses a set of [feature flags](https://doc.rust-lang.org/cargo/reference/manifest.html#the-features-section)
+//! to reduce the amount of compiled code by allowing projects to only enable the features they need.
+//! By default, BDK enables two internal features, `key-value-db` and `electrum`.
+//!
+//! If you are new to BDK we recommended that you use the default features which will enable
+//! basic descriptor wallet functionality. More advanced users can disable the `default` features
+//! (`--no-default-features`) and build the BDK library with only the features you need.
+
+//! Below is a list of the available feature flags and the additional functionality they provide.
+//!
+//! * `all-keys`: all features for working with bitcoin keys
+//! * `async-interface`: async functions in bdk traits
+//! * `keys-bip39`: [BIP-39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) mnemonic codes for generating deterministic keys
+//!
+//! # Internal features
+//!
+//! These features do not expose any new API, but influence internal implementation aspects of
+//! BDK.
+//!
+//! * `compact_filters`: [`compact_filters`](crate::blockchain::compact_filters) client protocol for interacting with the bitcoin P2P network
+//! * `electrum`: [`electrum`](crate::blockchain::electrum) client protocol for interacting with electrum servers
+//! * `esplora`: [`esplora`](crate::blockchain::esplora) client protocol for interacting with blockstream [electrs](https://github.com/Blockstream/electrs) servers
+//! * `key-value-db`: key value [`database`](crate::database) based on [`sled`](crate::sled) for caching blockchain data
+
 pub extern crate bitcoin;
 extern crate log;
 pub extern crate miniscript;
 extern crate serde;
 #[macro_use]
 extern crate serde_json;
+#[cfg(feature = "hardware-signer")]
+pub extern crate hwi;
 
-#[cfg(test)]
-#[macro_use]
-extern crate lazy_static;
+#[cfg(all(feature = "reqwest", feature = "ureq"))]
+compile_error!("Features reqwest and ureq are mutually exclusive and cannot be enabled together");
 
+#[cfg(all(feature = "async-interface", feature = "electrum"))]
+compile_error!(
+    "Features async-interface and electrum are mutually exclusive and cannot be enabled together"
+);
+
+#[cfg(all(feature = "async-interface", feature = "ureq"))]
+compile_error!(
+    "Features async-interface and ureq are mutually exclusive and cannot be enabled together"
+);
+
+#[cfg(all(feature = "async-interface", feature = "compact_filters"))]
+compile_error!(
+    "Features async-interface and compact_filters are mutually exclusive and cannot be enabled together"
+);
+
+#[cfg(feature = "keys-bip39")]
+extern crate bip39;
+
+#[cfg(feature = "async-interface")]
 #[macro_use]
 extern crate async_trait;
+#[macro_use]
+extern crate bdk_macros;
+
+#[cfg(feature = "rpc")]
+pub extern crate bitcoincore_rpc;
 
 #[cfg(feature = "electrum")]
 pub extern crate electrum_client;
-#[cfg(feature = "electrum")]
-pub use electrum_client::client::Client;
 
 #[cfg(feature = "esplora")]
-pub extern crate reqwest;
-#[cfg(feature = "esplora")]
-pub use blockchain::esplora::EsploraBlockchain;
+pub extern crate esplora_client;
 
 #[cfg(feature = "key-value-db")]
 pub extern crate sled;
 
-#[cfg(feature = "cli-utils")]
-pub mod cli;
+#[cfg(feature = "sqlite")]
+pub extern crate rusqlite;
 
+// We should consider putting this under a feature flag but we need the macro in doctests so we need
+// to wait until https://github.com/rust-lang/rust/issues/67295 is fixed.
+//
+// Stuff in here is too rough to document atm
+#[doc(hidden)]
 #[macro_use]
-pub mod error;
+pub mod testutils;
+
+#[cfg(test)]
+extern crate assert_matches;
+
+#[allow(unused_imports)]
+#[macro_use]
+pub(crate) mod error;
 pub mod blockchain;
 pub mod database;
 pub mod descriptor;
-#[cfg(feature = "multiparty")]
-pub mod multiparty;
+#[cfg(feature = "test-md-docs")]
+mod doctest;
+pub mod keys;
 pub mod psbt;
-pub mod signer;
-pub mod types;
+pub(crate) mod types;
 pub mod wallet;
 
-pub use descriptor::ExtendedDescriptor;
-pub use wallet::{OfflineWallet, Wallet};
+pub use descriptor::template;
+pub use descriptor::HdKeyPaths;
+pub use error::Error;
+pub use types::*;
+pub use wallet::signer;
+pub use wallet::signer::SignOptions;
+pub use wallet::tx_builder::TxBuilder;
+pub use wallet::SyncOptions;
+pub use wallet::Wallet;
+
+/// Get the version of BDK at runtime
+pub fn version() -> &'static str {
+    env!("CARGO_PKG_VERSION", "unknown")
+}

src/multiparty/mod.rs

@@ -1,231 +0,0 @@
-use std::cell::RefCell;
-use std::collections::BTreeMap;
-
-use bitcoin::secp256k1::Secp256k1;
-
-use crate::descriptor::error::Error;
-use crate::descriptor::keys::{parse_key, DummyKey, Key, KeyAlias, RealKey};
-use crate::descriptor::{ExtendedDescriptor, MiniscriptExtractPolicy, Policy, StringDescriptor};
-
-pub trait ParticipantType: Default {
-    fn validate_aliases(aliases: Vec<&String>) -> Result<(), Error>;
-}
-
-#[derive(Default)]
-pub struct Coordinator {}
-impl ParticipantType for Coordinator {
-    fn validate_aliases(aliases: Vec<&String>) -> Result<(), Error> {
-        if aliases.into_iter().any(|a| a == "[PEER]") {
-            Err(Error::InvalidAlias("[PEER]".into()))
-        } else {
-            Ok(())
-        }
-    }
-}
-
-#[derive(Default)]
-pub struct Peer;
-impl ParticipantType for Peer {
-    fn validate_aliases(aliases: Vec<&String>) -> Result<(), Error> {
-        if !aliases.into_iter().any(|a| a == "[PEER]") {
-            Err(Error::MissingAlias("[PEER]".into()))
-        } else {
-            Ok(())
-        }
-    }
-}
-
-pub struct Participant<T: ParticipantType> {
-    descriptor: StringDescriptor,
-    parsed_keys: BTreeMap<String, Box<dyn Key>>,
-    received_keys: BTreeMap<String, Box<dyn RealKey>>,
-
-    _data: T,
-}
-
-impl<T: ParticipantType> Participant<T> {
-    pub fn new(sd: StringDescriptor) -> Result<Self, Error> {
-        let parsed_keys = Self::parse_keys(&sd, vec![]);
-
-        T::validate_aliases(parsed_keys.keys().collect())?;
-
-        Ok(Participant {
-            descriptor: sd,
-            parsed_keys,
-            received_keys: Default::default(),
-            _data: Default::default(),
-        })
-    }
-
-    fn parse_keys(
-        sd: &StringDescriptor,
-        with_secrets: Vec<&str>,
-    ) -> BTreeMap<String, Box<dyn Key>> {
-        let keys: RefCell<BTreeMap<String, Box<dyn Key>>> = RefCell::new(BTreeMap::new());
-
-        let translatefpk = |string: &String| -> Result<_, Error> {
-            let (key, parsed) = match parse_key(string) {
-                Ok((key, parsed)) => (key, parsed.into_key()),
-                Err(_) => (
-                    string.clone(),
-                    KeyAlias::new_boxed(string.as_str(), with_secrets.contains(&string.as_str())),
-                ),
-            };
-            keys.borrow_mut().insert(key, parsed);
-
-            Ok(DummyKey::default())
-        };
-        let translatefpkh = |string: &String| -> Result<_, Error> {
-            let (key, parsed) = match parse_key(string) {
-                Ok((key, parsed)) => (key, parsed.into_key()),
-                Err(_) => (
-                    string.clone(),
-                    KeyAlias::new_boxed(string.as_str(), with_secrets.contains(&string.as_str())),
-                ),
-            };
-            keys.borrow_mut().insert(key, parsed);
-
-            Ok(DummyKey::default())
-        };
-
-        sd.translate_pk(translatefpk, translatefpkh).unwrap();
-
-        keys.into_inner()
-    }
-
-    pub fn policy_for(&self, with_secrets: Vec<&str>) -> Result<Option<Policy>, Error> {
-        let keys = Self::parse_keys(&self.descriptor, with_secrets);
-        self.descriptor.extract_policy(&keys)
-    }
-
-    fn _missing_keys(&self) -> Vec<&String> {
-        self.parsed_keys
-            .keys()
-            .filter(|k| !self.received_keys.contains_key(*k))
-            .collect()
-    }
-
-    pub fn completed(&self) -> bool {
-        self._missing_keys().is_empty()
-    }
-
-    pub fn finalize(self) -> Result<ExtendedDescriptor, Error> {
-        if !self.completed() {
-            return Err(Error::Incomplete);
-        }
-
-        let translatefpk = |string: &String| -> Result<_, Error> {
-            Ok(format!(
-                "{}",
-                self.received_keys
-                    .get(string)
-                    .expect(&format!("Missing key: `{}`", string))
-            ))
-        };
-        let translatefpkh = |string: &String| -> Result<_, Error> {
-            Ok(format!(
-                "{}",
-                self.received_keys
-                    .get(string)
-                    .expect(&format!("Missing key: `{}`", string))
-            ))
-        };
-
-        let internal = self.descriptor.translate_pk(translatefpk, translatefpkh)?;
-
-        Ok(ExtendedDescriptor {
-            internal,
-            keys: self.received_keys,
-            ctx: Secp256k1::gen_new(),
-        })
-    }
-}
-
-impl Participant<Coordinator> {
-    pub fn descriptor(&self) -> &StringDescriptor {
-        &self.descriptor
-    }
-
-    pub fn add_key(&mut self, alias: &str, key: Box<dyn RealKey>) -> Result<(), Error> {
-        // TODO: check network
-
-        if key.has_secret() {
-            return Err(Error::KeyHasSecret);
-        }
-
-        self.received_keys.insert(alias.into(), key);
-
-        Ok(())
-    }
-
-    pub fn received_keys(&self) -> Vec<&String> {
-        self.received_keys.keys().collect()
-    }
-
-    pub fn missing_keys(&self) -> Vec<&String> {
-        self._missing_keys()
-    }
-
-    pub fn descriptor_for(&self, alias: &str) -> Result<StringDescriptor, Error> {
-        if !self.parsed_keys.contains_key(alias) {
-            return Err(Error::MissingAlias(alias.into()));
-        }
-
-        let map_name = |s: &String| {
-            if s == alias {
-                "[PEER]".into()
-            } else {
-                s.into()
-            }
-        };
-
-        let translatefpk = |string: &String| -> Result<_, Error> { Ok(map_name(string)) };
-        let translatefpkh = |string: &String| -> Result<_, Error> { Ok(map_name(string)) };
-
-        Ok(self.descriptor.translate_pk(translatefpk, translatefpkh)?)
-    }
-
-    pub fn get_map(&self) -> Result<BTreeMap<String, String>, Error> {
-        if !self.completed() {
-            return Err(Error::Incomplete);
-        }
-
-        Ok(self
-            .received_keys
-            .iter()
-            .map(|(k, v)| (k.into(), format!("{}", v)))
-            .collect())
-    }
-}
-
-impl Participant<Peer> {
-    pub fn policy(&self) -> Result<Option<Policy>, Error> {
-        self.policy_for(vec!["[PEER]"])
-    }
-
-    pub fn use_key(&mut self, key: Box<dyn RealKey>) -> Result<(), Error> {
-        let secp = Secp256k1::gen_new();
-        self.received_keys
-            .insert("[PEER]".into(), key.public(&secp)?);
-
-        Ok(())
-    }
-
-    pub fn my_key(&mut self) -> Option<&Box<dyn RealKey>> {
-        self.received_keys.get("[PEER]".into())
-    }
-
-    pub fn apply_map(mut self, map: BTreeMap<String, String>) -> Result<ExtendedDescriptor, Error> {
-        let mut parsed_map: BTreeMap<_, _> = map
-            .into_iter()
-            .map(|(k, v)| -> Result<_, Error> {
-                let (_, parsed) = parse_key(&v)?;
-                Ok((k, parsed))
-            })
-            .collect::<Result<_, _>>()?;
-
-        self.received_keys.append(&mut parsed_map);
-
-        self.finalize()
-    }
-}

src/psbt/mod.rs

@@ -1,271 +1,239 @@
-use std::collections::BTreeMap;
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
-use bitcoin::hashes::{hash160, Hash};
-use bitcoin::util::bip143::SighashComponents;
-use bitcoin::util::bip32::{DerivationPath, ExtendedPrivKey, Fingerprint};
-use bitcoin::util::psbt;
-use bitcoin::{PrivateKey, PublicKey, Script, SigHashType, Transaction};
+//! Additional functions on the `rust-bitcoin` `PartiallySignedTransaction` structure.
 
-use bitcoin::secp256k1::{self, All, Message, Secp256k1};
+use crate::FeeRate;
+use bitcoin::psbt::PartiallySignedTransaction as Psbt;
+use bitcoin::TxOut;
 
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
+// TODO upstream the functions here to `rust-bitcoin`?
 
-use miniscript::{BitcoinSig, MiniscriptKey, Satisfier};
+/// Trait to add functions to extract utxos and calculate fees.
+pub trait PsbtUtils {
+    /// Get the `TxOut` for the specified input index, if it doesn't exist in the PSBT `None` is returned.
+    fn get_utxo_for(&self, input_index: usize) -> Option<TxOut>;
 
-use crate::descriptor::ExtendedDescriptor;
-use crate::error::Error;
-use crate::signer::Signer;
+    /// The total transaction fee amount, sum of input amounts minus sum of output amounts, in sats.
+    /// If the PSBT is missing a TxOut for an input returns None.
+    fn fee_amount(&self) -> Option<u64>;
 
-pub mod utils;
-
-pub struct PSBTSatisfier<'a> {
-    input: &'a psbt::Input,
-    assume_height_reached: bool,
-    create_height: Option<u32>,
-    current_height: Option<u32>,
+    /// The transaction's fee rate. This value will only be accurate if calculated AFTER the
+    /// `PartiallySignedTransaction` is finalized and all witness/signature data is added to the
+    /// transaction.
+    /// If the PSBT is missing a TxOut for an input returns None.
+    fn fee_rate(&self) -> Option<FeeRate>;
 }
 
-impl<'a> PSBTSatisfier<'a> {
-    pub fn new(
-        input: &'a psbt::Input,
-        assume_height_reached: bool,
-        create_height: Option<u32>,
-        current_height: Option<u32>,
-    ) -> Self {
-        PSBTSatisfier {
-            input,
-            assume_height_reached,
-            create_height,
-            current_height,
+impl PsbtUtils for Psbt {
+    #[allow(clippy::all)] // We want to allow `manual_map` but it is too new.
+    fn get_utxo_for(&self, input_index: usize) -> Option<TxOut> {
+        let tx = &self.unsigned_tx;
+
+        if input_index >= tx.input.len() {
+            return None;
         }
-    }
-}
 
-impl<'a> PSBTSatisfier<'a> {
-    fn parse_sig(rawsig: &Vec<u8>) -> Option<BitcoinSig> {
-        let (flag, sig) = rawsig.split_last().unwrap();
-        let flag = bitcoin::SigHashType::from_u32(*flag as u32);
-        let sig = match secp256k1::Signature::from_der(sig) {
-            Ok(sig) => sig,
-            Err(..) => return None,
-        };
-        Some((sig, flag))
-    }
-}
-
-// TODO: also support hash preimages through the "unknown" section of PSBT
-impl<'a> Satisfier<bitcoin::PublicKey> for PSBTSatisfier<'a> {
-    // from https://docs.rs/miniscript/0.12.0/src/miniscript/psbt/mod.rs.html#96
-    fn lookup_sig(&self, pk: &bitcoin::PublicKey) -> Option<BitcoinSig> {
-        debug!("lookup_sig: {}", pk);
-
-        if let Some(rawsig) = self.input.partial_sigs.get(pk) {
-            Self::parse_sig(&rawsig)
+        if let Some(input) = self.inputs.get(input_index) {
+            if let Some(wit_utxo) = &input.witness_utxo {
+                Some(wit_utxo.clone())
+            } else if let Some(in_tx) = &input.non_witness_utxo {
+                Some(in_tx.output[tx.input[input_index].previous_output.vout as usize].clone())
+            } else {
+                None
+            }
         } else {
             None
         }
     }
 
-    fn lookup_pkh_pk(&self, hash: &hash160::Hash) -> Option<bitcoin::PublicKey> {
-        debug!("lookup_pkh_pk: {}", hash);
+    fn fee_amount(&self) -> Option<u64> {
+        let tx = &self.unsigned_tx;
+        let utxos: Option<Vec<TxOut>> = (0..tx.input.len()).map(|i| self.get_utxo_for(i)).collect();
 
-        for (pk, _) in &self.input.partial_sigs {
-            if &pk.to_pubkeyhash() == hash {
-                return Some(*pk);
-            }
-        }
-
-        None
-    }
-
-    fn lookup_pkh_sig(&self, hash: &hash160::Hash) -> Option<(bitcoin::PublicKey, BitcoinSig)> {
-        debug!("lookup_pkh_sig: {}", hash);
-
-        for (pk, sig) in &self.input.partial_sigs {
-            if &pk.to_pubkeyhash() == hash {
-                return match Self::parse_sig(&sig) {
-                    Some(bitcoinsig) => Some((*pk, bitcoinsig)),
-                    None => None,
-                };
-            }
-        }
-
-        None
-    }
-
-    fn check_older(&self, height: u32) -> bool {
-        // TODO: also check if `nSequence` right
-        debug!("check_older: {}", height);
-
-        if let Some(current_height) = self.current_height {
-            // TODO: test >= / >
-            current_height as u64 >= self.create_height.unwrap_or(0) as u64 + height as u64
-        } else {
-            self.assume_height_reached
-        }
-    }
-
-    fn check_after(&self, height: u32) -> bool {
-        // TODO: also check if `nLockTime` is right
-        debug!("check_after: {}", height);
-
-        if let Some(current_height) = self.current_height {
-            current_height > height
-        } else {
-            self.assume_height_reached
-        }
-    }
-}
-
-#[derive(Debug)]
-pub struct PSBTSigner<'a> {
-    tx: &'a Transaction,
-    secp: Secp256k1<All>,
-
-    // psbt: &'b psbt::PartiallySignedTransaction,
-    extended_keys: BTreeMap<Fingerprint, ExtendedPrivKey>,
-    private_keys: BTreeMap<PublicKey, PrivateKey>,
-}
-
-impl<'a> PSBTSigner<'a> {
-    pub fn from_descriptor(tx: &'a Transaction, desc: &ExtendedDescriptor) -> Result<Self, Error> {
-        let secp = Secp256k1::gen_new();
-
-        let mut extended_keys = BTreeMap::new();
-        for xprv in desc.get_xprv() {
-            let fing = xprv.fingerprint(&secp);
-            extended_keys.insert(fing, xprv);
-        }
-
-        let mut private_keys = BTreeMap::new();
-        for privkey in desc.get_secret_keys() {
-            let pubkey = privkey.public_key(&secp);
-            private_keys.insert(pubkey, privkey);
-        }
-
-        Ok(PSBTSigner {
-            tx,
-            secp,
-            extended_keys,
-            private_keys,
+        utxos.map(|inputs| {
+            let input_amount: u64 = inputs.iter().map(|i| i.value).sum();
+            let output_amount: u64 = self.unsigned_tx.output.iter().map(|o| o.value).sum();
+            input_amount
+                .checked_sub(output_amount)
+                .expect("input amount must be greater than output amount")
         })
     }
 
-    pub fn extend(&mut self, mut other: PSBTSigner) -> Result<(), Error> {
-        if self.tx.txid() != other.tx.txid() {
-            return Err(Error::DifferentTransactions);
-        }
-
-        self.extended_keys.append(&mut other.extended_keys);
-        self.private_keys.append(&mut other.private_keys);
-
-        Ok(())
-    }
-
-    // TODO: temporary
-    pub fn all_public_keys(&self) -> impl IntoIterator<Item = &PublicKey> {
-        self.private_keys.keys()
+    fn fee_rate(&self) -> Option<FeeRate> {
+        let fee_amount = self.fee_amount();
+        fee_amount.map(|fee| {
+            let weight = self.clone().extract_tx().weight();
+            FeeRate::from_wu(fee, weight)
+        })
     }
 }
 
-impl<'a> Signer for PSBTSigner<'a> {
-    fn sig_legacy_from_fingerprint(
-        &self,
-        index: usize,
-        sighash: SigHashType,
-        fingerprint: &Fingerprint,
-        path: &DerivationPath,
-        script: &Script,
-    ) -> Result<Option<BitcoinSig>, Error> {
-        self.extended_keys
-            .get(fingerprint)
-            .map_or(Ok(None), |xprv| {
-                let privkey = xprv.derive_priv(&self.secp, path)?;
-                // let derived_pubkey = secp256k1::PublicKey::from_secret_key(&self.secp, &privkey.private_key.key);
+#[cfg(test)]
+mod test {
+    use crate::bitcoin::TxIn;
+    use crate::psbt::Psbt;
+    use crate::wallet::AddressIndex;
+    use crate::wallet::AddressIndex::New;
+    use crate::wallet::{get_funded_wallet, test::get_test_wpkh};
+    use crate::{psbt, FeeRate, SignOptions};
+    use std::str::FromStr;
 
-                let hash = self.tx.signature_hash(index, script, sighash.as_u32());
+    // from bip 174
+    const PSBT_STR: &str = "cHNidP8BAKACAAAAAqsJSaCMWvfEm4IS9Bfi8Vqz9cM9zxU4IagTn4d6W3vkAAAAAAD+////qwlJoIxa98SbghL0F+LxWrP1wz3PFTghqBOfh3pbe+QBAAAAAP7///8CYDvqCwAAAAAZdqkUdopAu9dAy+gdmI5x3ipNXHE5ax2IrI4kAAAAAAAAGXapFG9GILVT+glechue4O/p+gOcykWXiKwAAAAAAAEHakcwRAIgR1lmF5fAGwNrJZKJSGhiGDR9iYZLcZ4ff89X0eURZYcCIFMJ6r9Wqk2Ikf/REf3xM286KdqGbX+EhtdVRs7tr5MZASEDXNxh/HupccC1AaZGoqg7ECy0OIEhfKaC3Ibi1z+ogpIAAQEgAOH1BQAAAAAXqRQ1RebjO4MsRwUPJNPuuTycA5SLx4cBBBYAFIXRNTfy4mVAWjTbr6nj3aAfuCMIAAAA";
 
-                let signature = self.secp.sign(
-                    &Message::from_slice(&hash.into_inner()[..])?,
-                    &privkey.private_key.key,
-                );
-
-                Ok(Some((signature, sighash)))
-            })
+    #[test]
+    #[should_panic(expected = "InputIndexOutOfRange")]
+    fn test_psbt_malformed_psbt_input_legacy() {
+        let psbt_bip = Psbt::from_str(PSBT_STR).unwrap();
+        let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        let send_to = wallet.get_address(AddressIndex::New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.add_recipient(send_to.script_pubkey(), 10_000);
+        let (mut psbt, _) = builder.finish().unwrap();
+        psbt.inputs.push(psbt_bip.inputs[0].clone());
+        let options = SignOptions {
+            trust_witness_utxo: true,
+            ..Default::default()
+        };
+        let _ = wallet.sign(&mut psbt, options).unwrap();
     }
 
-    fn sig_legacy_from_pubkey(
-        &self,
-        index: usize,
-        sighash: SigHashType,
-        public_key: &PublicKey,
-        script: &Script,
-    ) -> Result<Option<BitcoinSig>, Error> {
-        self.private_keys
-            .get(public_key)
-            .map_or(Ok(None), |privkey| {
-                let hash = self.tx.signature_hash(index, script, sighash.as_u32());
-
-                let signature = self
-                    .secp
-                    .sign(&Message::from_slice(&hash.into_inner()[..])?, &privkey.key);
-
-                Ok(Some((signature, sighash)))
-            })
+    #[test]
+    #[should_panic(expected = "InputIndexOutOfRange")]
+    fn test_psbt_malformed_psbt_input_segwit() {
+        let psbt_bip = Psbt::from_str(PSBT_STR).unwrap();
+        let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        let send_to = wallet.get_address(AddressIndex::New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.add_recipient(send_to.script_pubkey(), 10_000);
+        let (mut psbt, _) = builder.finish().unwrap();
+        psbt.inputs.push(psbt_bip.inputs[1].clone());
+        let options = SignOptions {
+            trust_witness_utxo: true,
+            ..Default::default()
+        };
+        let _ = wallet.sign(&mut psbt, options).unwrap();
     }
 
-    fn sig_segwit_from_fingerprint(
-        &self,
-        index: usize,
-        sighash: SigHashType,
-        fingerprint: &Fingerprint,
-        path: &DerivationPath,
-        script: &Script,
-        value: u64,
-    ) -> Result<Option<BitcoinSig>, Error> {
-        self.extended_keys
-            .get(fingerprint)
-            .map_or(Ok(None), |xprv| {
-                let privkey = xprv.derive_priv(&self.secp, path)?;
-
-                let hash = SighashComponents::new(self.tx).sighash_all(
-                    &self.tx.input[index],
-                    script,
-                    value,
-                );
-
-                let signature = self.secp.sign(
-                    &Message::from_slice(&hash.into_inner()[..])?,
-                    &privkey.private_key.key,
-                );
-
-                Ok(Some((signature, sighash)))
-            })
+    #[test]
+    #[should_panic(expected = "InputIndexOutOfRange")]
+    fn test_psbt_malformed_tx_input() {
+        let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        let send_to = wallet.get_address(AddressIndex::New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.add_recipient(send_to.script_pubkey(), 10_000);
+        let (mut psbt, _) = builder.finish().unwrap();
+        psbt.unsigned_tx.input.push(TxIn::default());
+        let options = SignOptions {
+            trust_witness_utxo: true,
+            ..Default::default()
+        };
+        let _ = wallet.sign(&mut psbt, options).unwrap();
     }
 
-    fn sig_segwit_from_pubkey(
-        &self,
-        index: usize,
-        sighash: SigHashType,
-        public_key: &PublicKey,
-        script: &Script,
-        value: u64,
-    ) -> Result<Option<BitcoinSig>, Error> {
-        self.private_keys
-            .get(public_key)
-            .map_or(Ok(None), |privkey| {
-                let hash = SighashComponents::new(self.tx).sighash_all(
-                    &self.tx.input[index],
-                    script,
-                    value,
-                );
+    #[test]
+    fn test_psbt_sign_with_finalized() {
+        let psbt_bip = Psbt::from_str(PSBT_STR).unwrap();
+        let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        let send_to = wallet.get_address(AddressIndex::New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.add_recipient(send_to.script_pubkey(), 10_000);
+        let (mut psbt, _) = builder.finish().unwrap();
 
-                let signature = self
-                    .secp
-                    .sign(&Message::from_slice(&hash.into_inner()[..])?, &privkey.key);
+        // add a finalized input
+        psbt.inputs.push(psbt_bip.inputs[0].clone());
+        psbt.unsigned_tx
+            .input
+            .push(psbt_bip.unsigned_tx.input[0].clone());
 
-                Ok(Some((signature, sighash)))
-            })
+        let _ = wallet.sign(&mut psbt, SignOptions::default()).unwrap();
+    }
+
+    #[test]
+    fn test_psbt_fee_rate_with_witness_utxo() {
+        use psbt::PsbtUtils;
+
+        let expected_fee_rate = 1.2345;
+
+        let (wallet, _, _) = get_funded_wallet("wpkh(tprv8ZgxMBicQKsPd3EupYiPRhaMooHKUHJxNsTfYuScep13go8QFfHdtkG9nRkFGb7busX4isf6X9dURGCoKgitaApQ6MupRhZMcELAxTBRJgS/*)");
+        let addr = wallet.get_address(New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.drain_to(addr.script_pubkey()).drain_wallet();
+        builder.fee_rate(FeeRate::from_sat_per_vb(expected_fee_rate));
+        let (mut psbt, _) = builder.finish().unwrap();
+        let fee_amount = psbt.fee_amount();
+        assert!(fee_amount.is_some());
+
+        let unfinalized_fee_rate = psbt.fee_rate().unwrap();
+
+        let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
+        assert!(finalized);
+
+        let finalized_fee_rate = psbt.fee_rate().unwrap();
+        assert!(finalized_fee_rate.as_sat_per_vb() >= expected_fee_rate);
+        assert!(finalized_fee_rate.as_sat_per_vb() < unfinalized_fee_rate.as_sat_per_vb());
+    }
+
+    #[test]
+    fn test_psbt_fee_rate_with_nonwitness_utxo() {
+        use psbt::PsbtUtils;
+
+        let expected_fee_rate = 1.2345;
+
+        let (wallet, _, _) = get_funded_wallet("pkh(tprv8ZgxMBicQKsPd3EupYiPRhaMooHKUHJxNsTfYuScep13go8QFfHdtkG9nRkFGb7busX4isf6X9dURGCoKgitaApQ6MupRhZMcELAxTBRJgS/*)");
+        let addr = wallet.get_address(New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.drain_to(addr.script_pubkey()).drain_wallet();
+        builder.fee_rate(FeeRate::from_sat_per_vb(expected_fee_rate));
+        let (mut psbt, _) = builder.finish().unwrap();
+        let fee_amount = psbt.fee_amount();
+        assert!(fee_amount.is_some());
+        let unfinalized_fee_rate = psbt.fee_rate().unwrap();
+
+        let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
+        assert!(finalized);
+
+        let finalized_fee_rate = psbt.fee_rate().unwrap();
+        assert!(finalized_fee_rate.as_sat_per_vb() >= expected_fee_rate);
+        assert!(finalized_fee_rate.as_sat_per_vb() < unfinalized_fee_rate.as_sat_per_vb());
+    }
+
+    #[test]
+    fn test_psbt_fee_rate_with_missing_txout() {
+        use psbt::PsbtUtils;
+
+        let expected_fee_rate = 1.2345;
+
+        let (wpkh_wallet, _, _) = get_funded_wallet("wpkh(tprv8ZgxMBicQKsPd3EupYiPRhaMooHKUHJxNsTfYuScep13go8QFfHdtkG9nRkFGb7busX4isf6X9dURGCoKgitaApQ6MupRhZMcELAxTBRJgS/*)");
+        let addr = wpkh_wallet.get_address(New).unwrap();
+        let mut builder = wpkh_wallet.build_tx();
+        builder.drain_to(addr.script_pubkey()).drain_wallet();
+        builder.fee_rate(FeeRate::from_sat_per_vb(expected_fee_rate));
+        let (mut wpkh_psbt, _) = builder.finish().unwrap();
+
+        wpkh_psbt.inputs[0].witness_utxo = None;
+        wpkh_psbt.inputs[0].non_witness_utxo = None;
+        assert!(wpkh_psbt.fee_amount().is_none());
+        assert!(wpkh_psbt.fee_rate().is_none());
+
+        let (pkh_wallet, _, _) = get_funded_wallet("pkh(tprv8ZgxMBicQKsPd3EupYiPRhaMooHKUHJxNsTfYuScep13go8QFfHdtkG9nRkFGb7busX4isf6X9dURGCoKgitaApQ6MupRhZMcELAxTBRJgS/*)");
+        let addr = pkh_wallet.get_address(New).unwrap();
+        let mut builder = pkh_wallet.build_tx();
+        builder.drain_to(addr.script_pubkey()).drain_wallet();
+        builder.fee_rate(FeeRate::from_sat_per_vb(expected_fee_rate));
+        let (mut pkh_psbt, _) = builder.finish().unwrap();
+
+        pkh_psbt.inputs[0].non_witness_utxo = None;
+        assert!(pkh_psbt.fee_amount().is_none());
+        assert!(pkh_psbt.fee_rate().is_none());
     }
 }

src/psbt/utils.rs

@@ -1,28 +0,0 @@
-use bitcoin::util::psbt::PartiallySignedTransaction as PSBT;
-use bitcoin::TxOut;
-
-pub trait PSBTUtils {
-    fn get_utxo_for(&self, input_index: usize) -> Option<TxOut>;
-}
-
-impl PSBTUtils for PSBT {
-    fn get_utxo_for(&self, input_index: usize) -> Option<TxOut> {
-        let tx = &self.global.unsigned_tx;
-
-        if input_index >= tx.input.len() {
-            return None;
-        }
-
-        if let Some(input) = self.inputs.get(input_index) {
-            if let Some(wit_utxo) = &input.witness_utxo {
-                Some(wit_utxo.clone())
-            } else if let Some(in_tx) = &input.non_witness_utxo {
-                Some(in_tx.output[tx.input[input_index].previous_output.vout as usize].clone())
-            } else {
-                None
-            }
-        } else {
-            None
-        }
-    }
-}

src/signer.rs

@@ -1,87 +0,0 @@
-use bitcoin::util::bip32::{DerivationPath, Fingerprint};
-use bitcoin::{PublicKey, Script, SigHashType};
-
-use miniscript::miniscript::satisfy::BitcoinSig;
-
-use crate::error::Error;
-
-pub trait Signer {
-    fn sig_legacy_from_fingerprint(
-        &self,
-        index: usize,
-        sighash: SigHashType,
-        fingerprint: &Fingerprint,
-        path: &DerivationPath,
-        script: &Script,
-    ) -> Result<Option<BitcoinSig>, Error>;
-    fn sig_legacy_from_pubkey(
-        &self,
-        index: usize,
-        sighash: SigHashType,
-        public_key: &PublicKey,
-        script: &Script,
-    ) -> Result<Option<BitcoinSig>, Error>;
-
-    fn sig_segwit_from_fingerprint(
-        &self,
-        index: usize,
-        sighash: SigHashType,
-        fingerprint: &Fingerprint,
-        path: &DerivationPath,
-        script: &Script,
-        value: u64,
-    ) -> Result<Option<BitcoinSig>, Error>;
-    fn sig_segwit_from_pubkey(
-        &self,
-        index: usize,
-        sighash: SigHashType,
-        public_key: &PublicKey,
-        script: &Script,
-        value: u64,
-    ) -> Result<Option<BitcoinSig>, Error>;
-}
-
-#[allow(dead_code)]
-impl dyn Signer {
-    fn sig_legacy_from_fingerprint(
-        &self,
-        _index: usize,
-        _sighash: SigHashType,
-        _fingerprint: &Fingerprint,
-        _path: &DerivationPath,
-        _script: &Script,
-    ) -> Result<Option<BitcoinSig>, Error> {
-        Ok(None)
-    }
-    fn sig_legacy_from_pubkey(
-        &self,
-        _index: usize,
-        _sighash: SigHashType,
-        _public_key: &PublicKey,
-        _script: &Script,
-    ) -> Result<Option<BitcoinSig>, Error> {
-        Ok(None)
-    }
-
-    fn sig_segwit_from_fingerprint(
-        &self,
-        _index: usize,
-        _sighash: SigHashType,
-        _fingerprint: &Fingerprint,
-        _path: &DerivationPath,
-        _script: &Script,
-        _value: u64,
-    ) -> Result<Option<BitcoinSig>, Error> {
-        Ok(None)
-    }
-    fn sig_segwit_from_pubkey(
-        &self,
-        _index: usize,
-        _sighash: SigHashType,
-        _public_key: &PublicKey,
-        _script: &Script,
-        _value: u64,
-    ) -> Result<Option<BitcoinSig>, Error> {
-        Ok(None)
-    }
-}

src/testutils/configurable_blockchain_tests.rs

@@ -0,0 +1,257 @@
+use bitcoin::Network;
+
+use crate::{
+    blockchain::ConfigurableBlockchain, database::MemoryDatabase, testutils, wallet::AddressIndex,
+    Wallet,
+};
+
+use super::blockchain_tests::TestClient;
+
+/// Trait for testing [`ConfigurableBlockchain`] implementations.
+pub trait ConfigurableBlockchainTester<B: ConfigurableBlockchain>: Sized {
+    /// Blockchain name for logging.
+    const BLOCKCHAIN_NAME: &'static str;
+
+    /// Generates a blockchain config with a given stop_gap.
+    ///
+    /// If this returns [`Option::None`], then the associated tests will not run.
+    fn config_with_stop_gap(
+        &self,
+        _test_client: &mut TestClient,
+        _stop_gap: usize,
+    ) -> Option<B::Config> {
+        None
+    }
+
+    /// Runs all available tests.
+    fn run(&self) {
+        let test_client = &mut TestClient::default();
+
+        if self.config_with_stop_gap(test_client, 0).is_some() {
+            test_wallet_sync_with_stop_gaps(test_client, self);
+            test_wallet_sync_fulfills_missing_script_cache(test_client, self);
+            test_wallet_sync_self_transfer_tx(test_client, self);
+        } else {
+            println!(
+                "{}: Skipped tests requiring config_with_stop_gap.",
+                Self::BLOCKCHAIN_NAME
+            );
+        }
+    }
+}
+
+/// Test whether blockchain implementation syncs with expected behaviour given different `stop_gap`
+/// parameters.
+///
+/// For each test vector:
+/// * Fill wallet's derived addresses with balances (as specified by test vector).
+///    * [0..addrs_before]          => 1000sats for each address
+///    * [addrs_before..actual_gap] => empty addresses
+///    * [actual_gap..addrs_after]  => 1000sats for each address
+/// * Then, perform wallet sync and obtain wallet balance
+/// * Check balance is within expected range (we can compare `stop_gap` and `actual_gap` to
+///    determine this).
+fn test_wallet_sync_with_stop_gaps<T, B>(test_client: &mut TestClient, tester: &T)
+where
+    T: ConfigurableBlockchainTester<B>,
+    B: ConfigurableBlockchain,
+{
+    // Generates wallet descriptor
+    let descriptor_of_account = |account_index: usize| -> String {
+        format!("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/{account_index}/*)")
+    };
+
+    // Amount (in satoshis) provided to a single address (which expects to have a balance)
+    const AMOUNT_PER_TX: u64 = 1000;
+
+    // [stop_gap, actual_gap, addrs_before, addrs_after]
+    //
+    // [0]     stop_gap: Passed to [`ElectrumBlockchainConfig`]
+    // [1]   actual_gap: Range size of address indexes without a balance
+    // [2] addrs_before: Range size of address indexes (before gap) which contains a balance
+    // [3]  addrs_after: Range size of address indexes (after gap) which contains a balance
+    let test_vectors: Vec<[u64; 4]> = vec![
+        [0, 0, 0, 5],
+        [0, 0, 5, 5],
+        [0, 1, 5, 5],
+        [0, 2, 5, 5],
+        [1, 0, 5, 5],
+        [1, 1, 5, 5],
+        [1, 2, 5, 5],
+        [2, 1, 5, 5],
+        [2, 2, 5, 5],
+        [2, 3, 5, 5],
+    ];
+
+    for (account_index, vector) in test_vectors.into_iter().enumerate() {
+        let [stop_gap, actual_gap, addrs_before, addrs_after] = vector;
+        let descriptor = descriptor_of_account(account_index);
+
+        let blockchain = B::from_config(
+            &tester
+                .config_with_stop_gap(test_client, stop_gap as _)
+                .unwrap(),
+        )
+        .unwrap();
+
+        let wallet =
+            Wallet::new(&descriptor, None, Network::Regtest, MemoryDatabase::new()).unwrap();
+
+        // fill server-side with txs to specified address indexes
+        // return the max balance of the wallet (also the actual balance)
+        let max_balance = (0..addrs_before)
+            .chain(addrs_before + actual_gap..addrs_before + actual_gap + addrs_after)
+            .fold(0_u64, |sum, i| {
+                let address = wallet.get_address(AddressIndex::Peek(i as _)).unwrap();
+                test_client.receive(testutils! {
+                    @tx ( (@addr address.address) => AMOUNT_PER_TX )
+                });
+                sum + AMOUNT_PER_TX
+            });
+
+        // minimum allowed balance of wallet (based on stop gap)
+        let min_balance = if actual_gap > stop_gap {
+            addrs_before * AMOUNT_PER_TX
+        } else {
+            max_balance
+        };
+        let details = format!(
+            "test_vector: [stop_gap: {}, actual_gap: {}, addrs_before: {}, addrs_after: {}]",
+            stop_gap, actual_gap, addrs_before, addrs_after,
+        );
+        println!("{}", details);
+
+        // perform wallet sync
+        wallet.sync(&blockchain, Default::default()).unwrap();
+
+        let wallet_balance = wallet.get_balance().unwrap().get_total();
+        println!(
+            "max: {}, min: {}, actual: {}",
+            max_balance, min_balance, wallet_balance
+        );
+
+        assert!(
+            wallet_balance <= max_balance,
+            "wallet balance is greater than received amount: {}",
+            details
+        );
+        assert!(
+            wallet_balance >= min_balance,
+            "wallet balance is smaller than expected: {}",
+            details
+        );
+
+        // generate block to confirm new transactions
+        test_client.generate(1, None);
+    }
+}
+
+/// With a `stop_gap` of x and every x addresses having a balance of 1000 (for y addresses),
+/// we expect `Wallet::sync` to correctly self-cache addresses, so that the resulting balance,
+/// after sync, should be y * 1000.
+fn test_wallet_sync_fulfills_missing_script_cache<T, B>(test_client: &mut TestClient, tester: &T)
+where
+    T: ConfigurableBlockchainTester<B>,
+    B: ConfigurableBlockchain,
+{
+    // wallet descriptor
+    let descriptor = "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/200/*)";
+
+    // amount in sats per tx
+    const AMOUNT_PER_TX: u64 = 1000;
+
+    // addr constants
+    const ADDR_COUNT: usize = 6;
+    const ADDR_GAP: usize = 60;
+
+    let blockchain =
+        B::from_config(&tester.config_with_stop_gap(test_client, ADDR_GAP).unwrap()).unwrap();
+
+    let wallet = Wallet::new(descriptor, None, Network::Regtest, MemoryDatabase::new()).unwrap();
+
+    let expected_balance = (0..ADDR_COUNT).fold(0_u64, |sum, i| {
+        let addr_i = i * ADDR_GAP;
+        let address = wallet.get_address(AddressIndex::Peek(addr_i as _)).unwrap();
+
+        println!(
+            "tx: {} sats => [{}] {}",
+            AMOUNT_PER_TX,
+            addr_i,
+            address.to_string()
+        );
+
+        test_client.receive(testutils! {
+            @tx ( (@addr address.address) => AMOUNT_PER_TX )
+        });
+        test_client.generate(1, None);
+
+        sum + AMOUNT_PER_TX
+    });
+    println!("expected balance: {}, syncing...", expected_balance);
+
+    // perform sync
+    wallet.sync(&blockchain, Default::default()).unwrap();
+    println!("sync done!");
+
+    let balance = wallet.get_balance().unwrap().get_total();
+    assert_eq!(balance, expected_balance);
+}
+
+/// Given a `stop_gap`, a wallet with a 2 transactions, one sending to `scriptPubKey` at derivation
+/// index of `stop_gap`, and the other spending from the same `scriptPubKey` into another
+/// `scriptPubKey` at derivation index of `stop_gap * 2`, we expect `Wallet::sync` to perform
+/// correctly, so that we detect the total balance.
+fn test_wallet_sync_self_transfer_tx<T, B>(test_client: &mut TestClient, tester: &T)
+where
+    T: ConfigurableBlockchainTester<B>,
+    B: ConfigurableBlockchain,
+{
+    const TRANSFER_AMOUNT: u64 = 10_000;
+    const STOP_GAP: usize = 75;
+
+    let descriptor = "wpkh(tprv8i8F4EhYDMquzqiecEX8SKYMXqfmmb1Sm7deoA1Hokxzn281XgTkwsd6gL8aJevLE4aJugfVf9MKMvrcRvPawGMenqMBA3bRRfp4s1V7Eg3/*)";
+
+    let blockchain =
+        B::from_config(&tester.config_with_stop_gap(test_client, STOP_GAP).unwrap()).unwrap();
+
+    let wallet = Wallet::new(descriptor, None, Network::Regtest, MemoryDatabase::new()).unwrap();
+
+    let address1 = wallet
+        .get_address(AddressIndex::Peek(STOP_GAP as _))
+        .unwrap();
+    let address2 = wallet
+        .get_address(AddressIndex::Peek((STOP_GAP * 2) as _))
+        .unwrap();
+
+    test_client.receive(testutils! {
+        @tx ( (@addr address1.address) => TRANSFER_AMOUNT )
+    });
+    test_client.generate(1, None);
+
+    wallet.sync(&blockchain, Default::default()).unwrap();
+
+    let mut builder = wallet.build_tx();
+    builder.add_recipient(address2.script_pubkey(), TRANSFER_AMOUNT / 2);
+    let (mut psbt, details) = builder.finish().unwrap();
+    assert!(wallet.sign(&mut psbt, Default::default()).unwrap());
+    blockchain.broadcast(&psbt.extract_tx()).unwrap();
+
+    test_client.generate(1, None);
+
+    // obtain what is expected
+    let fee = details.fee.unwrap();
+    let expected_balance = TRANSFER_AMOUNT - fee;
+    println!("fee={}, expected_balance={}", fee, expected_balance);
+
+    // actually test the wallet
+    wallet.sync(&blockchain, Default::default()).unwrap();
+    let balance = wallet.get_balance().unwrap().get_total();
+    assert_eq!(balance, expected_balance);
+
+    // now try with a fresh wallet
+    let fresh_wallet =
+        Wallet::new(descriptor, None, Network::Regtest, MemoryDatabase::new()).unwrap();
+    fresh_wallet.sync(&blockchain, Default::default()).unwrap();
+    let fresh_balance = fresh_wallet.get_balance().unwrap().get_total();
+    assert_eq!(fresh_balance, expected_balance);
+}

src/testutils/mod.rs

@@ -0,0 +1,233 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+#![allow(missing_docs)]
+
+#[cfg(test)]
+#[cfg(feature = "test-blockchains")]
+pub mod blockchain_tests;
+
+#[cfg(test)]
+#[cfg(feature = "test-blockchains")]
+pub mod configurable_blockchain_tests;
+
+use bitcoin::{Address, Txid};
+
+#[derive(Clone, Debug)]
+pub struct TestIncomingInput {
+    pub txid: Txid,
+    pub vout: u32,
+    pub sequence: Option<u32>,
+}
+
+impl TestIncomingInput {
+    pub fn new(txid: Txid, vout: u32, sequence: Option<u32>) -> Self {
+        Self {
+            txid,
+            vout,
+            sequence,
+        }
+    }
+
+    #[cfg(feature = "test-blockchains")]
+    pub fn into_raw_tx_input(self) -> bitcoincore_rpc::json::CreateRawTransactionInput {
+        bitcoincore_rpc::json::CreateRawTransactionInput {
+            txid: self.txid,
+            vout: self.vout,
+            sequence: self.sequence,
+        }
+    }
+}
+
+#[derive(Clone, Debug)]
+pub struct TestIncomingOutput {
+    pub value: u64,
+    pub to_address: String,
+}
+
+impl TestIncomingOutput {
+    pub fn new(value: u64, to_address: Address) -> Self {
+        Self {
+            value,
+            to_address: to_address.to_string(),
+        }
+    }
+}
+
+#[derive(Clone, Debug)]
+pub struct TestIncomingTx {
+    pub input: Vec<TestIncomingInput>,
+    pub output: Vec<TestIncomingOutput>,
+    pub min_confirmations: Option<u64>,
+    pub locktime: Option<i64>,
+    pub replaceable: Option<bool>,
+}
+
+impl TestIncomingTx {
+    pub fn new(
+        input: Vec<TestIncomingInput>,
+        output: Vec<TestIncomingOutput>,
+        min_confirmations: Option<u64>,
+        locktime: Option<i64>,
+        replaceable: Option<bool>,
+    ) -> Self {
+        Self {
+            input,
+            output,
+            min_confirmations,
+            locktime,
+            replaceable,
+        }
+    }
+
+    pub fn add_input(&mut self, input: TestIncomingInput) {
+        self.input.push(input);
+    }
+
+    pub fn add_output(&mut self, output: TestIncomingOutput) {
+        self.output.push(output);
+    }
+}
+
+#[doc(hidden)]
+#[macro_export]
+macro_rules! testutils {
+    ( @external $descriptors:expr, $child:expr ) => ({
+        use $crate::bitcoin::secp256k1::Secp256k1;
+        use $crate::miniscript::descriptor::{Descriptor, DescriptorPublicKey};
+
+        let secp = Secp256k1::new();
+
+        let parsed = Descriptor::<DescriptorPublicKey>::parse_descriptor(&secp, &$descriptors.0).expect("Failed to parse descriptor in `testutils!(@external)`").0;
+        parsed.at_derivation_index($child).unwrap().address(bitcoin::Network::Regtest).expect("No address form")
+    });
+    ( @internal $descriptors:expr, $child:expr ) => ({
+        use $crate::bitcoin::secp256k1::Secp256k1;
+        use $crate::miniscript::descriptor::{Descriptor, DescriptorPublicKey};
+
+        let secp = Secp256k1::new();
+
+        let parsed = Descriptor::<DescriptorPublicKey>::parse_descriptor(&secp, &$descriptors.1.expect("Missing internal descriptor")).expect("Failed to parse descriptor in `testutils!(@internal)`").0;
+        parsed.at_derivation_index($child).address($crate::bitcoin::Network::Regtest).expect("No address form")
+    });
+    ( @e $descriptors:expr, $child:expr ) => ({ testutils!(@external $descriptors, $child) });
+    ( @i $descriptors:expr, $child:expr ) => ({ testutils!(@internal $descriptors, $child) });
+    ( @addr $addr:expr ) => ({ $addr });
+
+    ( @tx ( $( ( $( $addr:tt )* ) => $amount:expr ),+ ) $( ( @inputs $( ($txid:expr, $vout:expr) ),+ ) )? $( ( @locktime $locktime:expr ) )? $( ( @confirmations $confirmations:expr ) )? $( ( @replaceable $replaceable:expr ) )? ) => ({
+        let outs = vec![$( $crate::testutils::TestIncomingOutput::new($amount, testutils!( $($addr)* ))),+];
+        let _ins: Vec<$crate::testutils::TestIncomingInput> = vec![];
+        $(
+            let _ins = vec![$( $crate::testutils::TestIncomingInput { txid: $txid, vout: $vout, sequence: None }),+];
+        )?
+
+        let locktime = None::<i64>$(.or(Some($locktime)))?;
+
+        let min_confirmations = None::<u64>$(.or(Some($confirmations)))?;
+        let replaceable = None::<bool>$(.or(Some($replaceable)))?;
+
+        $crate::testutils::TestIncomingTx::new(_ins, outs, min_confirmations, locktime, replaceable)
+    });
+
+    ( @literal $key:expr ) => ({
+        let key = $key.to_string();
+        (key, None::<String>, None::<String>)
+    });
+    ( @generate_xprv $( $external_path:expr )? $( ,$internal_path:expr )? ) => ({
+        use rand::Rng;
+
+        let mut seed = [0u8; 32];
+        rand::thread_rng().fill(&mut seed[..]);
+
+        let key = $crate::bitcoin::bip32::ExtendedPrivKey::new_master(
+            $crate::bitcoin::Network::Testnet,
+            &seed,
+        );
+
+        let external_path = None::<String>$(.or(Some($external_path.to_string())))?;
+        let internal_path = None::<String>$(.or(Some($internal_path.to_string())))?;
+
+        (key.unwrap().to_string(), external_path, internal_path)
+    });
+    ( @generate_wif ) => ({
+        use rand::Rng;
+
+        let mut key = [0u8; $crate::bitcoin::secp256k1::constants::SECRET_KEY_SIZE];
+        rand::thread_rng().fill(&mut key[..]);
+
+        ($crate::bitcoin::PrivateKey {
+            compressed: true,
+            network: $crate::bitcoin::Network::Testnet,
+            key: $crate::bitcoin::secp256k1::SecretKey::from_slice(&key).unwrap(),
+        }.to_string(), None::<String>, None::<String>)
+    });
+
+    ( @keys ( $( $alias:expr => ( $( $key_type:tt )* ) ),+ ) ) => ({
+        let mut map = std::collections::HashMap::new();
+        $(
+            let alias: &str = $alias;
+            map.insert(alias, testutils!( $($key_type)* ));
+        )+
+
+        map
+    });
+
+    ( @descriptors ( $external_descriptor:expr ) $( ( $internal_descriptor:expr ) )? $( ( @keys $( $keys:tt )* ) )* ) => ({
+        use std::str::FromStr;
+        use std::collections::HashMap;
+        use std::convert::Infallible;
+
+        use $crate::miniscript::descriptor::Descriptor;
+        use $crate::miniscript::TranslatePk;
+
+        struct Translator {
+            keys: HashMap<&'static str, (String, Option<String>, Option<String>)>,
+            is_internal: bool,
+        }
+
+        impl $crate::miniscript::Translator<String, String, Infallible> for Translator {
+            fn pk(&mut self, pk: &String) -> Result<String, Infallible> {
+                match self.keys.get(pk.as_str()) {
+                    Some((key, ext_path, int_path)) => {
+                        let path = if self.is_internal { int_path } else { ext_path };
+                        Ok(format!("{}{}", key, path.clone().unwrap_or_default()))
+                    }
+                    None => Ok(pk.clone()),
+                }
+            }
+            fn sha256(&mut self, sha256: &String) -> Result<String, Infallible> { Ok(sha256.clone()) }
+            fn hash256(&mut self, hash256: &String) -> Result<String, Infallible> { Ok(hash256.clone()) }
+            fn ripemd160(&mut self, ripemd160: &String) -> Result<String, Infallible> { Ok(ripemd160.clone()) }
+            fn hash160(&mut self, hash160: &String) -> Result<String, Infallible> { Ok(hash160.clone()) }
+        }
+
+        #[allow(unused_assignments, unused_mut)]
+        let mut keys = HashMap::new();
+        $(
+            keys = testutils!{ @keys $( $keys )* };
+        )*
+
+        let mut translator = Translator { keys, is_internal: false };
+
+        let external: Descriptor<String> = FromStr::from_str($external_descriptor).unwrap();
+        let external = external.translate_pk(&mut translator).expect("Infallible conversion");
+        let external = external.to_string();
+
+        translator.is_internal = true;
+
+        let internal = None::<String>$(.or({
+            let internal: Descriptor<String> = FromStr::from_str($internal_descriptor).unwrap();
+            let internal = internal.translate_pk(&mut translator).expect("Infallible conversion");
+            Some(internal.to_string())
+        }))?;
+
+        (external, internal)
+    })
+}

src/types.rs

@@ -1,47 +1,517 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
 use std::convert::AsRef;
+use std::ops::Sub;
 
 use bitcoin::blockdata::transaction::{OutPoint, Transaction, TxOut};
-use bitcoin::hash_types::Txid;
+use bitcoin::{hash_types::Txid, psbt, Weight};
 
 use serde::{Deserialize, Serialize};
 
-// TODO serde flatten?
-#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
-pub enum ScriptType {
+/// Types of keychains
+#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum KeychainKind {
+    /// External
     External = 0,
+    /// Internal, usually used for change outputs
     Internal = 1,
 }
 
-impl ScriptType {
+impl KeychainKind {
+    /// Return [`KeychainKind`] as a byte
     pub fn as_byte(&self) -> u8 {
         match self {
-            ScriptType::External => 'e' as u8,
-            ScriptType::Internal => 'i' as u8,
+            KeychainKind::External => b'e',
+            KeychainKind::Internal => b'i',
         }
     }
 }
 
-impl AsRef<[u8]> for ScriptType {
+impl AsRef<[u8]> for KeychainKind {
     fn as_ref(&self) -> &[u8] {
         match self {
-            ScriptType::External => b"e",
-            ScriptType::Internal => b"i",
+            KeychainKind::External => b"e",
+            KeychainKind::Internal => b"i",
         }
     }
 }
 
-#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
-pub struct UTXO {
-    pub outpoint: OutPoint,
-    pub txout: TxOut,
+/// Fee rate
+#[derive(Debug, Copy, Clone, PartialEq, PartialOrd)]
+// Internally stored as satoshi/vbyte
+pub struct FeeRate(f32);
+
+impl FeeRate {
+    /// Create a new instance checking the value provided
+    ///
+    /// ## Panics
+    ///
+    /// Panics if the value is not [normal](https://doc.rust-lang.org/std/primitive.f32.html#method.is_normal) (except if it's a positive zero) or negative.
+    fn new_checked(value: f32) -> Self {
+        assert!(value.is_normal() || value == 0.0);
+        assert!(value.is_sign_positive());
+
+        FeeRate(value)
+    }
+
+    /// Create a new instance of [`FeeRate`] given a float fee rate in sats/kwu
+    pub fn from_sat_per_kwu(sat_per_kwu: f32) -> Self {
+        FeeRate::new_checked(sat_per_kwu / 250.0_f32)
+    }
+
+    /// Create a new instance of [`FeeRate`] given a float fee rate in sats/kvb
+    pub fn from_sat_per_kvb(sat_per_kvb: f32) -> Self {
+        FeeRate::new_checked(sat_per_kvb / 1000.0_f32)
+    }
+
+    /// Create a new instance of [`FeeRate`] given a float fee rate in btc/kvbytes
+    ///
+    /// ## Panics
+    ///
+    /// Panics if the value is not [normal](https://doc.rust-lang.org/std/primitive.f32.html#method.is_normal) (except if it's a positive zero) or negative.
+    pub fn from_btc_per_kvb(btc_per_kvb: f32) -> Self {
+        FeeRate::new_checked(btc_per_kvb * 1e5)
+    }
+
+    /// Create a new instance of [`FeeRate`] given a float fee rate in satoshi/vbyte
+    ///
+    /// ## Panics
+    ///
+    /// Panics if the value is not [normal](https://doc.rust-lang.org/std/primitive.f32.html#method.is_normal) (except if it's a positive zero) or negative.
+    pub fn from_sat_per_vb(sat_per_vb: f32) -> Self {
+        FeeRate::new_checked(sat_per_vb)
+    }
+
+    /// Create a new [`FeeRate`] with the default min relay fee value
+    pub const fn default_min_relay_fee() -> Self {
+        FeeRate(1.0)
+    }
+
+    /// Calculate fee rate from `fee` and weight units (`wu`).
+    pub fn from_wu(fee: u64, wu: Weight) -> FeeRate {
+        Self::from_vb(fee, wu.to_vbytes_ceil() as usize)
+    }
+
+    /// Calculate fee rate from `fee` and `vbytes`.
+    pub fn from_vb(fee: u64, vbytes: usize) -> FeeRate {
+        let rate = fee as f32 / vbytes as f32;
+        Self::from_sat_per_vb(rate)
+    }
+
+    /// Return the value as satoshi/vbyte
+    pub fn as_sat_per_vb(&self) -> f32 {
+        self.0
+    }
+
+    /// Calculate absolute fee in Satoshis using size in weight units.
+    pub fn fee_wu(&self, wu: Weight) -> u64 {
+        self.fee_vb(wu.to_vbytes_ceil() as usize)
+    }
+
+    /// Calculate absolute fee in Satoshis using size in virtual bytes.
+    pub fn fee_vb(&self, vbytes: usize) -> u64 {
+        (self.as_sat_per_vb() * vbytes as f32).ceil() as u64
+    }
 }
 
-#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, Default)]
-pub struct TransactionDetails {
-    pub transaction: Option<Transaction>,
-    pub txid: Txid,
-    pub timestamp: u64,
-    pub received: u64,
-    pub sent: u64,
-    pub height: Option<u32>,
+impl std::default::Default for FeeRate {
+    fn default() -> Self {
+        FeeRate::default_min_relay_fee()
+    }
+}
+
+impl Sub for FeeRate {
+    type Output = Self;
+
+    fn sub(self, other: FeeRate) -> Self::Output {
+        FeeRate(self.0 - other.0)
+    }
+}
+
+/// Trait implemented by types that can be used to measure weight units.
+pub trait Vbytes {
+    /// Convert weight units to virtual bytes.
+    fn vbytes(self) -> usize;
+}
+
+impl Vbytes for usize {
+    fn vbytes(self) -> usize {
+        // ref: https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki#transaction-size-calculations
+        (self as f32 / 4.0).ceil() as usize
+    }
+}
+
+/// An unspent output owned by a [`Wallet`].
+///
+/// [`Wallet`]: crate::Wallet
+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, Hash)]
+pub struct LocalUtxo {
+    /// Reference to a transaction output
+    pub outpoint: OutPoint,
+    /// Transaction output
+    pub txout: TxOut,
+    /// Type of keychain
+    pub keychain: KeychainKind,
+    /// Whether this UTXO is spent or not
+    pub is_spent: bool,
+}
+
+/// A [`Utxo`] with its `satisfaction_weight`.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct WeightedUtxo {
+    /// The weight of the witness data and `scriptSig` expressed in [weight units]. This is used to
+    /// properly maintain the feerate when adding this input to a transaction during coin selection.
+    ///
+    /// [weight units]: https://en.bitcoin.it/wiki/Weight_units
+    pub satisfaction_weight: usize,
+    /// The UTXO
+    pub utxo: Utxo,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+/// An unspent transaction output (UTXO).
+pub enum Utxo {
+    /// A UTXO owned by the local wallet.
+    Local(LocalUtxo),
+    /// A UTXO owned by another wallet.
+    Foreign {
+        /// The location of the output.
+        outpoint: OutPoint,
+        /// The information about the input we require to add it to a PSBT.
+        // Box it to stop the type being too big.
+        psbt_input: Box<psbt::Input>,
+    },
+}
+
+impl Utxo {
+    /// Get the location of the UTXO
+    pub fn outpoint(&self) -> OutPoint {
+        match &self {
+            Utxo::Local(local) => local.outpoint,
+            Utxo::Foreign { outpoint, .. } => *outpoint,
+        }
+    }
+
+    /// Get the `TxOut` of the UTXO
+    pub fn txout(&self) -> &TxOut {
+        match &self {
+            Utxo::Local(local) => &local.txout,
+            Utxo::Foreign {
+                outpoint,
+                psbt_input,
+            } => {
+                if let Some(prev_tx) = &psbt_input.non_witness_utxo {
+                    return &prev_tx.output[outpoint.vout as usize];
+                }
+
+                if let Some(txout) = &psbt_input.witness_utxo {
+                    return txout;
+                }
+
+                unreachable!("Foreign UTXOs will always have one of these set")
+            }
+        }
+    }
+}
+
+/// A wallet transaction
+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
+pub struct TransactionDetails {
+    /// Optional transaction
+    pub transaction: Option<Transaction>,
+    /// Transaction id
+    pub txid: Txid,
+    /// Received value (sats)
+    /// Sum of owned outputs of this transaction.
+    pub received: u64,
+    /// Sent value (sats)
+    /// Sum of owned inputs of this transaction.
+    pub sent: u64,
+    /// Fee value (sats) if confirmed.
+    /// The availability of the fee depends on the backend. It's never `None` with an Electrum
+    /// Server backend, but it could be `None` with a Bitcoin RPC node without txindex that receive
+    /// funds while offline.
+    pub fee: Option<u64>,
+    /// If the transaction is confirmed, contains height and Unix timestamp of the block containing the
+    /// transaction, unconfirmed transaction contains `None`.
+    pub confirmation_time: Option<BlockTime>,
+}
+
+impl PartialOrd for TransactionDetails {
+    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
+        Some(self.cmp(other))
+    }
+}
+
+impl Ord for TransactionDetails {
+    fn cmp(&self, other: &Self) -> std::cmp::Ordering {
+        self.confirmation_time
+            .cmp(&other.confirmation_time)
+            .then_with(|| self.txid.cmp(&other.txid))
+    }
+}
+
+/// Block height and timestamp of a block
+#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, Default)]
+pub struct BlockTime {
+    /// confirmation block height
+    pub height: u32,
+    /// confirmation block timestamp
+    pub timestamp: u64,
+}
+
+impl PartialOrd for BlockTime {
+    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
+        Some(self.cmp(other))
+    }
+}
+
+impl Ord for BlockTime {
+    fn cmp(&self, other: &Self) -> std::cmp::Ordering {
+        self.height
+            .cmp(&other.height)
+            .then_with(|| self.timestamp.cmp(&other.timestamp))
+    }
+}
+
+/// **DEPRECATED**: Confirmation time of a transaction
+///
+/// The structure has been renamed to `BlockTime`
+#[deprecated(note = "This structure has been renamed to `BlockTime`")]
+pub type ConfirmationTime = BlockTime;
+
+impl BlockTime {
+    /// Returns `Some` `BlockTime` if both `height` and `timestamp` are `Some`
+    pub fn new(height: Option<u32>, timestamp: Option<u64>) -> Option<Self> {
+        match (height, timestamp) {
+            (Some(height), Some(timestamp)) => Some(BlockTime { height, timestamp }),
+            _ => None,
+        }
+    }
+}
+
+/// Balance differentiated in various categories
+#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone, Default)]
+pub struct Balance {
+    /// All coinbase outputs not yet matured
+    pub immature: u64,
+    /// Unconfirmed UTXOs generated by a wallet tx
+    pub trusted_pending: u64,
+    /// Unconfirmed UTXOs received from an external wallet
+    pub untrusted_pending: u64,
+    /// Confirmed and immediately spendable balance
+    pub confirmed: u64,
+}
+
+impl Balance {
+    /// Get sum of trusted_pending and confirmed coins
+    pub fn get_spendable(&self) -> u64 {
+        self.confirmed + self.trusted_pending
+    }
+
+    /// Get the whole balance visible to the wallet
+    pub fn get_total(&self) -> u64 {
+        self.confirmed + self.trusted_pending + self.untrusted_pending + self.immature
+    }
+}
+
+impl std::fmt::Display for Balance {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(
+            f,
+            "{{ immature: {}, trusted_pending: {}, untrusted_pending: {}, confirmed: {} }}",
+            self.immature, self.trusted_pending, self.untrusted_pending, self.confirmed
+        )
+    }
+}
+
+impl std::ops::Add for Balance {
+    type Output = Self;
+
+    fn add(self, other: Self) -> Self {
+        Self {
+            immature: self.immature + other.immature,
+            trusted_pending: self.trusted_pending + other.trusted_pending,
+            untrusted_pending: self.untrusted_pending + other.untrusted_pending,
+            confirmed: self.confirmed + other.confirmed,
+        }
+    }
+}
+
+impl std::iter::Sum for Balance {
+    fn sum<I: Iterator<Item = Self>>(iter: I) -> Self {
+        iter.fold(
+            Balance {
+                ..Default::default()
+            },
+            |a, b| a + b,
+        )
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use bitcoin::hashes::Hash;
+
+    #[test]
+    fn sort_block_time() {
+        let block_time_a = BlockTime {
+            height: 100,
+            timestamp: 100,
+        };
+
+        let block_time_b = BlockTime {
+            height: 100,
+            timestamp: 110,
+        };
+
+        let block_time_c = BlockTime {
+            height: 0,
+            timestamp: 0,
+        };
+
+        let mut vec = vec![
+            block_time_a.clone(),
+            block_time_b.clone(),
+            block_time_c.clone(),
+        ];
+        vec.sort();
+        let expected = vec![block_time_c, block_time_a, block_time_b];
+
+        assert_eq!(vec, expected)
+    }
+
+    #[test]
+    fn sort_tx_details() {
+        let block_time_a = BlockTime {
+            height: 100,
+            timestamp: 100,
+        };
+
+        let block_time_b = BlockTime {
+            height: 0,
+            timestamp: 0,
+        };
+
+        let tx_details_a = TransactionDetails {
+            transaction: None,
+            txid: Txid::all_zeros(),
+            received: 0,
+            sent: 0,
+            fee: None,
+            confirmation_time: None,
+        };
+
+        let tx_details_b = TransactionDetails {
+            transaction: None,
+            txid: Txid::all_zeros(),
+            received: 0,
+            sent: 0,
+            fee: None,
+            confirmation_time: Some(block_time_a),
+        };
+
+        let tx_details_c = TransactionDetails {
+            transaction: None,
+            txid: Txid::all_zeros(),
+            received: 0,
+            sent: 0,
+            fee: None,
+            confirmation_time: Some(block_time_b.clone()),
+        };
+
+        let tx_details_d = TransactionDetails {
+            transaction: None,
+            txid: Txid::from_byte_array([1; Txid::LEN]),
+            received: 0,
+            sent: 0,
+            fee: None,
+            confirmation_time: Some(block_time_b),
+        };
+
+        let mut vec = vec![
+            tx_details_a.clone(),
+            tx_details_b.clone(),
+            tx_details_c.clone(),
+            tx_details_d.clone(),
+        ];
+        vec.sort();
+        let expected = vec![tx_details_a, tx_details_c, tx_details_d, tx_details_b];
+
+        assert_eq!(vec, expected)
+    }
+
+    #[test]
+    fn can_store_feerate_in_const() {
+        const _MIN_RELAY: FeeRate = FeeRate::default_min_relay_fee();
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_invalid_feerate_neg_zero() {
+        let _ = FeeRate::from_sat_per_vb(-0.0);
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_invalid_feerate_neg_value() {
+        let _ = FeeRate::from_sat_per_vb(-5.0);
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_invalid_feerate_nan() {
+        let _ = FeeRate::from_sat_per_vb(f32::NAN);
+    }
+
+    #[test]
+    #[should_panic]
+    fn test_invalid_feerate_inf() {
+        let _ = FeeRate::from_sat_per_vb(f32::INFINITY);
+    }
+
+    #[test]
+    fn test_valid_feerate_pos_zero() {
+        let _ = FeeRate::from_sat_per_vb(0.0);
+    }
+
+    #[test]
+    fn test_fee_from_btc_per_kvb() {
+        let fee = FeeRate::from_btc_per_kvb(1e-5);
+        assert!((fee.as_sat_per_vb() - 1.0).abs() < f32::EPSILON);
+    }
+
+    #[test]
+    fn test_fee_from_sat_per_vbyte() {
+        let fee = FeeRate::from_sat_per_vb(1.0);
+        assert!((fee.as_sat_per_vb() - 1.0).abs() < f32::EPSILON);
+    }
+
+    #[test]
+    fn test_fee_default_min_relay_fee() {
+        let fee = FeeRate::default_min_relay_fee();
+        assert!((fee.as_sat_per_vb() - 1.0).abs() < f32::EPSILON);
+    }
+
+    #[test]
+    fn test_fee_from_sat_per_kvb() {
+        let fee = FeeRate::from_sat_per_kvb(1000.0);
+        assert!((fee.as_sat_per_vb() - 1.0).abs() < f32::EPSILON);
+    }
+
+    #[test]
+    fn test_fee_from_sat_per_kwu() {
+        let fee = FeeRate::from_sat_per_kwu(250.0);
+        assert!((fee.as_sat_per_vb() - 1.0).abs() < f32::EPSILON);
+    }
 }

src/wallet/export.rs

@@ -0,0 +1,382 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Wallet export
+//!
+//! This modules implements the wallet export format used by [FullyNoded](https://github.com/Fonta1n3/FullyNoded/blob/10b7808c8b929b171cca537fb50522d015168ac9/Docs/Wallets/Wallet-Export-Spec.md).
+//!
+//! ## Examples
+//!
+//! ### Import from JSON
+//!
+//! ```
+//! # use std::str::FromStr;
+//! # use bitcoin::*;
+//! # use bdk::database::*;
+//! # use bdk::wallet::export::*;
+//! # use bdk::*;
+//! let import = r#"{
+//!     "descriptor": "wpkh([c258d2e4\/84h\/1h\/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe\/0\/*)",
+//!     "blockheight":1782088,
+//!     "label":"testnet"
+//! }"#;
+//!
+//! let import = FullyNodedExport::from_str(import)?;
+//! let wallet = Wallet::new(
+//!     &import.descriptor(),
+//!     import.change_descriptor().as_ref(),
+//!     Network::Testnet,
+//!     MemoryDatabase::default(),
+//! )?;
+//! # Ok::<_, bdk::Error>(())
+//! ```
+//!
+//! ### Export a `Wallet`
+//! ```
+//! # use bitcoin::*;
+//! # use bdk::database::*;
+//! # use bdk::wallet::export::*;
+//! # use bdk::*;
+//! let wallet = Wallet::new(
+//!     "wpkh([c258d2e4/84h/1h/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe/0/*)",
+//!     Some("wpkh([c258d2e4/84h/1h/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe/1/*)"),
+//!     Network::Testnet,
+//!     MemoryDatabase::default()
+//! )?;
+//! let export = FullyNodedExport::export_wallet(&wallet, "exported wallet", true)
+//!     .map_err(ToString::to_string)
+//!     .map_err(bdk::Error::Generic)?;
+//!
+//! println!("Exported: {}", export.to_string());
+//! # Ok::<_, bdk::Error>(())
+//! ```
+
+use std::str::FromStr;
+
+use serde::{Deserialize, Serialize};
+
+use miniscript::descriptor::{ShInner, WshInner};
+use miniscript::{Descriptor, ScriptContext, Terminal};
+
+use crate::database::BatchDatabase;
+use crate::types::KeychainKind;
+use crate::wallet::Wallet;
+
+/// Alias for [`FullyNodedExport`]
+#[deprecated(since = "0.18.0", note = "Please use [`FullyNodedExport`] instead")]
+pub type WalletExport = FullyNodedExport;
+
+/// Structure that contains the export of a wallet
+///
+/// For a usage example see [this module](crate::wallet::export)'s documentation.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct FullyNodedExport {
+    descriptor: String,
+    /// Earliest block to rescan when looking for the wallet's transactions
+    pub blockheight: u32,
+    /// Arbitrary label for the wallet
+    pub label: String,
+}
+
+impl ToString for FullyNodedExport {
+    fn to_string(&self) -> String {
+        serde_json::to_string(self).unwrap()
+    }
+}
+
+impl FromStr for FullyNodedExport {
+    type Err = serde_json::Error;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        serde_json::from_str(s)
+    }
+}
+
+fn remove_checksum(s: String) -> String {
+    s.split_once('#').map(|(a, _)| String::from(a)).unwrap()
+}
+
+impl FullyNodedExport {
+    /// Export a wallet
+    ///
+    /// This function returns an error if it determines that the `wallet`'s descriptor(s) are not
+    /// supported by Bitcoin Core or don't follow the standard derivation paths defined by BIP44
+    /// and others.
+    ///
+    /// If `include_blockheight` is `true`, this function will look into the `wallet`'s database
+    /// for the oldest transaction it knows and use that as the earliest block to rescan.
+    ///
+    /// If the database is empty or `include_blockheight` is false, the `blockheight` field
+    /// returned will be `0`.
+    pub fn export_wallet<D: BatchDatabase>(
+        wallet: &Wallet<D>,
+        label: &str,
+        include_blockheight: bool,
+    ) -> Result<Self, &'static str> {
+        let descriptor = wallet
+            .get_descriptor_for_keychain(KeychainKind::External)
+            .to_string_with_secret(
+                &wallet
+                    .get_signers(KeychainKind::External)
+                    .as_key_map(wallet.secp_ctx()),
+            );
+        let descriptor = remove_checksum(descriptor);
+        Self::is_compatible_with_core(&descriptor)?;
+
+        let blockheight = match wallet.database.borrow().iter_txs(false) {
+            _ if !include_blockheight => 0,
+            Err(_) => 0,
+            Ok(txs) => txs
+                .into_iter()
+                .filter_map(|tx| tx.confirmation_time.map(|c| c.height))
+                .min()
+                .unwrap_or(0),
+        };
+
+        let export = FullyNodedExport {
+            descriptor,
+            label: label.into(),
+            blockheight,
+        };
+
+        let change_descriptor = match wallet
+            .public_descriptor(KeychainKind::Internal)
+            .map_err(|_| "Invalid change descriptor")?
+            .is_some()
+        {
+            false => None,
+            true => {
+                let descriptor = wallet
+                    .get_descriptor_for_keychain(KeychainKind::Internal)
+                    .to_string_with_secret(
+                        &wallet
+                            .get_signers(KeychainKind::Internal)
+                            .as_key_map(wallet.secp_ctx()),
+                    );
+                Some(remove_checksum(descriptor))
+            }
+        };
+        if export.change_descriptor() != change_descriptor {
+            return Err("Incompatible change descriptor");
+        }
+
+        Ok(export)
+    }
+
+    fn is_compatible_with_core(descriptor: &str) -> Result<(), &'static str> {
+        fn check_ms<Ctx: ScriptContext>(
+            terminal: &Terminal<String, Ctx>,
+        ) -> Result<(), &'static str> {
+            if let Terminal::Multi(_, _) = terminal {
+                Ok(())
+            } else {
+                Err("The descriptor contains operators not supported by Bitcoin Core")
+            }
+        }
+
+        // pkh(), wpkh(), sh(wpkh()) are always fine, as well as multi() and sortedmulti()
+        match Descriptor::<String>::from_str(descriptor).map_err(|_| "Invalid descriptor")? {
+            Descriptor::Pkh(_) | Descriptor::Wpkh(_) => Ok(()),
+            Descriptor::Sh(sh) => match sh.as_inner() {
+                ShInner::Wpkh(_) => Ok(()),
+                ShInner::SortedMulti(_) => Ok(()),
+                ShInner::Wsh(wsh) => match wsh.as_inner() {
+                    WshInner::SortedMulti(_) => Ok(()),
+                    WshInner::Ms(ms) => check_ms(&ms.node),
+                },
+                ShInner::Ms(ms) => check_ms(&ms.node),
+            },
+            Descriptor::Wsh(wsh) => match wsh.as_inner() {
+                WshInner::SortedMulti(_) => Ok(()),
+                WshInner::Ms(ms) => check_ms(&ms.node),
+            },
+            _ => Err("The descriptor is not compatible with Bitcoin Core"),
+        }
+    }
+
+    /// Return the external descriptor
+    pub fn descriptor(&self) -> String {
+        self.descriptor.clone()
+    }
+
+    /// Return the internal descriptor, if present
+    pub fn change_descriptor(&self) -> Option<String> {
+        let replaced = self.descriptor.replace("/0/*", "/1/*");
+
+        if replaced != self.descriptor {
+            Some(replaced)
+        } else {
+            None
+        }
+    }
+}
+
+#[cfg(test)]
+mod test {
+    use std::str::FromStr;
+
+    use bitcoin::{Network, Txid};
+
+    use super::*;
+    use crate::database::{memory::MemoryDatabase, BatchOperations};
+    use crate::types::TransactionDetails;
+    use crate::wallet::Wallet;
+    use crate::BlockTime;
+
+    fn get_test_db() -> MemoryDatabase {
+        let mut db = MemoryDatabase::new();
+        db.set_tx(&TransactionDetails {
+            transaction: None,
+            txid: Txid::from_str(
+                "4ddff1fa33af17f377f62b72357b43107c19110a8009b36fb832af505efed98a",
+            )
+            .unwrap(),
+
+            received: 100_000,
+            sent: 0,
+            fee: Some(500),
+            confirmation_time: Some(BlockTime {
+                timestamp: 12345678,
+                height: 5001,
+            }),
+        })
+        .unwrap();
+
+        db.set_tx(&TransactionDetails {
+            transaction: None,
+            txid: Txid::from_str(
+                "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+            )
+            .unwrap(),
+            received: 25_000,
+            sent: 0,
+            fee: Some(300),
+            confirmation_time: Some(BlockTime {
+                timestamp: 12345677,
+                height: 5000,
+            }),
+        })
+        .unwrap();
+
+        db
+    }
+
+    #[test]
+    fn test_export_bip44() {
+        let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
+        let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
+
+        let wallet = Wallet::new(
+            descriptor,
+            Some(change_descriptor),
+            Network::Bitcoin,
+            get_test_db(),
+        )
+        .unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
+
+        assert_eq!(export.descriptor(), descriptor);
+        assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));
+        assert_eq!(export.blockheight, 5000);
+        assert_eq!(export.label, "Test Label");
+    }
+
+    #[test]
+    #[should_panic(expected = "Incompatible change descriptor")]
+    fn test_export_no_change() {
+        // This wallet explicitly doesn't have a change descriptor. It should be impossible to
+        // export, because exporting this kind of external descriptor normally implies the
+        // existence of an internal descriptor
+
+        let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
+
+        let wallet = Wallet::new(descriptor, None, Network::Bitcoin, get_test_db()).unwrap();
+        FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
+    }
+
+    #[test]
+    #[should_panic(expected = "Incompatible change descriptor")]
+    fn test_export_incompatible_change() {
+        // This wallet has a change descriptor, but the derivation path is not in the "standard"
+        // bip44/49/etc format
+
+        let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
+        let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/50'/0'/1/*)";
+
+        let wallet = Wallet::new(
+            descriptor,
+            Some(change_descriptor),
+            Network::Bitcoin,
+            get_test_db(),
+        )
+        .unwrap();
+        FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
+    }
+
+    #[test]
+    fn test_export_multi() {
+        let descriptor = "wsh(multi(2,\
+                                [73756c7f/48'/0'/0'/2']tpubDCKxNyM3bLgbEX13Mcd8mYxbVg9ajDkWXMh29hMWBurKfVmBfWAM96QVP3zaUcN51HvkZ3ar4VwP82kC8JZhhux8vFQoJintSpVBwpFvyU3/0/*,\
+                                [f9f62194/48'/0'/0'/2']tpubDDp3ZSH1yCwusRppH7zgSxq2t1VEUyXSeEp8E5aFS8m43MknUjiF1bSLo3CGWAxbDyhF1XowA5ukPzyJZjznYk3kYi6oe7QxtX2euvKWsk4/0/*,\
+                                [c98b1535/48'/0'/0'/2']tpubDCDi5W4sP6zSnzJeowy8rQDVhBdRARaPhK1axABi8V1661wEPeanpEXj4ZLAUEoikVtoWcyK26TKKJSecSfeKxwHCcRrge9k1ybuiL71z4a/0/*\
+                          ))";
+        let change_descriptor = "wsh(multi(2,\
+                                       [73756c7f/48'/0'/0'/2']tpubDCKxNyM3bLgbEX13Mcd8mYxbVg9ajDkWXMh29hMWBurKfVmBfWAM96QVP3zaUcN51HvkZ3ar4VwP82kC8JZhhux8vFQoJintSpVBwpFvyU3/1/*,\
+                                       [f9f62194/48'/0'/0'/2']tpubDDp3ZSH1yCwusRppH7zgSxq2t1VEUyXSeEp8E5aFS8m43MknUjiF1bSLo3CGWAxbDyhF1XowA5ukPzyJZjznYk3kYi6oe7QxtX2euvKWsk4/1/*,\
+                                       [c98b1535/48'/0'/0'/2']tpubDCDi5W4sP6zSnzJeowy8rQDVhBdRARaPhK1axABi8V1661wEPeanpEXj4ZLAUEoikVtoWcyK26TKKJSecSfeKxwHCcRrge9k1ybuiL71z4a/1/*\
+                                 ))";
+
+        let wallet = Wallet::new(
+            descriptor,
+            Some(change_descriptor),
+            Network::Testnet,
+            get_test_db(),
+        )
+        .unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
+
+        assert_eq!(export.descriptor(), descriptor);
+        assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));
+        assert_eq!(export.blockheight, 5000);
+        assert_eq!(export.label, "Test Label");
+    }
+
+    #[test]
+    fn test_export_to_json() {
+        let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
+        let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
+
+        let wallet = Wallet::new(
+            descriptor,
+            Some(change_descriptor),
+            Network::Bitcoin,
+            get_test_db(),
+        )
+        .unwrap();
+        let export = FullyNodedExport::export_wallet(&wallet, "Test Label", true).unwrap();
+
+        assert_eq!(export.to_string(), "{\"descriptor\":\"wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44\'/0\'/0\'/0/*)\",\"blockheight\":5000,\"label\":\"Test Label\"}");
+    }
+
+    #[test]
+    fn test_export_from_json() {
+        let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
+        let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
+
+        let import_str = "{\"descriptor\":\"wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44\'/0\'/0\'/0/*)\",\"blockheight\":5000,\"label\":\"Test Label\"}";
+        let export = FullyNodedExport::from_str(import_str).unwrap();
+
+        assert_eq!(export.descriptor(), descriptor);
+        assert_eq!(export.change_descriptor(), Some(change_descriptor.into()));
+        assert_eq!(export.blockheight, 5000);
+        assert_eq!(export.label, "Test Label");
+    }
+}

src/wallet/hardwaresigner.rs

@@ -0,0 +1,100 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! HWI Signer
+//!
+//! This module contains HWISigner, an implementation of a [TransactionSigner] to be
+//! used with hardware wallets.
+//! ```no_run
+//! # use bdk::bitcoin::Network;
+//! # use bdk::database::MemoryDatabase;
+//! # use bdk::signer::SignerOrdering;
+//! # use bdk::wallet::hardwaresigner::HWISigner;
+//! # use bdk::wallet::AddressIndex::New;
+//! # use bdk::{FeeRate, KeychainKind, SignOptions, SyncOptions, Wallet};
+//! # use hwi::HWIClient;
+//! # use std::sync::Arc;
+//! #
+//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
+//! let mut devices = HWIClient::enumerate()?;
+//! if devices.is_empty() {
+//!     panic!("No devices found!");
+//! }
+//! let first_device = devices.remove(0)?;
+//! let custom_signer = HWISigner::from_device(&first_device, Network::Testnet.into())?;
+//!
+//! # let mut wallet = Wallet::new(
+//! #     "",
+//! #     None,
+//! #     Network::Testnet,
+//! #     MemoryDatabase::default(),
+//! # )?;
+//! #
+//! // Adding the hardware signer to the BDK wallet
+//! wallet.add_signer(
+//!     KeychainKind::External,
+//!     SignerOrdering(200),
+//!     Arc::new(custom_signer),
+//! );
+//!
+//! # Ok(())
+//! # }
+//! ```
+
+use bitcoin::bip32::Fingerprint;
+use bitcoin::psbt::PartiallySignedTransaction;
+use bitcoin::secp256k1::{All, Secp256k1};
+
+use hwi::error::Error;
+use hwi::types::{HWIChain, HWIDevice};
+use hwi::HWIClient;
+
+use crate::signer::{SignerCommon, SignerError, SignerId, TransactionSigner};
+
+#[derive(Debug)]
+/// Custom signer for Hardware Wallets
+///
+/// This ignores `sign_options` and leaves the decisions up to the hardware wallet.
+pub struct HWISigner {
+    fingerprint: Fingerprint,
+    client: HWIClient,
+}
+
+impl HWISigner {
+    /// Create a instance from the specified device and chain
+    pub fn from_device(device: &HWIDevice, chain: HWIChain) -> Result<HWISigner, Error> {
+        let client = HWIClient::get_client(device, false, chain)?;
+        Ok(HWISigner {
+            fingerprint: device.fingerprint,
+            client,
+        })
+    }
+}
+
+impl SignerCommon for HWISigner {
+    fn id(&self, _secp: &Secp256k1<All>) -> SignerId {
+        SignerId::Fingerprint(self.fingerprint)
+    }
+}
+
+/// This implementation ignores `sign_options`
+impl TransactionSigner for HWISigner {
+    fn sign_transaction(
+        &self,
+        psbt: &mut PartiallySignedTransaction,
+        _sign_options: &crate::SignOptions,
+        _secp: &crate::wallet::utils::SecpCtx,
+    ) -> Result<(), SignerError> {
+        psbt.combine(self.client.sign_tx(psbt)?.psbt)
+            .expect("Failed to combine HW signed psbt with passed PSBT");
+        Ok(())
+    }
+}

src/wallet/offline_stream.rs

@@ -1,52 +0,0 @@
-use std::io::{self, Error, ErrorKind, Read, Write};
-
-#[derive(Clone, Debug)]
-pub struct OfflineStream;
-
-impl Read for OfflineStream {
-    fn read(&mut self, _buf: &mut [u8]) -> io::Result<usize> {
-        Err(Error::new(
-            ErrorKind::NotConnected,
-            "Trying to read from an OfflineStream",
-        ))
-    }
-}
-
-impl Write for OfflineStream {
-    fn write(&mut self, _buf: &[u8]) -> io::Result<usize> {
-        Err(Error::new(
-            ErrorKind::NotConnected,
-            "Trying to read from an OfflineStream",
-        ))
-    }
-
-    fn flush(&mut self) -> io::Result<()> {
-        Err(Error::new(
-            ErrorKind::NotConnected,
-            "Trying to read from an OfflineStream",
-        ))
-    }
-}
-
-// #[cfg(any(feature = "electrum", feature = "default"))]
-// use electrum_client::Client;
-//
-// #[cfg(any(feature = "electrum", feature = "default"))]
-// impl OfflineStream {
-//     fn new_client() -> {
-//         use std::io::bufreader;
-//
-//         let stream = OfflineStream{};
-//         let buf_reader = BufReader::new(stream.clone());
-//
-//         Client {
-//             stream,
-//             buf_reader,
-//             headers: VecDeque::new(),
-//             script_notifications: BTreeMap::new(),
-//
-//             #[cfg(feature = "debug-calls")]
-//             calls: 0,
-//         }
-//     }
-// }

src/wallet/time.rs

@@ -0,0 +1,73 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Cross-platform time
+//!
+//! This module provides a function to get the current timestamp that works on all the platforms
+//! supported by the library.
+//!
+//! It can be useful to compare it with the timestamps found in
+//! [`TransactionDetails`](crate::types::TransactionDetails).
+
+use std::time::Duration;
+
+#[cfg(target_arch = "wasm32")]
+use js_sys::Date;
+#[cfg(not(target_arch = "wasm32"))]
+use std::time::{Instant as SystemInstant, SystemTime, UNIX_EPOCH};
+
+/// Return the current timestamp in seconds
+#[cfg(not(target_arch = "wasm32"))]
+pub fn get_timestamp() -> u64 {
+    SystemTime::now()
+        .duration_since(UNIX_EPOCH)
+        .unwrap()
+        .as_secs()
+}
+/// Return the current timestamp in seconds
+#[cfg(target_arch = "wasm32")]
+pub fn get_timestamp() -> u64 {
+    let millis = Date::now();
+
+    (millis / 1000.0) as u64
+}
+
+#[cfg(not(target_arch = "wasm32"))]
+pub(crate) struct Instant(SystemInstant);
+#[cfg(target_arch = "wasm32")]
+pub(crate) struct Instant(Duration);
+
+impl Instant {
+    #[cfg(not(target_arch = "wasm32"))]
+    pub fn new() -> Self {
+        Instant(SystemInstant::now())
+    }
+    #[cfg(target_arch = "wasm32")]
+    pub fn new() -> Self {
+        let millis = Date::now();
+
+        let secs = millis / 1000.0;
+        let nanos = (millis % 1000.0) * 1e6;
+
+        Instant(Duration::new(secs as u64, nanos as u32))
+    }
+
+    #[cfg(not(target_arch = "wasm32"))]
+    pub fn elapsed(&self) -> Duration {
+        self.0.elapsed()
+    }
+    #[cfg(target_arch = "wasm32")]
+    pub fn elapsed(&self) -> Duration {
+        let now = Instant::new();
+
+        now.0.checked_sub(self.0).unwrap_or(Duration::new(0, 0))
+    }
+}

src/wallet/tx_builder.rs

@@ -0,0 +1,933 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Transaction builder
+//!
+//! ## Example
+//!
+//! ```
+//! # use std::str::FromStr;
+//! # use bitcoin::*;
+//! # use bdk::*;
+//! # use bdk::wallet::tx_builder::CreateTx;
+//! # let to_address = Address::from_str("2N4eQYCbKUHCCTUjBJeHcJp9ok6J2GZsTDt").unwrap().assume_checked();
+//! # let wallet = doctest_wallet!();
+//! // create a TxBuilder from a wallet
+//! let mut tx_builder = wallet.build_tx();
+//!
+//! tx_builder
+//!     // Create a transaction with one output to `to_address` of 50_000 satoshi
+//!     .add_recipient(to_address.script_pubkey(), 50_000)
+//!     // With a custom fee rate of 5.0 satoshi/vbyte
+//!     .fee_rate(bdk::FeeRate::from_sat_per_vb(5.0))
+//!     // Only spend non-change outputs
+//!     .do_not_spend_change()
+//!     // Turn on RBF signaling
+//!     .enable_rbf();
+//! let (psbt, tx_details) = tx_builder.finish()?;
+//! # Ok::<(), bdk::Error>(())
+//! ```
+
+use std::collections::BTreeMap;
+use std::collections::HashSet;
+use std::default::Default;
+use std::marker::PhantomData;
+
+use bitcoin::psbt::{self, PartiallySignedTransaction as Psbt};
+use bitcoin::{absolute, script::PushBytes, OutPoint, ScriptBuf, Sequence, Transaction};
+
+use super::coin_selection::{CoinSelectionAlgorithm, DefaultCoinSelectionAlgorithm};
+use crate::{database::BatchDatabase, Error, Utxo, Wallet};
+use crate::{
+    types::{FeeRate, KeychainKind, LocalUtxo, WeightedUtxo},
+    TransactionDetails,
+};
+/// Context in which the [`TxBuilder`] is valid
+pub trait TxBuilderContext: std::fmt::Debug + Default + Clone {}
+
+/// Marker type to indicate the [`TxBuilder`] is being used to create a new transaction (as opposed
+/// to bumping the fee of an existing one).
+#[derive(Debug, Default, Clone)]
+pub struct CreateTx;
+impl TxBuilderContext for CreateTx {}
+
+/// Marker type to indicate the [`TxBuilder`] is being used to bump the fee of an existing transaction.
+#[derive(Debug, Default, Clone)]
+pub struct BumpFee;
+impl TxBuilderContext for BumpFee {}
+
+/// A transaction builder
+///
+/// A `TxBuilder` is created by calling [`build_tx`] or [`build_fee_bump`] on a wallet. After
+/// assigning it, you set options on it until finally calling [`finish`] to consume the builder and
+/// generate the transaction.
+///
+/// Each option setting method on `TxBuilder` takes and returns `&mut self` so you can chain calls
+/// as in the following example:
+///
+/// ```
+/// # use bdk::*;
+/// # use bdk::wallet::tx_builder::*;
+/// # use bitcoin::*;
+/// # use core::str::FromStr;
+/// # let wallet = doctest_wallet!();
+/// # let addr1 = Address::from_str("2N4eQYCbKUHCCTUjBJeHcJp9ok6J2GZsTDt").unwrap().assume_checked();
+/// # let addr2 = addr1.clone();
+/// // chaining
+/// let (psbt1, details) = {
+///     let mut builder = wallet.build_tx();
+///     builder
+///         .ordering(TxOrdering::Untouched)
+///         .add_recipient(addr1.script_pubkey(), 50_000)
+///         .add_recipient(addr2.script_pubkey(), 50_000);
+///     builder.finish()?
+/// };
+///
+/// // non-chaining
+/// let (psbt2, details) = {
+///     let mut builder = wallet.build_tx();
+///     builder.ordering(TxOrdering::Untouched);
+///     for addr in &[addr1, addr2] {
+///         builder.add_recipient(addr.script_pubkey(), 50_000);
+///     }
+///     builder.finish()?
+/// };
+///
+/// assert_eq!(psbt1.unsigned_tx.output[..2], psbt2.unsigned_tx.output[..2]);
+/// # Ok::<(), bdk::Error>(())
+/// ```
+///
+/// At the moment [`coin_selection`] is an exception to the rule as it consumes `self`.
+/// This means it is usually best to call [`coin_selection`] on the return value of `build_tx` before assigning it.
+///
+/// For further examples see [this module](super::tx_builder)'s documentation;
+///
+/// [`build_tx`]: Wallet::build_tx
+/// [`build_fee_bump`]: Wallet::build_fee_bump
+/// [`finish`]: Self::finish
+/// [`coin_selection`]: Self::coin_selection
+#[derive(Debug)]
+pub struct TxBuilder<'a, D, Cs, Ctx> {
+    pub(crate) wallet: &'a Wallet<D>,
+    pub(crate) params: TxParams,
+    pub(crate) coin_selection: Cs,
+    pub(crate) phantom: PhantomData<Ctx>,
+}
+
+/// The parameters for transaction creation sans coin selection algorithm.
+//TODO: TxParams should eventually be exposed publicly.
+#[derive(Default, Debug, Clone)]
+pub(crate) struct TxParams {
+    pub(crate) recipients: Vec<(ScriptBuf, u64)>,
+    pub(crate) drain_wallet: bool,
+    pub(crate) drain_to: Option<ScriptBuf>,
+    pub(crate) fee_policy: Option<FeePolicy>,
+    pub(crate) internal_policy_path: Option<BTreeMap<String, Vec<usize>>>,
+    pub(crate) external_policy_path: Option<BTreeMap<String, Vec<usize>>>,
+    pub(crate) utxos: Vec<WeightedUtxo>,
+    pub(crate) unspendable: HashSet<OutPoint>,
+    pub(crate) manually_selected_only: bool,
+    pub(crate) sighash: Option<psbt::PsbtSighashType>,
+    pub(crate) ordering: TxOrdering,
+    pub(crate) locktime: Option<absolute::LockTime>,
+    pub(crate) rbf: Option<RbfValue>,
+    pub(crate) version: Option<Version>,
+    pub(crate) change_policy: ChangeSpendPolicy,
+    pub(crate) only_witness_utxo: bool,
+    pub(crate) add_global_xpubs: bool,
+    pub(crate) include_output_redeem_witness_script: bool,
+    pub(crate) bumping_fee: Option<PreviousFee>,
+    pub(crate) current_height: Option<absolute::LockTime>,
+    pub(crate) allow_dust: bool,
+}
+
+#[derive(Clone, Copy, Debug)]
+pub(crate) struct PreviousFee {
+    pub absolute: u64,
+    pub rate: f32,
+}
+
+#[derive(Debug, Clone, Copy)]
+pub(crate) enum FeePolicy {
+    FeeRate(FeeRate),
+    FeeAmount(u64),
+}
+
+impl std::default::Default for FeePolicy {
+    fn default() -> Self {
+        FeePolicy::FeeRate(FeeRate::default_min_relay_fee())
+    }
+}
+
+impl<'a, Cs: Clone, Ctx, D> Clone for TxBuilder<'a, D, Cs, Ctx> {
+    fn clone(&self) -> Self {
+        TxBuilder {
+            wallet: self.wallet,
+            params: self.params.clone(),
+            coin_selection: self.coin_selection.clone(),
+            phantom: PhantomData,
+        }
+    }
+}
+
+// methods supported by both contexts, for any CoinSelectionAlgorithm
+impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
+    TxBuilder<'a, D, Cs, Ctx>
+{
+    /// Set a custom fee rate
+    pub fn fee_rate(&mut self, fee_rate: FeeRate) -> &mut Self {
+        self.params.fee_policy = Some(FeePolicy::FeeRate(fee_rate));
+        self
+    }
+
+    /// Set an absolute fee
+    pub fn fee_absolute(&mut self, fee_amount: u64) -> &mut Self {
+        self.params.fee_policy = Some(FeePolicy::FeeAmount(fee_amount));
+        self
+    }
+
+    /// Set the policy path to use while creating the transaction for a given keychain.
+    ///
+    /// This method accepts a map where the key is the policy node id (see
+    /// [`Policy::id`](crate::descriptor::Policy::id)) and the value is the list of the indexes of
+    /// the items that are intended to be satisfied from the policy node (see
+    /// [`SatisfiableItem::Thresh::items`](crate::descriptor::policy::SatisfiableItem::Thresh::items)).
+    ///
+    /// ## Example
+    ///
+    /// An example of when the policy path is needed is the following descriptor:
+    /// `wsh(thresh(2,pk(A),sj:and_v(v:pk(B),n:older(6)),snj:and_v(v:pk(C),after(630000))))`,
+    /// derived from the miniscript policy `thresh(2,pk(A),and(pk(B),older(6)),and(pk(C),after(630000)))`.
+    /// It declares three descriptor fragments, and at the top level it uses `thresh()` to
+    /// ensure that at least two of them are satisfied. The individual fragments are:
+    ///
+    /// 1. `pk(A)`
+    /// 2. `and(pk(B),older(6))`
+    /// 3. `and(pk(C),after(630000))`
+    ///
+    /// When those conditions are combined in pairs, it's clear that the transaction needs to be created
+    /// differently depending on how the user intends to satisfy the policy afterwards:
+    ///
+    /// * If fragments `1` and `2` are used, the transaction will need to use a specific
+    ///   `n_sequence` in order to spend an `OP_CSV` branch.
+    /// * If fragments `1` and `3` are used, the transaction will need to use a specific `locktime`
+    ///   in order to spend an `OP_CLTV` branch.
+    /// * If fragments `2` and `3` are used, the transaction will need both.
+    ///
+    /// When the spending policy is represented as a tree (see
+    /// [`Wallet::policies`](super::Wallet::policies)), every node
+    /// is assigned a unique identifier that can be used in the policy path to specify which of
+    /// the node's children the user intends to satisfy: for instance, assuming the `thresh()`
+    /// root node of this example has an id of `aabbccdd`, the policy path map would look like:
+    ///
+    /// `{ "aabbccdd" => [0, 1] }`
+    ///
+    /// where the key is the node's id, and the value is a list of the children that should be
+    /// used, in no particular order.
+    ///
+    /// If a particularly complex descriptor has multiple ambiguous thresholds in its structure,
+    /// multiple entries can be added to the map, one for each node that requires an explicit path.
+    ///
+    /// ```
+    /// # use std::str::FromStr;
+    /// # use std::collections::BTreeMap;
+    /// # use bitcoin::*;
+    /// # use bdk::*;
+    /// # let to_address =
+    /// Address::from_str("2N4eQYCbKUHCCTUjBJeHcJp9ok6J2GZsTDt")
+    ///     .unwrap()
+    ///     .assume_checked();
+    /// # let wallet = doctest_wallet!();
+    /// let mut path = BTreeMap::new();
+    /// path.insert("aabbccdd".to_string(), vec![0, 1]);
+    ///
+    /// let builder = wallet
+    ///     .build_tx()
+    ///     .add_recipient(to_address.script_pubkey(), 50_000)
+    ///     .policy_path(path, KeychainKind::External);
+    ///
+    /// # Ok::<(), bdk::Error>(())
+    /// ```
+    pub fn policy_path(
+        &mut self,
+        policy_path: BTreeMap<String, Vec<usize>>,
+        keychain: KeychainKind,
+    ) -> &mut Self {
+        let to_update = match keychain {
+            KeychainKind::Internal => &mut self.params.internal_policy_path,
+            KeychainKind::External => &mut self.params.external_policy_path,
+        };
+
+        *to_update = Some(policy_path);
+        self
+    }
+
+    /// Add the list of outpoints to the internal list of UTXOs that **must** be spent.
+    ///
+    /// If an error occurs while adding any of the UTXOs then none of them are added and the error is returned.
+    ///
+    /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
+    /// the "utxos" and the "unspendable" list, it will be spent.
+    pub fn add_utxos(&mut self, outpoints: &[OutPoint]) -> Result<&mut Self, Error> {
+        let utxos = outpoints
+            .iter()
+            .map(|outpoint| self.wallet.get_utxo(*outpoint)?.ok_or(Error::UnknownUtxo))
+            .collect::<Result<Vec<_>, _>>()?;
+
+        for utxo in utxos {
+            let descriptor = self.wallet.get_descriptor_for_keychain(utxo.keychain);
+            #[allow(deprecated)]
+            let satisfaction_weight = descriptor.max_satisfaction_weight().unwrap();
+            self.params.utxos.push(WeightedUtxo {
+                satisfaction_weight,
+                utxo: Utxo::Local(utxo),
+            });
+        }
+
+        Ok(self)
+    }
+
+    /// Add a utxo to the internal list of utxos that **must** be spent
+    ///
+    /// These have priority over the "unspendable" utxos, meaning that if a utxo is present both in
+    /// the "utxos" and the "unspendable" list, it will be spent.
+    pub fn add_utxo(&mut self, outpoint: OutPoint) -> Result<&mut Self, Error> {
+        self.add_utxos(&[outpoint])
+    }
+
+    /// Add a foreign UTXO i.e. a UTXO not owned by this wallet.
+    ///
+    /// At a minimum to add a foreign UTXO we need:
+    ///
+    /// 1. `outpoint`: To add it to the raw transaction.
+    /// 2. `psbt_input`: To know the value.
+    /// 3. `satisfaction_weight`: To know how much weight/vbytes the input will add to the transaction for fee calculation.
+    ///
+    /// There are several security concerns about adding foreign UTXOs that application
+    /// developers should consider. First, how do you know the value of the input is correct? If a
+    /// `non_witness_utxo` is provided in the `psbt_input` then this method implicitly verifies the
+    /// value by checking it against the transaction. If only a `witness_utxo` is provided then this
+    /// method doesn't verify the value but just takes it as a given -- it is up to you to check
+    /// that whoever sent you the `input_psbt` was not lying!
+    ///
+    /// Secondly, you must somehow provide `satisfaction_weight` of the input. Depending on your
+    /// application it may be important that this be known precisely. If not, a malicious
+    /// counterparty may fool you into putting in a value that is too low, giving the transaction a
+    /// lower than expected feerate. They could also fool you into putting a value that is too high
+    /// causing you to pay a fee that is too high. The party who is broadcasting the transaction can
+    /// of course check the real input weight matches the expected weight prior to broadcasting.
+    ///
+    /// To guarantee the `satisfaction_weight` is correct, you can require the party providing the
+    /// `psbt_input` provide a miniscript descriptor for the input so you can check it against the
+    /// `script_pubkey` and then ask it for the [`max_satisfaction_weight`].
+    ///
+    /// This is an **EXPERIMENTAL** feature, API and other major changes are expected.
+    ///
+    /// # Errors
+    ///
+    /// This method returns errors in the following circumstances:
+    ///
+    /// 1. The `psbt_input` does not contain a `witness_utxo` or `non_witness_utxo`.
+    /// 2. The data in `non_witness_utxo` does not match what is in `outpoint`.
+    ///
+    /// Note unless you set [`only_witness_utxo`] any non-taproot `psbt_input` you pass to this
+    /// method must have `non_witness_utxo` set otherwise you will get an error when [`finish`]
+    /// is called.
+    ///
+    /// [`only_witness_utxo`]: Self::only_witness_utxo
+    /// [`finish`]: Self::finish
+    /// [`max_satisfaction_weight`]: miniscript::Descriptor::max_satisfaction_weight
+    pub fn add_foreign_utxo(
+        &mut self,
+        outpoint: OutPoint,
+        psbt_input: psbt::Input,
+        satisfaction_weight: usize,
+    ) -> Result<&mut Self, Error> {
+        if psbt_input.witness_utxo.is_none() {
+            match psbt_input.non_witness_utxo.as_ref() {
+                Some(tx) => {
+                    if tx.txid() != outpoint.txid {
+                        return Err(Error::Generic(
+                            "Foreign utxo outpoint does not match PSBT input".into(),
+                        ));
+                    }
+                    if tx.output.len() <= outpoint.vout as usize {
+                        return Err(Error::InvalidOutpoint(outpoint));
+                    }
+                }
+                None => {
+                    return Err(Error::Generic(
+                        "Foreign utxo missing witness_utxo or non_witness_utxo".into(),
+                    ))
+                }
+            }
+        }
+
+        self.params.utxos.push(WeightedUtxo {
+            satisfaction_weight,
+            utxo: Utxo::Foreign {
+                outpoint,
+                psbt_input: Box::new(psbt_input),
+            },
+        });
+
+        Ok(self)
+    }
+
+    /// Only spend utxos added by [`add_utxo`].
+    ///
+    /// The wallet will **not** add additional utxos to the transaction even if they are needed to
+    /// make the transaction valid.
+    ///
+    /// [`add_utxo`]: Self::add_utxo
+    pub fn manually_selected_only(&mut self) -> &mut Self {
+        self.params.manually_selected_only = true;
+        self
+    }
+
+    /// Replace the internal list of unspendable utxos with a new list
+    ///
+    /// It's important to note that the "must-be-spent" utxos added with [`TxBuilder::add_utxo`]
+    /// have priority over these. See the docs of the two linked methods for more details.
+    pub fn unspendable(&mut self, unspendable: Vec<OutPoint>) -> &mut Self {
+        self.params.unspendable = unspendable.into_iter().collect();
+        self
+    }
+
+    /// Add a utxo to the internal list of unspendable utxos
+    ///
+    /// It's important to note that the "must-be-spent" utxos added with [`TxBuilder::add_utxo`]
+    /// have priority over this. See the docs of the two linked methods for more details.
+    pub fn add_unspendable(&mut self, unspendable: OutPoint) -> &mut Self {
+        self.params.unspendable.insert(unspendable);
+        self
+    }
+
+    /// Sign with a specific sig hash
+    ///
+    /// **Use this option very carefully**
+    pub fn sighash(&mut self, sighash: psbt::PsbtSighashType) -> &mut Self {
+        self.params.sighash = Some(sighash);
+        self
+    }
+
+    /// Choose the ordering for inputs and outputs of the transaction
+    pub fn ordering(&mut self, ordering: TxOrdering) -> &mut Self {
+        self.params.ordering = ordering;
+        self
+    }
+
+    /// Use a specific nLockTime while creating the transaction
+    ///
+    /// This can cause conflicts if the wallet's descriptors contain an "after" (OP_CLTV) operator.
+    pub fn nlocktime(&mut self, locktime: absolute::LockTime) -> &mut Self {
+        self.params.locktime = Some(locktime);
+        self
+    }
+
+    /// Build a transaction with a specific version
+    ///
+    /// The `version` should always be greater than `0` and greater than `1` if the wallet's
+    /// descriptors contain an "older" (OP_CSV) operator.
+    pub fn version(&mut self, version: i32) -> &mut Self {
+        self.params.version = Some(Version(version));
+        self
+    }
+
+    /// Do not spend change outputs
+    ///
+    /// This effectively adds all the change outputs to the "unspendable" list. See
+    /// [`TxBuilder::unspendable`].
+    pub fn do_not_spend_change(&mut self) -> &mut Self {
+        self.params.change_policy = ChangeSpendPolicy::ChangeForbidden;
+        self
+    }
+
+    /// Only spend change outputs
+    ///
+    /// This effectively adds all the non-change outputs to the "unspendable" list. See
+    /// [`TxBuilder::unspendable`].
+    pub fn only_spend_change(&mut self) -> &mut Self {
+        self.params.change_policy = ChangeSpendPolicy::OnlyChange;
+        self
+    }
+
+    /// Set a specific [`ChangeSpendPolicy`]. See [`TxBuilder::do_not_spend_change`] and
+    /// [`TxBuilder::only_spend_change`] for some shortcuts.
+    pub fn change_policy(&mut self, change_policy: ChangeSpendPolicy) -> &mut Self {
+        self.params.change_policy = change_policy;
+        self
+    }
+
+    /// Only Fill-in the [`psbt::Input::witness_utxo`](bitcoin::psbt::Input::witness_utxo) field when spending from
+    /// SegWit descriptors.
+    ///
+    /// This reduces the size of the PSBT, but some signers might reject them due to the lack of
+    /// the `non_witness_utxo`.
+    pub fn only_witness_utxo(&mut self) -> &mut Self {
+        self.params.only_witness_utxo = true;
+        self
+    }
+
+    /// Fill-in the [`psbt::Output::redeem_script`](bitcoin::psbt::Output::redeem_script) and
+    /// [`psbt::Output::witness_script`](bitcoin::psbt::Output::witness_script) fields.
+    ///
+    /// This is useful for signers which always require it, like ColdCard hardware wallets.
+    pub fn include_output_redeem_witness_script(&mut self) -> &mut Self {
+        self.params.include_output_redeem_witness_script = true;
+        self
+    }
+
+    /// Fill-in the `PSBT_GLOBAL_XPUB` field with the extended keys contained in both the external
+    /// and internal descriptors
+    ///
+    /// This is useful for offline signers that take part to a multisig. Some hardware wallets like
+    /// BitBox and ColdCard are known to require this.
+    pub fn add_global_xpubs(&mut self) -> &mut Self {
+        self.params.add_global_xpubs = true;
+        self
+    }
+
+    /// Spend all the available inputs. This respects filters like [`TxBuilder::unspendable`] and the change policy.
+    pub fn drain_wallet(&mut self) -> &mut Self {
+        self.params.drain_wallet = true;
+        self
+    }
+
+    /// Choose the coin selection algorithm
+    ///
+    /// Overrides the [`DefaultCoinSelectionAlgorithm`](super::coin_selection::DefaultCoinSelectionAlgorithm).
+    ///
+    /// Note that this function consumes the builder and returns it so it is usually best to put this as the first call on the builder.
+    pub fn coin_selection<P: CoinSelectionAlgorithm<D>>(
+        self,
+        coin_selection: P,
+    ) -> TxBuilder<'a, D, P, Ctx> {
+        TxBuilder {
+            wallet: self.wallet,
+            params: self.params,
+            coin_selection,
+            phantom: PhantomData,
+        }
+    }
+
+    /// Finish building the transaction.
+    ///
+    /// Returns the [`BIP174`] "PSBT" and summary details about the transaction.
+    ///
+    /// [`BIP174`]: https://github.com/bitcoin/bips/blob/master/bip-0174.mediawiki
+    pub fn finish(self) -> Result<(Psbt, TransactionDetails), Error> {
+        self.wallet.create_tx(self.coin_selection, self.params)
+    }
+
+    /// Enable signaling RBF
+    ///
+    /// This will use the default nSequence value of `0xFFFFFFFD`.
+    pub fn enable_rbf(&mut self) -> &mut Self {
+        self.params.rbf = Some(RbfValue::Default);
+        self
+    }
+
+    /// Enable signaling RBF with a specific nSequence value
+    ///
+    /// This can cause conflicts if the wallet's descriptors contain an "older" (OP_CSV) operator
+    /// and the given `nsequence` is lower than the CSV value.
+    ///
+    /// If the `nsequence` is higher than `0xFFFFFFFD` an error will be thrown, since it would not
+    /// be a valid nSequence to signal RBF.
+    pub fn enable_rbf_with_sequence(&mut self, nsequence: Sequence) -> &mut Self {
+        self.params.rbf = Some(RbfValue::Value(nsequence));
+        self
+    }
+
+    /// Set the current blockchain height.
+    ///
+    /// This will be used to:
+    /// 1. Set the nLockTime for preventing fee sniping.
+    /// **Note**: This will be ignored if you manually specify a nlocktime using [`TxBuilder::nlocktime`].
+    /// 2. Decide whether coinbase outputs are mature or not. If the coinbase outputs are not
+    ///    mature at `current_height`, we ignore them in the coin selection.
+    ///    If you want to create a transaction that spends immature coinbase inputs, manually
+    ///    add them using [`TxBuilder::add_utxos`].
+    ///
+    /// In both cases, if you don't provide a current height, we use the last sync height.
+    pub fn current_height(&mut self, height: u32) -> &mut Self {
+        self.params.current_height =
+            Some(absolute::LockTime::from_height(height).expect("Invalid height"));
+        self
+    }
+
+    /// Set whether or not the dust limit is checked.
+    ///
+    /// **Note**: by avoiding a dust limit check you may end up with a transaction that is non-standard.
+    pub fn allow_dust(&mut self, allow_dust: bool) -> &mut Self {
+        self.params.allow_dust = allow_dust;
+        self
+    }
+}
+
+impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, D, Cs, CreateTx> {
+    /// Replace the recipients already added with a new list
+    pub fn set_recipients(&mut self, recipients: Vec<(ScriptBuf, u64)>) -> &mut Self {
+        self.params.recipients = recipients;
+        self
+    }
+
+    /// Add a recipient to the internal list
+    pub fn add_recipient(&mut self, script_pubkey: ScriptBuf, amount: u64) -> &mut Self {
+        self.params.recipients.push((script_pubkey, amount));
+        self
+    }
+
+    /// Add data as an output, using OP_RETURN
+    pub fn add_data<T: AsRef<PushBytes>>(&mut self, data: &T) -> &mut Self {
+        let script = ScriptBuf::new_op_return(data);
+        self.add_recipient(script, 0u64);
+        self
+    }
+
+    /// Sets the address to *drain* excess coins to.
+    ///
+    /// Usually, when there are excess coins they are sent to a change address generated by the
+    /// wallet. This option replaces the usual change address with an arbitrary `script_pubkey` of
+    /// your choosing. Just as with a change output, if the drain output is not needed (the excess
+    /// coins are too small) it will not be included in the resulting transaction. The only
+    /// difference is that it is valid to use `drain_to` without setting any ordinary recipients
+    /// with [`add_recipient`] (but it is perfectly fine to add recipients as well).
+    ///
+    /// If you choose not to set any recipients, you should either provide the utxos that the
+    /// transaction should spend via [`add_utxos`], or set [`drain_wallet`] to spend all of them.
+    ///
+    /// When bumping the fees of a transaction made with this option, you probably want to
+    /// use [`allow_shrinking`] to allow this output to be reduced to pay for the extra fees.
+    ///
+    /// # Example
+    ///
+    /// `drain_to` is very useful for draining all the coins in a wallet with [`drain_wallet`] to a
+    /// single address.
+    ///
+    /// ```
+    /// # use std::str::FromStr;
+    /// # use bitcoin::*;
+    /// # use bdk::*;
+    /// # use bdk::wallet::tx_builder::CreateTx;
+    /// # let to_address =
+    /// Address::from_str("2N4eQYCbKUHCCTUjBJeHcJp9ok6J2GZsTDt")
+    ///     .unwrap()
+    ///     .assume_checked();
+    /// # let wallet = doctest_wallet!();
+    /// let mut tx_builder = wallet.build_tx();
+    ///
+    /// tx_builder
+    ///     // Spend all outputs in this wallet.
+    ///     .drain_wallet()
+    ///     // Send the excess (which is all the coins minus the fee) to this address.
+    ///     .drain_to(to_address.script_pubkey())
+    ///     .fee_rate(bdk::FeeRate::from_sat_per_vb(5.0))
+    ///     .enable_rbf();
+    /// let (psbt, tx_details) = tx_builder.finish()?;
+    /// # Ok::<(), bdk::Error>(())
+    /// ```
+    ///
+    /// [`allow_shrinking`]: Self::allow_shrinking
+    /// [`add_recipient`]: Self::add_recipient
+    /// [`add_utxos`]: Self::add_utxos
+    /// [`drain_wallet`]: Self::drain_wallet
+    pub fn drain_to(&mut self, script_pubkey: ScriptBuf) -> &mut Self {
+        self.params.drain_to = Some(script_pubkey);
+        self
+    }
+}
+
+// methods supported only by bump_fee
+impl<'a, D: BatchDatabase> TxBuilder<'a, D, DefaultCoinSelectionAlgorithm, BumpFee> {
+    /// Explicitly tells the wallet that it is allowed to reduce the amount of the output matching this
+    /// `script_pubkey` in order to bump the transaction fee. Without specifying this the wallet
+    /// will attempt to find a change output to shrink instead.
+    ///
+    /// **Note** that the output may shrink to below the dust limit and therefore be removed. If it is
+    /// preserved then it is currently not guaranteed to be in the same position as it was
+    /// originally.
+    ///
+    /// Returns an `Err` if `script_pubkey` can't be found among the recipients of the
+    /// transaction we are bumping.
+    pub fn allow_shrinking(&mut self, script_pubkey: ScriptBuf) -> Result<&mut Self, Error> {
+        match self
+            .params
+            .recipients
+            .iter()
+            .position(|(recipient_script, _)| *recipient_script == script_pubkey)
+        {
+            Some(position) => {
+                self.params.recipients.remove(position);
+                self.params.drain_to = Some(script_pubkey);
+                Ok(self)
+            }
+            None => Err(Error::Generic(format!(
+                "{} was not in the original transaction",
+                script_pubkey
+            ))),
+        }
+    }
+}
+
+/// Ordering of the transaction's inputs and outputs
+#[derive(Debug, Ord, PartialOrd, Eq, PartialEq, Hash, Clone, Copy)]
+pub enum TxOrdering {
+    /// Randomized (default)
+    Shuffle,
+    /// Unchanged
+    Untouched,
+    /// BIP69 / Lexicographic
+    Bip69Lexicographic,
+}
+
+impl Default for TxOrdering {
+    fn default() -> Self {
+        TxOrdering::Shuffle
+    }
+}
+
+impl TxOrdering {
+    /// Sort transaction inputs and outputs by [`TxOrdering`] variant
+    pub fn sort_tx(&self, tx: &mut Transaction) {
+        match self {
+            TxOrdering::Untouched => {}
+            TxOrdering::Shuffle => {
+                use rand::seq::SliceRandom;
+                #[cfg(test)]
+                use rand::SeedableRng;
+
+                #[cfg(not(test))]
+                let mut rng = rand::thread_rng();
+                #[cfg(test)]
+                let mut rng = rand::rngs::StdRng::seed_from_u64(12345);
+
+                tx.output.shuffle(&mut rng);
+            }
+            TxOrdering::Bip69Lexicographic => {
+                tx.input.sort_unstable_by_key(|txin| {
+                    (txin.previous_output.txid, txin.previous_output.vout)
+                });
+                tx.output
+                    .sort_unstable_by_key(|txout| (txout.value, txout.script_pubkey.clone()));
+            }
+        }
+    }
+}
+
+/// Transaction version
+///
+/// Has a default value of `1`
+#[derive(Debug, Ord, PartialOrd, Eq, PartialEq, Hash, Clone, Copy)]
+pub(crate) struct Version(pub(crate) i32);
+
+impl Default for Version {
+    fn default() -> Self {
+        Version(1)
+    }
+}
+
+/// RBF nSequence value
+///
+/// Has a default value of `0xFFFFFFFD`
+#[derive(Debug, Ord, PartialOrd, Eq, PartialEq, Hash, Clone, Copy)]
+pub(crate) enum RbfValue {
+    Default,
+    Value(Sequence),
+}
+
+impl RbfValue {
+    pub(crate) fn get_value(&self) -> Sequence {
+        match self {
+            RbfValue::Default => Sequence::ENABLE_RBF_NO_LOCKTIME,
+            RbfValue::Value(v) => *v,
+        }
+    }
+}
+
+/// Policy regarding the use of change outputs when creating a transaction
+#[derive(Debug, Ord, PartialOrd, Eq, PartialEq, Hash, Clone, Copy)]
+pub enum ChangeSpendPolicy {
+    /// Use both change and non-change outputs (default)
+    ChangeAllowed,
+    /// Only use change outputs (see [`TxBuilder::only_spend_change`])
+    OnlyChange,
+    /// Only use non-change outputs (see [`TxBuilder::do_not_spend_change`])
+    ChangeForbidden,
+}
+
+impl Default for ChangeSpendPolicy {
+    fn default() -> Self {
+        ChangeSpendPolicy::ChangeAllowed
+    }
+}
+
+impl ChangeSpendPolicy {
+    pub(crate) fn is_satisfied_by(&self, utxo: &LocalUtxo) -> bool {
+        match self {
+            ChangeSpendPolicy::ChangeAllowed => true,
+            ChangeSpendPolicy::OnlyChange => utxo.keychain == KeychainKind::Internal,
+            ChangeSpendPolicy::ChangeForbidden => utxo.keychain == KeychainKind::External,
+        }
+    }
+}
+
+#[cfg(test)]
+mod test {
+    const ORDERING_TEST_TX: &str = "0200000003c26f3eb7932f7acddc5ddd26602b77e7516079b03090a16e2c2f54\
+                                    85d1fd600f0100000000ffffffffc26f3eb7932f7acddc5ddd26602b77e75160\
+                                    79b03090a16e2c2f5485d1fd600f0000000000ffffffff571fb3e02278217852\
+                                    dd5d299947e2b7354a639adc32ec1fa7b82cfb5dec530e0500000000ffffffff\
+                                    03e80300000000000002aaeee80300000000000001aa200300000000000001ff\
+                                    00000000";
+    macro_rules! ordering_test_tx {
+        () => {
+            deserialize::<bitcoin::Transaction>(&Vec::<u8>::from_hex(ORDERING_TEST_TX).unwrap())
+                .unwrap()
+        };
+    }
+
+    use bitcoin::consensus::deserialize;
+    use bitcoin::hashes::hex::FromHex;
+    use std::str::FromStr;
+
+    use super::*;
+
+    #[test]
+    fn test_output_ordering_default_shuffle() {
+        assert_eq!(TxOrdering::default(), TxOrdering::Shuffle);
+    }
+
+    #[test]
+    fn test_output_ordering_untouched() {
+        let original_tx = ordering_test_tx!();
+        let mut tx = original_tx.clone();
+
+        TxOrdering::Untouched.sort_tx(&mut tx);
+
+        assert_eq!(original_tx, tx);
+    }
+
+    #[test]
+    fn test_output_ordering_shuffle() {
+        let original_tx = ordering_test_tx!();
+        let mut tx = original_tx.clone();
+
+        TxOrdering::Shuffle.sort_tx(&mut tx);
+
+        assert_eq!(original_tx.input, tx.input);
+        assert_ne!(original_tx.output, tx.output);
+    }
+
+    #[test]
+    fn test_output_ordering_bip69() {
+        let original_tx = ordering_test_tx!();
+        let mut tx = original_tx;
+
+        TxOrdering::Bip69Lexicographic.sort_tx(&mut tx);
+
+        assert_eq!(
+            tx.input[0].previous_output,
+            bitcoin::OutPoint::from_str(
+                "0e53ec5dfb2cb8a71fec32dc9a634a35b7e24799295ddd5278217822e0b31f57:5"
+            )
+            .unwrap()
+        );
+        assert_eq!(
+            tx.input[1].previous_output,
+            bitcoin::OutPoint::from_str(
+                "0f60fdd185542f2c6ea19030b0796051e7772b6026dd5ddccd7a2f93b73e6fc2:0"
+            )
+            .unwrap()
+        );
+        assert_eq!(
+            tx.input[2].previous_output,
+            bitcoin::OutPoint::from_str(
+                "0f60fdd185542f2c6ea19030b0796051e7772b6026dd5ddccd7a2f93b73e6fc2:1"
+            )
+            .unwrap()
+        );
+
+        assert_eq!(tx.output[0].value, 800);
+        assert_eq!(tx.output[1].script_pubkey, ScriptBuf::from(vec![0xAA]));
+        assert_eq!(
+            tx.output[2].script_pubkey,
+            ScriptBuf::from(vec![0xAA, 0xEE])
+        );
+    }
+
+    fn get_test_utxos() -> Vec<LocalUtxo> {
+        use bitcoin::hashes::Hash;
+
+        vec![
+            LocalUtxo {
+                outpoint: OutPoint {
+                    txid: bitcoin::Txid::from_slice(&[0; 32]).unwrap(),
+                    vout: 0,
+                },
+                txout: Default::default(),
+                keychain: KeychainKind::External,
+                is_spent: false,
+            },
+            LocalUtxo {
+                outpoint: OutPoint {
+                    txid: bitcoin::Txid::from_slice(&[0; 32]).unwrap(),
+                    vout: 1,
+                },
+                txout: Default::default(),
+                keychain: KeychainKind::Internal,
+                is_spent: false,
+            },
+        ]
+    }
+
+    #[test]
+    fn test_change_spend_policy_default() {
+        let change_spend_policy = ChangeSpendPolicy::default();
+        let filtered = get_test_utxos()
+            .into_iter()
+            .filter(|u| change_spend_policy.is_satisfied_by(u))
+            .count();
+
+        assert_eq!(filtered, 2);
+    }
+
+    #[test]
+    fn test_change_spend_policy_no_internal() {
+        let change_spend_policy = ChangeSpendPolicy::ChangeForbidden;
+        let filtered = get_test_utxos()
+            .into_iter()
+            .filter(|u| change_spend_policy.is_satisfied_by(u))
+            .collect::<Vec<_>>();
+
+        assert_eq!(filtered.len(), 1);
+        assert_eq!(filtered[0].keychain, KeychainKind::External);
+    }
+
+    #[test]
+    fn test_change_spend_policy_only_internal() {
+        let change_spend_policy = ChangeSpendPolicy::OnlyChange;
+        let filtered = get_test_utxos()
+            .into_iter()
+            .filter(|u| change_spend_policy.is_satisfied_by(u))
+            .collect::<Vec<_>>();
+
+        assert_eq!(filtered.len(), 1);
+        assert_eq!(filtered[0].keychain, KeychainKind::Internal);
+    }
+
+    #[test]
+    fn test_default_tx_version_1() {
+        let version = Version::default();
+        assert_eq!(version.0, 1);
+    }
+}

src/wallet/utils.rs

@@ -1,48 +1,186 @@
-// De-facto standard "dust limit" (even though it should change based on the output type)
-const DUST_LIMIT_SATOSHI: u64 = 546;
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
 
+use bitcoin::secp256k1::{All, Secp256k1};
+use bitcoin::{absolute, Script, Sequence};
+
+use miniscript::{MiniscriptKey, Satisfier, ToPublicKey};
+
+/// Trait to check if a value is below the dust limit.
+/// We are performing dust value calculation for a given script public key using rust-bitcoin to
+/// keep it compatible with network dust rate
 // we implement this trait to make sure we don't mess up the comparison with off-by-one like a <
-// instead of a <= etc. The constant value for the dust limit is not public on purpose, to
-// encourage the usage of this trait.
+// instead of a <= etc.
 pub trait IsDust {
-    fn is_dust(&self) -> bool;
+    /// Check whether or not a value is below dust limit
+    fn is_dust(&self, script: &Script) -> bool;
 }
 
 impl IsDust for u64 {
-    fn is_dust(&self) -> bool {
-        *self <= DUST_LIMIT_SATOSHI
+    fn is_dust(&self, script: &Script) -> bool {
+        *self < script.dust_value().to_sat()
     }
 }
 
-pub struct ChunksIterator<I: Iterator> {
-    iter: I,
-    size: usize,
+pub struct After {
+    pub current_height: Option<u32>,
+    pub assume_height_reached: bool,
 }
 
-impl<I: Iterator> ChunksIterator<I> {
-    pub fn new(iter: I, size: usize) -> Self {
-        ChunksIterator { iter, size }
-    }
-}
-
-impl<I: Iterator> Iterator for ChunksIterator<I> {
-    type Item = Vec<<I as std::iter::Iterator>::Item>;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        let mut v = Vec::new();
-        for _ in 0..self.size {
-            let e = self.iter.next();
-
-            match e {
-                None => break,
-                Some(val) => v.push(val),
-            }
+impl After {
+    pub(crate) fn new(current_height: Option<u32>, assume_height_reached: bool) -> After {
+        After {
+            current_height,
+            assume_height_reached,
         }
-
-        if v.is_empty() {
-            return None;
-        }
-
-        Some(v)
+    }
+}
+
+pub(crate) fn check_nsequence_rbf(rbf: Sequence, csv: Sequence) -> bool {
+    // The RBF value must enable relative timelocks
+    if !rbf.is_relative_lock_time() {
+        return false;
+    }
+
+    // Both values should be represented in the same unit (either time-based or
+    // block-height based)
+    if rbf.is_time_locked() != csv.is_time_locked() {
+        return false;
+    }
+
+    // The value should be at least `csv`
+    if rbf < csv {
+        return false;
+    }
+
+    true
+}
+
+impl<Pk: MiniscriptKey + ToPublicKey> Satisfier<Pk> for After {
+    fn check_after(&self, n: absolute::LockTime) -> bool {
+        if let Some(current_height) = self.current_height {
+            current_height >= n.to_consensus_u32()
+        } else {
+            self.assume_height_reached
+        }
+    }
+}
+
+pub struct Older {
+    pub current_height: Option<u32>,
+    pub create_height: Option<u32>,
+    pub assume_height_reached: bool,
+}
+
+impl Older {
+    pub(crate) fn new(
+        current_height: Option<u32>,
+        create_height: Option<u32>,
+        assume_height_reached: bool,
+    ) -> Older {
+        Older {
+            current_height,
+            create_height,
+            assume_height_reached,
+        }
+    }
+}
+
+impl<Pk: MiniscriptKey + ToPublicKey> Satisfier<Pk> for Older {
+    fn check_older(&self, n: Sequence) -> bool {
+        if let Some(current_height) = self.current_height {
+            // TODO: test >= / >
+            current_height
+                >= self
+                    .create_height
+                    .unwrap_or(0)
+                    .checked_add(n.to_consensus_u32())
+                    .expect("Overflowing addition")
+        } else {
+            self.assume_height_reached
+        }
+    }
+}
+
+pub(crate) type SecpCtx = Secp256k1<All>;
+
+#[cfg(test)]
+mod test {
+    use std::str::FromStr;
+
+    // When nSequence is lower than this flag the timelock is interpreted as block-height-based,
+    // otherwise it's time-based
+    pub(crate) const SEQUENCE_LOCKTIME_TYPE_FLAG: u32 = 1 << 22;
+
+    use super::{check_nsequence_rbf, IsDust};
+    use crate::bitcoin::{Address, Network, Sequence};
+
+    #[test]
+    fn test_is_dust() {
+        let script_p2pkh = Address::from_str("1GNgwA8JfG7Kc8akJ8opdNWJUihqUztfPe")
+            .unwrap()
+            .require_network(Network::Bitcoin)
+            .unwrap()
+            .script_pubkey();
+        assert!(script_p2pkh.is_p2pkh());
+        assert!(545.is_dust(&script_p2pkh));
+        assert!(!546.is_dust(&script_p2pkh));
+
+        let script_p2wpkh = Address::from_str("bc1qxlh2mnc0yqwas76gqq665qkggee5m98t8yskd8")
+            .unwrap()
+            .require_network(Network::Bitcoin)
+            .unwrap()
+            .script_pubkey();
+        assert!(script_p2wpkh.is_v0_p2wpkh());
+        assert!(293.is_dust(&script_p2wpkh));
+        assert!(!294.is_dust(&script_p2wpkh));
+    }
+
+    #[test]
+    fn test_check_nsequence_rbf_msb_set() {
+        let result = check_nsequence_rbf(Sequence(0x80000000), Sequence(5000));
+        assert!(!result);
+    }
+
+    #[test]
+    fn test_check_nsequence_rbf_lt_csv() {
+        let result = check_nsequence_rbf(Sequence(4000), Sequence(5000));
+        assert!(!result);
+    }
+
+    #[test]
+    fn test_check_nsequence_rbf_different_unit() {
+        let result =
+            check_nsequence_rbf(Sequence(SEQUENCE_LOCKTIME_TYPE_FLAG + 5000), Sequence(5000));
+        assert!(!result);
+    }
+
+    #[test]
+    fn test_check_nsequence_rbf_mask() {
+        let result = check_nsequence_rbf(Sequence(0x3f + 10_000), Sequence(5000));
+        assert!(result);
+    }
+
+    #[test]
+    fn test_check_nsequence_rbf_same_unit_blocks() {
+        let result = check_nsequence_rbf(Sequence(10_000), Sequence(5000));
+        assert!(result);
+    }
+
+    #[test]
+    fn test_check_nsequence_rbf_same_unit_time() {
+        let result = check_nsequence_rbf(
+            Sequence(SEQUENCE_LOCKTIME_TYPE_FLAG + 10_000),
+            Sequence(SEQUENCE_LOCKTIME_TYPE_FLAG + 5000),
+        );
+        assert!(result);
     }
 }

src/wallet/verify.rs

@@ -0,0 +1,166 @@
+// Bitcoin Dev Kit
+// Written in 2021 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Verify transactions against the consensus rules
+
+use std::collections::HashMap;
+use std::fmt;
+
+use bitcoin::consensus::serialize;
+use bitcoin::{OutPoint, Transaction, Txid};
+
+use crate::blockchain::GetTx;
+use crate::database::Database;
+use crate::error::Error;
+
+/// Verify a transaction against the consensus rules
+///
+/// This function uses [`bitcoinconsensus`] to verify transactions by fetching the required data
+/// either from the [`Database`] or using the [`Blockchain`].
+///
+/// Depending on the [capabilities](crate::blockchain::Blockchain::get_capabilities) of the
+/// [`Blockchain`] backend, the method could fail when called with old "historical" transactions or
+/// with unconfirmed transactions that have been evicted from the backend's memory.
+///
+/// [`Blockchain`]: crate::blockchain::Blockchain
+pub fn verify_tx<D: Database, B: GetTx>(
+    tx: &Transaction,
+    database: &D,
+    blockchain: &B,
+) -> Result<(), VerifyError> {
+    log::debug!("Verifying {}", tx.txid());
+
+    let serialized_tx = serialize(tx);
+    let mut tx_cache = HashMap::<_, Transaction>::new();
+
+    for (index, input) in tx.input.iter().enumerate() {
+        let prev_tx = if let Some(prev_tx) = tx_cache.get(&input.previous_output.txid) {
+            prev_tx.clone()
+        } else if let Some(prev_tx) = database.get_raw_tx(&input.previous_output.txid)? {
+            prev_tx
+        } else if let Some(prev_tx) = blockchain.get_tx(&input.previous_output.txid)? {
+            prev_tx
+        } else {
+            return Err(VerifyError::MissingInputTx(input.previous_output.txid));
+        };
+
+        let spent_output = prev_tx
+            .output
+            .get(input.previous_output.vout as usize)
+            .ok_or(VerifyError::InvalidInput(input.previous_output))?;
+
+        bitcoinconsensus::verify(
+            &spent_output.script_pubkey.to_bytes(),
+            spent_output.value,
+            &serialized_tx,
+            index,
+        )?;
+
+        // Since we have a local cache we might as well cache stuff from the db, as it will very
+        // likely decrease latency compared to reading from disk or performing an SQL query.
+        tx_cache.insert(prev_tx.txid(), prev_tx);
+    }
+
+    Ok(())
+}
+
+/// Error during validation of a tx agains the consensus rules
+#[derive(Debug)]
+pub enum VerifyError {
+    /// The transaction being spent is not available in the database or the blockchain client
+    MissingInputTx(Txid),
+    /// The transaction being spent doesn't have the requested output
+    InvalidInput(OutPoint),
+
+    /// Consensus error
+    Consensus(bitcoinconsensus::Error),
+
+    /// Generic error
+    ///
+    /// It has to be wrapped in a `Box` since `Error` has a variant that contains this enum
+    Global(Box<Error>),
+}
+
+impl fmt::Display for VerifyError {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            Self::MissingInputTx(txid) => write!(f, "The transaction being spent is not available in the database or the blockchain client: {}", txid),
+            Self::InvalidInput(outpoint) => write!(f, "The transaction being spent doesn't have the requested output: {}", outpoint),
+            Self::Consensus(err) => write!(f, "Consensus error: {:?}", err),
+            Self::Global(err) => write!(f, "Generic error: {}", err),
+        }
+    }
+}
+
+impl std::error::Error for VerifyError {}
+
+impl From<Error> for VerifyError {
+    fn from(other: Error) -> Self {
+        VerifyError::Global(Box::new(other))
+    }
+}
+impl_error!(bitcoinconsensus::Error, Consensus, VerifyError);
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::database::{BatchOperations, MemoryDatabase};
+    use assert_matches::assert_matches;
+    use bitcoin::consensus::encode::deserialize;
+    use bitcoin::hashes::hex::FromHex;
+    use bitcoin::{Transaction, Txid};
+
+    struct DummyBlockchain;
+
+    impl GetTx for DummyBlockchain {
+        fn get_tx(&self, _txid: &Txid) -> Result<Option<Transaction>, Error> {
+            Ok(None)
+        }
+    }
+
+    #[test]
+    fn test_verify_fail_unsigned_tx() {
+        // https://blockstream.info/tx/95da344585fcf2e5f7d6cbf2c3df2dcce84f9196f7a7bb901a43275cd6eb7c3f
+        let prev_tx: Transaction = deserialize(&Vec::<u8>::from_hex("020000000101192dea5e66d444380e106f8e53acb171703f00d43fb6b3ae88ca5644bdb7e1000000006b48304502210098328d026ce138411f957966c1cf7f7597ccbb170f5d5655ee3e9f47b18f6999022017c3526fc9147830e1340e04934476a3d1521af5b4de4e98baf49ec4c072079e01210276f847f77ec8dd66d78affd3c318a0ed26d89dab33fa143333c207402fcec352feffffff023d0ac203000000001976a9144bfbaf6afb76cc5771bc6404810d1cc041a6933988aca4b956050000000017a91494d5543c74a3ee98e0cf8e8caef5dc813a0f34b48768cb0700").unwrap()).unwrap();
+        // https://blockstream.info/tx/aca326a724eda9a461c10a876534ecd5ae7b27f10f26c3862fb996f80ea2d45d
+        let signed_tx: Transaction = deserialize(&Vec::<u8>::from_hex("02000000013f7cebd65c27431a90bba7f796914fe8cc2ddfc3f2cbd6f7e5f2fc854534da95000000006b483045022100de1ac3bcdfb0332207c4a91f3832bd2c2915840165f876ab47c5f8996b971c3602201c6c053d750fadde599e6f5c4e1963df0f01fc0d97815e8157e3d59fe09ca30d012103699b464d1d8bc9e47d4fb1cdaa89a1c5783d68363c4dbc4b524ed3d857148617feffffff02836d3c01000000001976a914fc25d6d5c94003bf5b0c7b640a248e2c637fcfb088ac7ada8202000000001976a914fbed3d9b11183209a57999d54d59f67c019e756c88ac6acb0700").unwrap()).unwrap();
+
+        let mut database = MemoryDatabase::new();
+        let blockchain = DummyBlockchain;
+
+        let mut unsigned_tx = signed_tx.clone();
+        for input in &mut unsigned_tx.input {
+            input.script_sig = Default::default();
+            input.witness = Default::default();
+        }
+
+        let result = verify_tx(&signed_tx, &database, &blockchain);
+        assert_matches!(result, Err(VerifyError::MissingInputTx(txid)) if txid == prev_tx.txid(),
+            "Error should be a `MissingInputTx` error"
+        );
+
+        // insert the prev_tx
+        database.set_raw_tx(&prev_tx).unwrap();
+
+        let result = verify_tx(&unsigned_tx, &database, &blockchain);
+        assert_matches!(
+            result,
+            Err(VerifyError::Consensus(_)),
+            "Error should be a `Consensus` error"
+        );
+
+        let result = verify_tx(&signed_tx, &database, &blockchain);
+        assert!(
+            result.is_ok(),
+            "Should work since the TX is correctly signed"
+        );
+    }
+}