Add Peer Manager

The Peer Manager structure is responsible for maintaining a live
directory of connected CBF/Non-CBF peers. It can send and receive
messages from one or multiple peers.

It manages an underlying address manager to fetch network addresses,
and it maintains a ban score for each peer. A threshold banscore (100)
is used to signal banning of a peer. Once a peer is banned it will not
be connected to in subsequent runs.

Peer manager will be used to handle parallel fetching of filter data
from multiple peers.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,26 +0,0 @@
----
-name: Bug report
-about: Create a report to help us improve
-title: ''
-labels: 'bug'
-assignees: ''
-
----
-
-**Describe the bug**  
-<!-- A clear and concise description of what the bug is. -->
-
-**To Reproduce**  
-<!-- Steps or code to reproduce the behavior. -->
-
-**Expected behavior**  
-<!-- A clear and concise description of what you expected to happen. -->
-
-**Build environment**  
- - BDK tag/commit: <!-- e.g. v0.13.0, 3a07614 --> 
- - OS+version: <!-- e.g. ubuntu 20.04.01, macOS 12.0.1, windows -->  
- - Rust/Cargo version: <!-- e.g. 1.56.0 --> 
- - Rust/Cargo target: <!-- e.g. x86_64-apple-darwin, x86_64-unknown-linux-gnu, etc. -->  
-
-**Additional context**  
-<!-- Add any other context about the problem here. --> 

.github/ISSUE_TEMPLATE/summer_project.md

@@ -1,77 +0,0 @@
----
-name: Summer of Bitcoin Project
-about: Template to suggest a new https://www.summerofbitcoin.org/ project.
-title: ''
-labels: 'summer-of-bitcoin'
-assignees: ''
-
----
-
-<!--
-## Overview  
-
-Project ideas are scoped for a university-level student with a basic background in CS and bitcoin 
-fundamentals - achievable over 12-weeks. Below are just a few types of ideas: 
-
- - Low-hanging fruit: Relatively short projects with clear goals; requires basic technical knowledge 
-   and minimal familiarity with the codebase.
- - Core development: These projects derive from the ongoing work from the core of your development 
-   team. The list of features and bugs is never-ending, and help is always welcome.
- - Risky/Exploratory: These projects push the scope boundaries of your development effort. They 
-   might require expertise in an area not covered by your current development team. They might take 
-   advantage of a new technology. There is a reasonable chance that the project might be less 
-   successful, but the potential rewards make it worth the attempt.
- - Infrastructure/Automation: These projects are the code that your organization uses to get its 
-   development work done; for example, projects that improve the automation of releases, regression 
-   tests and automated builds. This is a category where a Summer of Bitcoin student can be really 
-   helpful, doing work that the development team has been putting off while they focus on core 
-   development.
- - Quality Assurance/Testing: Projects that work on and test your project's software development 
-   process. Additionally, projects that involve a thorough test and review of individual PRs.
- - Fun/Peripheral: These projects might not be related to the current core development focus, but 
-   create new innovations and new perspectives for your project.
--->
-
-**Description**
-<!-- Description: 3-7 sentences describing the project background and tasks to be done. -->  
-
-**Expected Outcomes**  
-<!-- Short bullet list describing what is to be accomplished -->   
-
-**Resources**  
-<!-- 2-3 reading materials for candidate to learn about the repo, project, scope etc -->  
-<!-- Recommended reading such as a developer/contributor guide -->  
-<!-- [Another example a paper citation](https://arxiv.org/pdf/1802.08091.pdf) -->  
-<!-- [Another example an existing issue](https://github.com/opencv/opencv/issues/11013) -->  
-<!-- [An existing related module](https://github.com/opencv/opencv_contrib/tree/master/modules/optflow) -->  
-
-**Skills Required**  
-<!-- 3-4 technical skills that the candidate should know -->  
-<!-- hands on experience with git -->  
-<!-- mastery plus experience coding in C++ -->  
-<!-- basic knowledge in matrix and tensor computations, college course work in cryptography -->  
-<!-- strong mathematical background -->
-<!-- Bonus - has experience with React Native. Best if you have also worked with OSSFuzz -->
-
-**Mentor(s)**  
-<!-- names of mentor(s) for this project go here -->
-
-**Difficulty**  
-<!-- Easy, Medium, Hard -->
-
-**Competency Test (optional)**  
-<!-- 2-3 technical tasks related to the project idea or repository you’d like a candidate to 
-    perform in order to demonstrate competency, good first bugs, warm-up exercises -->
-<!-- ex. Read the instructions here to get Bitcoin core running on your machine -->
-<!-- ex. pick an issue labeled as “newcomer” in the repository, and send a merge request to the 
-    repository. You can also suggest some other improvement that we did not think of yet, or 
-    something that you find interesting or useful -->
-<!-- ex. fixes for coding style are usually easy to do, and are good issues for first time 
-    contributions for those learning how to interact with the project. After you are done with the 
-    coding style issue, try making a different contribution. -->
-<!-- ex. setup a full Debian packaging development environment and learn the basics of Debian 
-    packaging. Then identify and package the missing dependencies to package Specter Desktop -->
-<!-- ex. write a pull parser for CSV files. You'll be judged by the decisions to store the parser 
-    state and how flexible it is to wrap this parser in other scenarios. -->
-<!-- ex. Stretch Goal: Implement some basic metaprogram/app to prove you're very familiar with BDK. 
-    Be prepared to make adjustments as we judge your solution. -->

.github/workflows/code_coverage.yml

@@ -24,14 +24,14 @@ jobs:
       - name: Update toolchain
         run: rustup update
       - name: Test
-        run: cargo test --features all-keys,compiler,esplora,ureq,compact_filters --no-default-features
+        run: cargo test --features all-keys,compiler,esplora,compact_filters --no-default-features
                     
       - id: coverage
         name: Generate coverage
         uses: actions-rs/grcov@v0.1.5
 
       - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@v2
+        uses: codecov/codecov-action@v1
         with:
           file: ${{ steps.coverage.outputs.report }}
           directory: ./coverage/reports/

.github/workflows/cont_integration.yml

@@ -10,29 +10,25 @@ jobs:
     strategy:
       matrix:
         rust:
-          - version: 1.56.0 # STABLE
-            clippy: true
-          - version: 1.46.0 # MSRV
+          - 1.53.0 # STABLE
+          - 1.46.0 # MSRV
         features:
           - default
           - minimal
           - all-keys
-          - minimal,use-esplora-ureq
+          - minimal,esplora
           - key-value-db
           - electrum
           - compact_filters
-          - esplora,ureq,key-value-db,electrum
+          - esplora,key-value-db,electrum
           - compiler
           - rpc
           - verify
-          - async-interface
-          - use-esplora-reqwest
-          - sqlite
     steps:
       - name: checkout
         uses: actions/checkout@v2
       - name: Generate cache key
-        run: echo "${{ matrix.rust.version }} ${{ matrix.features }}" | tee .cache_key
+        run: echo "${{ matrix.rust }} ${{ matrix.features }}" | tee .cache_key
       - name: cache
         uses: actions/cache@v2
         with:
@@ -42,18 +38,16 @@ jobs:
             target
           key: ${{ runner.os }}-cargo-${{ hashFiles('.cache_key') }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
       - name: Set default toolchain
-        run: rustup default ${{ matrix.rust.version }}
+        run: rustup default ${{ matrix.rust }}
       - name: Set profile
         run: rustup set profile minimal
       - name: Add clippy
-        if: ${{ matrix.rust.clippy }}
         run: rustup component add clippy
       - name: Update toolchain
         run: rustup update
       - name: Build
         run: cargo build --features ${{ matrix.features }} --no-default-features
       - name: Clippy
-        if: ${{ matrix.rust.clippy }}
         run: cargo clippy --all-targets --features ${{ matrix.features }} --no-default-features -- -D warnings
       - name: Test
         run: cargo test --features ${{ matrix.features }} --no-default-features
@@ -82,20 +76,15 @@ jobs:
         run: cargo test --features test-md-docs --no-default-features -- doctest::ReadmeDoctests
 
   test-blockchains:
-    name: Blockchain ${{ matrix.blockchain.features }}
+    name: Test ${{ matrix.blockchain.name }}
     runs-on: ubuntu-20.04
     strategy:
       fail-fast: false
       matrix:
         blockchain:
           - name: electrum
-            features: test-electrum,verify
           - name: rpc
-            features: test-rpc
           - name: esplora
-            features: test-esplora,use-esplora-reqwest,verify
-          - name: esplora
-            features: test-esplora,use-esplora-ureq,verify
     steps:
       - name: Checkout
         uses: actions/checkout@v2
@@ -104,6 +93,8 @@ jobs:
         with:
           path: |
             ~/.cargo/registry
+            ~/.cargo/bitcoin
+            ~/.cargo/electrs
             ~/.cargo/git
             target
           key: ${{ runner.os }}-cargo-${{ github.job }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
@@ -113,11 +104,11 @@ jobs:
           toolchain: stable
           override: true
       - name: Test
-        run: cargo test --no-default-features --features ${{ matrix.blockchain.features }} ${{ matrix.blockchain.name }}::bdk_blockchain_tests
-  
+        run: cargo test --features test-${{ matrix.blockchain.name }} ${{ matrix.blockchain.name }}::bdk_blockchain_tests
+
   check-wasm:
     name: Check WASM
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-16.04
     env:
       CC: clang-10
       CFLAGS: -I/usr/include
@@ -134,11 +125,11 @@ jobs:
           key: ${{ runner.os }}-cargo-${{ github.job }}-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
         # Install a recent version of clang that supports wasm32
       - run: wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - || exit 1
-      - run: sudo apt-add-repository "deb http://apt.llvm.org/focal/ llvm-toolchain-focal-10 main" || exit 1
+      - run: sudo apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-10 main" || exit 1
       - run: sudo apt-get update || exit 1
       - run: sudo apt-get install -y libclang-common-10-dev clang-10 libc6-dev-i386 || exit 1
       - name: Set default toolchain
-        run: rustup default 1.56.0 # STABLE
+        run: rustup default 1.53.0 # STABLE
       - name: Set profile
         run: rustup set profile minimal
       - name: Add target wasm32
@@ -146,7 +137,7 @@ jobs:
       - name: Update toolchain
         run: rustup update
       - name: Check
-        run: cargo check --target wasm32-unknown-unknown --features use-esplora-reqwest --no-default-features
+        run: cargo check --target wasm32-unknown-unknown --features esplora --no-default-features
 
   fmt:
     name: Rust fmt

.github/workflows/nightly_docs.yml

@@ -18,13 +18,13 @@ jobs:
             target
           key: nightly-docs-${{ hashFiles('**/Cargo.toml','**/Cargo.lock') }}
       - name: Set default toolchain
-        run: rustup default nightly-2022-01-25
+        run: rustup default nightly
       - name: Set profile
         run: rustup set profile minimal
       - name: Update toolchain
         run: rustup update
       - name: Build docs
-        run: cargo rustdoc --verbose --features=compiler,electrum,esplora,ureq,compact_filters,key-value-db,all-keys,sqlite -- --cfg docsrs -Dwarnings
+        run: cargo rustdoc --verbose --features=compiler,electrum,esplora,compact_filters,key-value-db,all-keys -- --cfg docsrs -Dwarnings
       - name: Upload artifact
         uses: actions/upload-artifact@v2
         with:
@@ -44,18 +44,18 @@ jobs:
           repository: bitcoindevkit/bitcoindevkit.org
           ref: master
       - name: Create directories
-        run: mkdir -p ./docs/.vuepress/public/docs-rs/bdk/nightly
+        run: mkdir -p ./static/docs-rs/bdk/nightly
       - name: Remove old latest
-        run: rm -rf ./docs/.vuepress/public/docs-rs/bdk/nightly/latest
+        run: rm -rf ./static/docs-rs/bdk/nightly/latest
       - name: Download built docs
         uses: actions/download-artifact@v1
         with:
           name: built-docs
-          path: ./docs/.vuepress/public/docs-rs/bdk/nightly/latest
+          path: ./static/docs-rs/bdk/nightly/latest
       - name: Configure git
         run: git config user.email "github-actions@github.com" && git config user.name "github-actions"
       - name: Commit
         continue-on-error: true         # If there's nothing to commit this step fails, but it's fine
-        run: git add ./docs/.vuepress/public/docs-rs && git commit -m "Publish autogenerated nightly docs"
+        run: git add ./static && git commit -m "Publish autogenerated nightly docs"
       - name: Push
         run: git push origin master

CHANGELOG.md

@@ -6,76 +6,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-## [v0.17.0] - [v0.16.1]
-
-- Removed default verification from `wallet::sync`. sync-time verification is added in `script_sync` and is activated by `verify` feature flag.
-- `verify` flag removed from `TransactionDetails`.
-- Add `get_internal_address` to allow you to get internal addresses just as you get external addresses.
-- added `ensure_addresses_cached` to `Wallet` to let offline wallets load and cache addresses in their database
-
-### Sync API change
-
-To decouple the `Wallet` from the `Blockchain` we've made major changes:
-
-- Removed `Blockchain` from Wallet.
-- Removed `Wallet::broadcast` (just use `Blockchain::broadcast`)
-- Deprecated `Wallet::new_offline` (all wallets are offline now)
-- Changed `Wallet::sync` to take a `Blockchain`.
-- Stop making a request for the block height when calling `Wallet:new`.
-- Added `SyncOptions` to capture extra (future) arguments to `Wallet::sync`.
-- Removed `max_addresses` sync parameter which determined how many addresses to cache before syncing since this can just be done with `ensure_addresses_cached`.
-
-## [v0.16.1] - [v0.16.0]
-
-- Pin tokio dependency version to ~1.14 to prevent errors due to their new MSRV 1.49.0
-
-## [v0.16.0] - [v0.15.0]
-
-- Disable `reqwest` default features.
-- Added `reqwest-default-tls` feature: Use this to restore the TLS defaults of reqwest if you don't want to add a dependency to it in your own manifest.
-- Use dust_value from rust-bitcoin
-- Fixed generating WIF in the correct network format.
-
-## [v0.15.0] - [v0.14.0]
-
-- Overhauled sync logic for electrum and esplora.
-- Unify ureq and reqwest esplora backends to have the same configuration parameters. This means reqwest now has a timeout parameter and ureq has a concurrency parameter.
-- Fixed esplora fee estimation.
-
-## [v0.14.0] - [v0.13.0]
-
-- BIP39 implementation dependency, in `keys::bip39` changed from tiny-bip39 to rust-bip39.
-- Add new method on the `TxBuilder` to embed data in the transaction via `OP_RETURN`. To allow that a fix to check the dust only on spendable output has been introduced.
-- Update the `Database` trait to store the last sync timestamp and block height
-- Rename `ConfirmationTime` to `BlockTime`
-
-## [v0.13.0] - [v0.12.0]
-
-- Exposed `get_tx()` method from `Database` to `Wallet`.
-
-## [v0.12.0] - [v0.11.0]
-
-- Activate `miniscript/use-serde` feature to allow consumers of the library to access it via the re-exported `miniscript` crate.
-- Add support for proxies in `EsploraBlockchain`
-- Added `SqliteDatabase` that implements `Database` backed by a sqlite database using `rusqlite` crate.
-
-## [v0.11.0] - [v0.10.0]
-
-- Added `flush` method to the `Database` trait to explicitly flush to disk latest changes on the db.
-
-## [v0.10.0] - [v0.9.0]
-
-- Added `RpcBlockchain` in the `AnyBlockchain` struct to allow using Rpc backend where `AnyBlockchain` is used (eg `bdk-cli`)
-- Removed hard dependency on `tokio`.
-
 ### Wallet
 
 - Removed and replaced `set_single_recipient` with more general `drain_to` and replaced `maintain_single_recipient` with `allow_shrinking`.
 
 ### Blockchain
 
-- Removed `stop_gap` from `Blockchain` trait and added it to only `ElectrumBlockchain` and `EsploraBlockchain` structs.
-- Added a `ureq` backend for use when not using feature `async-interface` or target WASM. `ureq` is a blocking HTTP client.
+- Removed `stop_gap` from `Blockchain` trait and added it to only `ElectrumBlockchain` and `EsploraBlockchain` structs  
 
 ## [v0.9.0] - [v0.8.0]
 
@@ -417,6 +354,7 @@ final transaction is created by calling `finish` on the builder.
 - Use `MemoryDatabase` in the compiler example
 - Make the REPL return JSON
 
+[unreleased]: https://github.com/bitcoindevkit/bdk/compare/v0.9.0...HEAD
 [0.1.0-beta.1]: https://github.com/bitcoindevkit/bdk/compare/96c87ea5...0.1.0-beta.1
 [v0.2.0]: https://github.com/bitcoindevkit/bdk/compare/0.1.0-beta.1...v0.2.0
 [v0.3.0]: https://github.com/bitcoindevkit/bdk/compare/v0.2.0...v0.3.0
@@ -427,13 +365,3 @@ final transaction is created by calling `finish` on the builder.
 [v0.7.0]: https://github.com/bitcoindevkit/bdk/compare/v0.6.0...v0.7.0
 [v0.8.0]: https://github.com/bitcoindevkit/bdk/compare/v0.7.0...v0.8.0
 [v0.9.0]: https://github.com/bitcoindevkit/bdk/compare/v0.8.0...v0.9.0
-[v0.10.0]: https://github.com/bitcoindevkit/bdk/compare/v0.9.0...v0.10.0
-[v0.11.0]: https://github.com/bitcoindevkit/bdk/compare/v0.10.0...v0.11.0
-[v0.12.0]: https://github.com/bitcoindevkit/bdk/compare/v0.11.0...v0.12.0
-[v0.13.0]: https://github.com/bitcoindevkit/bdk/compare/v0.12.0...v0.13.0
-[v0.14.0]: https://github.com/bitcoindevkit/bdk/compare/v0.13.0...v0.14.0
-[v0.15.0]: https://github.com/bitcoindevkit/bdk/compare/v0.14.0...v0.15.0
-[v0.16.0]: https://github.com/bitcoindevkit/bdk/compare/v0.15.0...v0.16.0
-[v0.16.1]: https://github.com/bitcoindevkit/bdk/compare/v0.16.0...v0.16.1
-[v0.17.0]: https://github.com/bitcoindevkit/bdk/compare/v0.16.1...v0.17.0
-[unreleased]: https://github.com/bitcoindevkit/bdk/compare/v0.17.0...HEAD

CONTRIBUTING.md

@@ -57,21 +57,6 @@ comment suggesting that you're working on it. If someone is already assigned,
 don't hesitate to ask if the assigned party or previous commenters are still
 working on it if it has been awhile.
 
-Deprecation policy
-------------------
-
-Where possible, breaking existing APIs should be avoided. Instead, add new APIs and
-use [`#[deprecated]`](https://github.com/rust-lang/rfcs/blob/master/text/1270-deprecation.md)
-to discourage use of the old one.
-
-Deprecated APIs are typically maintained for one release cycle. In other words, an
-API that has been deprecated with the 0.10 release can be expected to be removed in the
-0.11 release. This allows for smoother upgrades without incurring too much technical
-debt inside this library.
-
-If you deprecated an API as part of a contribution, we encourage you to "own" that API
-and send a follow-up to remove it as part of the next release cycle.
-
 Peer review
 -----------
 

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "bdk"
-version = "0.17.1-dev"
+version = "0.9.1-dev"
 edition = "2018"
 authors = ["Alekos Filini <alekos.filini@gmail.com>", "Riccardo Casatta <riccardo@casatta.it>"]
 homepage = "https://bitcoindevkit.org"
@@ -12,37 +12,34 @@ readme = "README.md"
 license = "MIT OR Apache-2.0"
 
 [dependencies]
-bdk-macros = "^0.6"
+bdk-macros = "^0.4"
 log = "^0.4"
-miniscript = { version = "^6.0", features = ["use-serde"] }
-bitcoin = { version = "^0.27", features = ["use-serde", "base64"] }
+miniscript = "5.1"
+bitcoin = { version = "~0.26.2", features = ["use-serde", "base64"] }
 serde = { version = "^1.0", features = ["derive"] }
 serde_json = { version = "^1.0" }
 rand = "^0.7"
 
 # Optional dependencies
 sled = { version = "0.34", optional = true }
-electrum-client = { version = "0.8", optional = true }
-rusqlite = { version = "0.25.3", optional = true }
-ahash = { version = "=0.7.4", optional = true }
-reqwest = { version = "0.11", optional = true, default-features = false, features = ["json"] }
-ureq = { version = "~2.2.0", features = ["json"], optional = true }
+electrum-client = { version = "0.7", optional = true }
+reqwest = { version = "0.11", optional = true, features = ["json"] }
 futures = { version = "0.3", optional = true }
 async-trait = { version = "0.1", optional = true }
-rocksdb = { version = "0.14", default-features = false, features = ["snappy"], optional = true }
+rocksdb = { version = "0.14", optional = true }
 cc = { version = ">=1.0.64", optional = true }
 socks = { version = "0.3", optional = true }
 lazy_static = { version = "1.4", optional = true }
-
-bip39 = { version = "1.0.1", optional = true }
+tiny-bip39 = { version = "^0.8", optional = true }
+zeroize = { version = "<1.4.0", optional = true }
 bitcoinconsensus = { version = "0.19.0-3", optional = true }
 
 # Needed by bdk_blockchain_tests macro
-bitcoincore-rpc = { version = "0.14", optional = true }
+bitcoincore-rpc = { version = "0.13", optional = true }
 
 # Platform-specific dependencies
 [target.'cfg(not(target_arch = "wasm32"))'.dependencies]
-tokio = { version = "~1.14", features = ["rt"] }
+tokio = { version = "1", features = ["rt"] }
 
 [target.'cfg(target_arch = "wasm32")'.dependencies]
 async-trait = "0.1"
@@ -54,49 +51,28 @@ minimal = []
 compiler = ["miniscript/compiler"]
 verify = ["bitcoinconsensus"]
 default = ["key-value-db", "electrum"]
-sqlite = ["rusqlite", "ahash"]
+electrum = ["electrum-client"]
+esplora = ["reqwest", "futures"]
 compact_filters = ["rocksdb", "socks", "lazy_static", "cc"]
 key-value-db = ["sled"]
+async-interface = ["async-trait"]
 all-keys = ["keys-bip39"]
-keys-bip39 = ["bip39"]
+keys-bip39 = ["tiny-bip39", "zeroize"]
 rpc = ["bitcoincore-rpc"]
 
-# We currently provide mulitple implementations of `Blockchain`, all are
-# blocking except for the `EsploraBlockchain` which can be either async or
-# blocking, depending on the HTTP client in use.
-#
-# - Users wanting asynchronous HTTP calls should enable `async-interface` to get
-#   access to the asynchronous method implementations. Then, if Esplora is wanted,
-#   enable `esplora` AND `reqwest` (`--features=use-esplora-reqwest`).
-# - Users wanting blocking HTTP calls can use any of the other blockchain
-#   implementations (`compact_filters`, `electrum`, or `esplora`). Users wanting to
-#   use Esplora should enable `esplora` AND `ureq`  (`--features=use-esplora-ureq`).
-#
-# WARNING: Please take care with the features below, various combinations will
-# fail to build. We cannot currently build `bdk` with `--all-features`.
-async-interface = ["async-trait"]
-electrum = ["electrum-client"]
-# MUST ALSO USE `--no-default-features`.
-use-esplora-reqwest = ["esplora", "reqwest", "reqwest/socks", "futures"]
-use-esplora-ureq = ["esplora", "ureq", "ureq/socks"]
-# Typical configurations will not need to use `esplora` feature directly.
-esplora = []
-
-# Use below feature with `use-esplora-reqwest` to enable reqwest default TLS support
-reqwest-default-tls = ["reqwest/default-tls"]
 
 # Debug/Test features
 test-blockchains = ["bitcoincore-rpc", "electrum-client"]
 test-electrum = ["electrum", "electrsd/electrs_0_8_10", "test-blockchains"]
 test-rpc = ["rpc", "electrsd/electrs_0_8_10", "test-blockchains"]
-test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]
+test-esplora = ["esplora", "electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]
 test-md-docs = ["electrum"]
 
 [dev-dependencies]
 lazy_static = "1.4"
 env_logger = "0.7"
 clap = "2.33"
-electrsd = { version= "0.13", features = ["trigger", "bitcoind_22_0"] }
+electrsd = { version= "0.6", features = ["trigger", "bitcoind_0_21_1"] }
 
 [[example]]
 name = "address_validator"
@@ -112,6 +88,6 @@ required-features = ["compiler"]
 [workspace]
 members = ["macros"]
 [package.metadata.docs.rs]
-features = ["compiler", "electrum", "esplora", "ureq", "compact_filters", "rpc", "key-value-db", "sqlite", "all-keys", "verify"]
+features = ["compiler", "electrum", "esplora", "compact_filters", "rpc", "key-value-db", "all-keys", "verify"]
 # defines the configuration attribute `docsrs`
 rustdoc-args = ["--cfg", "docsrs"]

DEVELOPMENT_CYCLE.md

@@ -32,14 +32,14 @@ Pre-`v1.0.0` our "major" releases only affect the "minor" semver value. Accordin
     - If it's a minor issue you can just fix it in the release branch, since it will be merged back to `master` eventually
     - For bigger issues you can fix them on `master` and then *cherry-pick* the commit to the release branch
 6. Update the changelog with the new release version.
-7. Update `src/lib.rs` with the new version (line ~43)
+7. Update `src/lib.rs` with the new version (line ~59)
 8. On release day, make a commit on the release branch to bump the version to `x.y.z`. The message should be "Bump version to x.y.z".
 9. Add a tag to this commit. The tag name should be `vx.y.z` (for example `v0.5.0`), and the message "Release x.y.z". Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
 10. Push the new commits to the upstream release branch, wait for the CI to finish one last time.
 11. Publish **all** the updated crates to crates.io.
 12. Make a new commit to bump the version value to `x.y.(z+1)-dev`. The message should be "Bump version to x.y.(z+1)-dev".
 13. Merge the release branch back into `master`.
-14. If the `master` branch contains any unreleased changes to the `bdk-macros` crate, change the `bdk` Cargo.toml `[dependencies]` to point to the local path (ie. `bdk-macros = { path = "./macros"}`)
+14. If the `master` branch contains any unreleased changes to the `bdk-macros`, `bdk-testutils`, or `bdk-testutils-macros` crates, change the `bdk` Cargo.toml `[dev-dependencies]` to point to the local path (ie. `bdk-testutils-macros = { path = "./testutils-macros"}`)
 15. Create the release on GitHub: go to "tags", click on the dots on the right and select "Create Release". Then set the title to `vx.y.z` and write down some brief release notes.
 16. Make sure the new release shows up on crates.io and that the docs are built correctly on docs.rs.
 17. Announce the release on Twitter, Discord and Telegram.

README.md

@@ -1,7 +1,7 @@
 <div align="center">
   <h1>BDK</h1>
 
-  <img src="./static/bdk.png" width="220" />
+  <img src="./static/bdk.svg" width="220" />
 
   <p>
     <strong>A modern, lightweight, descriptor-based wallet library written in Rust!</strong>
@@ -41,21 +41,21 @@ The `bdk` library aims to be the core building block for Bitcoin wallets of any
 ```rust,no_run
 use bdk::Wallet;
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::ElectrumBlockchain;
-use bdk::SyncOptions;
+use bdk::blockchain::{noop_progress, ElectrumBlockchain};
 
 use bdk::electrum_client::Client;
 
 fn main() -> Result<(), bdk::Error> {
-    let blockchain = ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
+    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
+        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(&blockchain, SyncOptions::default())?;
+    wallet.sync(noop_progress(), None)?;
 
     println!("Descriptor balance: {} SAT", wallet.get_balance()?);
 
@@ -70,7 +70,7 @@ use bdk::{Wallet, database::MemoryDatabase};
 use bdk::wallet::AddressIndex::New;
 
 fn main() -> Result<(), bdk::Error> {
-    let wallet = Wallet::new(
+    let wallet = Wallet::new_offline(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
@@ -88,9 +88,9 @@ fn main() -> Result<(), bdk::Error> {
 ### Create a transaction
 
 ```rust,no_run
-use bdk::{FeeRate, Wallet, SyncOptions};
+use bdk::{FeeRate, Wallet};
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::ElectrumBlockchain;
+use bdk::blockchain::{noop_progress, ElectrumBlockchain};
 
 use bdk::electrum_client::Client;
 use bdk::wallet::AddressIndex::New;
@@ -98,15 +98,16 @@ use bdk::wallet::AddressIndex::New;
 use bitcoin::consensus::serialize;
 
 fn main() -> Result<(), bdk::Error> {
-    let blockchain = ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
+    let client = Client::new("ssl://electrum.blockstream.info:60002")?;
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
+        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(&blockchain, SyncOptions::default())?;
+    wallet.sync(noop_progress(), None)?;
 
     let send_to = wallet.get_address(New)?;
     let (psbt, details) = {
@@ -134,7 +135,7 @@ use bdk::{Wallet, SignOptions, database::MemoryDatabase};
 use bitcoin::consensus::deserialize;
 
 fn main() -> Result<(), bdk::Error> {
-    let wallet = Wallet::new(
+    let wallet = Wallet::new_offline(
         "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
         bitcoin::Network::Testnet,

examples/address_validator.rs

@@ -48,7 +48,8 @@ impl AddressValidator for DummyValidator {
 
 fn main() -> Result<(), bdk::Error> {
     let descriptor = "sh(and_v(v:pk(tpubDDpWvmUrPZrhSPmUzCMBHffvC3HyMAPnWDSAQNBTnj1iZeJa7BZQEttFiP4DS4GCcXQHezdXhn86Hj6LHX5EDstXPWrMaSneRWM8yUf6NFd/*),after(630000)))";
-    let mut wallet = Wallet::new(descriptor, None, Network::Regtest, MemoryDatabase::new())?;
+    let mut wallet =
+        Wallet::new_offline(descriptor, None, Network::Regtest, MemoryDatabase::new())?;
 
     wallet.add_address_validator(Arc::new(DummyValidator));
 

examples/compact_filters_balance.rs

@@ -10,6 +10,7 @@
 // licenses.
 
 use bdk::blockchain::compact_filters::*;
+use bdk::blockchain::noop_progress;
 use bdk::database::MemoryDatabase;
 use bdk::*;
 use bitcoin::*;
@@ -34,8 +35,9 @@ fn main() -> Result<(), CompactFiltersError> {
     let descriptor = "wpkh(tpubD6NzVbkrYhZ4X2yy78HWrr1M9NT8dKeWfzNiQqDdMqqa9UmmGztGGz6TaLFGsLfdft5iu32gxq1T4eMNxExNNWzVCpf9Y6JZi5TnqoC9wJq/*)";
 
     let database = MemoryDatabase::default();
-    let wallet = Arc::new(Wallet::new(descriptor, None, Network::Testnet, database).unwrap());
-    wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+    let wallet =
+        Arc::new(Wallet::new(descriptor, None, Network::Testnet, database, blockchain).unwrap());
+    wallet.sync(noop_progress(), None).unwrap();
     info!("balance: {}", wallet.get_balance()?);
     Ok(())
 }

examples/compiler.rs

@@ -70,7 +70,7 @@ fn main() -> Result<(), Box<dyn Error>> {
     let policy_str = matches.value_of("POLICY").unwrap();
     info!("Compiling policy: {}", policy_str);
 
-    let policy = Concrete::<String>::from_str(policy_str)?;
+    let policy = Concrete::<String>::from_str(&policy_str)?;
 
     let descriptor = match matches.value_of("TYPE").unwrap() {
         "sh" => Descriptor::new_sh(policy.compile()?)?,
@@ -89,7 +89,7 @@ fn main() -> Result<(), Box<dyn Error>> {
         .transpose()
         .unwrap()
         .unwrap_or(Network::Testnet);
-    let wallet = Wallet::new(&format!("{}", descriptor), None, network, database)?;
+    let wallet = Wallet::new_offline(&format!("{}", descriptor), None, network, database)?;
 
     info!("... First address: {}", wallet.get_address(New)?);
 

macros/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "bdk-macros"
-version = "0.6.0"
+version = "0.4.0"
 authors = ["Alekos Filini <alekos.filini@gmail.com>"]
 edition = "2018"
 homepage = "https://bitcoindevkit.org"

src/blockchain/any.rs

@@ -16,18 +16,59 @@
 //!
 //! ## Example
 //!
-//! When paired with the use of [`ConfigurableBlockchain`], it allows creating any
+//! In this example both `wallet_electrum` and `wallet_esplora` have the same type of
+//! `Wallet<AnyBlockchain, MemoryDatabase>`. This means that they could both, for instance, be
+//! assigned to a struct member.
+//!
+//! ```no_run
+//! # use bitcoin::Network;
+//! # use bdk::blockchain::*;
+//! # use bdk::database::MemoryDatabase;
+//! # use bdk::Wallet;
+//! # #[cfg(feature = "electrum")]
+//! # {
+//! let electrum_blockchain = ElectrumBlockchain::from(electrum_client::Client::new("...")?);
+//! let wallet_electrum: Wallet<AnyBlockchain, _> = Wallet::new(
+//!     "...",
+//!     None,
+//!     Network::Testnet,
+//!     MemoryDatabase::default(),
+//!     electrum_blockchain.into(),
+//! )?;
+//! # }
+//!
+//! # #[cfg(feature = "esplora")]
+//! # {
+//! let esplora_blockchain = EsploraBlockchain::new("...", None, 20);
+//! let wallet_esplora: Wallet<AnyBlockchain, _> = Wallet::new(
+//!     "...",
+//!     None,
+//!     Network::Testnet,
+//!     MemoryDatabase::default(),
+//!     esplora_blockchain.into(),
+//! )?;
+//! # }
+//!
+//! # Ok::<(), bdk::Error>(())
+//! ```
+//!
+//! When paired with the use of [`ConfigurableBlockchain`], it allows creating wallets with any
 //! blockchain type supported using a single line of code:
 //!
 //! ```no_run
 //! # use bitcoin::Network;
 //! # use bdk::blockchain::*;
-//! # #[cfg(all(feature = "esplora", feature = "ureq"))]
-//! # {
+//! # use bdk::database::MemoryDatabase;
+//! # use bdk::Wallet;
 //! let config = serde_json::from_str("...")?;
 //! let blockchain = AnyBlockchain::from_config(&config)?;
-//! let height = blockchain.get_height();
-//! # }
+//! let wallet = Wallet::new(
+//!     "...",
+//!     None,
+//!     Network::Testnet,
+//!     MemoryDatabase::default(),
+//!     blockchain,
+//! )?;
 //! # Ok::<(), bdk::Error>(())
 //! ```
 
@@ -53,8 +94,6 @@ macro_rules! impl_inner_method {
             AnyBlockchain::Esplora(inner) => inner.$name( $($args, )* ),
             #[cfg(feature = "compact_filters")]
             AnyBlockchain::CompactFilters(inner) => inner.$name( $($args, )* ),
-            #[cfg(feature = "rpc")]
-            AnyBlockchain::Rpc(inner) => inner.$name( $($args, )* ),
         }
     }
 }
@@ -77,10 +116,6 @@ pub enum AnyBlockchain {
     #[cfg_attr(docsrs, doc(cfg(feature = "compact_filters")))]
     /// Compact filters client
     CompactFilters(compact_filters::CompactFiltersBlockchain),
-    #[cfg(feature = "rpc")]
-    #[cfg_attr(docsrs, doc(cfg(feature = "rpc")))]
-    /// RPC client
-    Rpc(rpc::RpcBlockchain),
 }
 
 #[maybe_async]
@@ -89,62 +124,39 @@ impl Blockchain for AnyBlockchain {
         maybe_await!(impl_inner_method!(self, get_capabilities))
     }
 
+    fn setup<D: BatchDatabase, P: 'static + Progress>(
+        &self,
+        database: &mut D,
+        progress_update: P,
+    ) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(self, setup, database, progress_update))
+    }
+    fn sync<D: BatchDatabase, P: 'static + Progress>(
+        &self,
+        database: &mut D,
+        progress_update: P,
+    ) -> Result<(), Error> {
+        maybe_await!(impl_inner_method!(self, sync, database, progress_update))
+    }
+
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        maybe_await!(impl_inner_method!(self, get_tx, txid))
+    }
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         maybe_await!(impl_inner_method!(self, broadcast, tx))
     }
 
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
-        maybe_await!(impl_inner_method!(self, estimate_fee, target))
-    }
-}
-
-#[maybe_async]
-impl GetHeight for AnyBlockchain {
     fn get_height(&self) -> Result<u32, Error> {
         maybe_await!(impl_inner_method!(self, get_height))
     }
-}
-
-#[maybe_async]
-impl GetTx for AnyBlockchain {
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        maybe_await!(impl_inner_method!(self, get_tx, txid))
-    }
-}
-
-#[maybe_async]
-impl WalletSync for AnyBlockchain {
-    fn wallet_sync<D: BatchDatabase>(
-        &self,
-        database: &mut D,
-        progress_update: Box<dyn Progress>,
-    ) -> Result<(), Error> {
-        maybe_await!(impl_inner_method!(
-            self,
-            wallet_sync,
-            database,
-            progress_update
-        ))
-    }
-
-    fn wallet_setup<D: BatchDatabase>(
-        &self,
-        database: &mut D,
-        progress_update: Box<dyn Progress>,
-    ) -> Result<(), Error> {
-        maybe_await!(impl_inner_method!(
-            self,
-            wallet_setup,
-            database,
-            progress_update
-        ))
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        maybe_await!(impl_inner_method!(self, estimate_fee, target))
     }
 }
 
 impl_from!(electrum::ElectrumBlockchain, AnyBlockchain, Electrum, #[cfg(feature = "electrum")]);
 impl_from!(esplora::EsploraBlockchain, AnyBlockchain, Esplora, #[cfg(feature = "esplora")]);
 impl_from!(compact_filters::CompactFiltersBlockchain, AnyBlockchain, CompactFilters, #[cfg(feature = "compact_filters")]);
-impl_from!(rpc::RpcBlockchain, AnyBlockchain, Rpc, #[cfg(feature = "rpc")]);
 
 /// Type that can contain any of the blockchain configurations defined by the library
 ///
@@ -194,10 +206,6 @@ pub enum AnyBlockchainConfig {
     #[cfg_attr(docsrs, doc(cfg(feature = "compact_filters")))]
     /// Compact filters client
     CompactFilters(compact_filters::CompactFiltersBlockchainConfig),
-    #[cfg(feature = "rpc")]
-    #[cfg_attr(docsrs, doc(cfg(feature = "rpc")))]
-    /// RPC client configuration
-    Rpc(rpc::RpcConfig),
 }
 
 impl ConfigurableBlockchain for AnyBlockchain {
@@ -217,10 +225,6 @@ impl ConfigurableBlockchain for AnyBlockchain {
             AnyBlockchainConfig::CompactFilters(inner) => AnyBlockchain::CompactFilters(
                 compact_filters::CompactFiltersBlockchain::from_config(inner)?,
             ),
-            #[cfg(feature = "rpc")]
-            AnyBlockchainConfig::Rpc(inner) => {
-                AnyBlockchain::Rpc(rpc::RpcBlockchain::from_config(inner)?)
-            }
         })
     }
 }
@@ -228,4 +232,3 @@ impl ConfigurableBlockchain for AnyBlockchain {
 impl_from!(electrum::ElectrumBlockchainConfig, AnyBlockchainConfig, Electrum, #[cfg(feature = "electrum")]);
 impl_from!(esplora::EsploraBlockchainConfig, AnyBlockchainConfig, Esplora, #[cfg(feature = "esplora")]);
 impl_from!(compact_filters::CompactFiltersBlockchainConfig, AnyBlockchainConfig, CompactFilters, #[cfg(feature = "compact_filters")]);
-impl_from!(rpc::RpcConfig, AnyBlockchainConfig, Rpc, #[cfg(feature = "rpc")]);

src/blockchain/compact_filters/address_manager.rs

@@ -0,0 +1,764 @@
+// Bitcoin Dev Kit
+// Written in 2021 by Rajarshi Maitra <rajarshi149@gmail.com>
+//                    John Cantrell <johncantrell97@protonmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use std::fs::File;
+use std::io::prelude::*;
+
+use std::collections::HashSet;
+use std::collections::VecDeque;
+use std::sync::{
+    mpsc::{channel, Receiver, SendError, Sender},
+    Arc, RwLock,
+};
+use std::thread::{self, JoinHandle};
+use std::time::Duration;
+
+use std::net::{SocketAddr, ToSocketAddrs};
+
+use std::sync::PoisonError;
+use std::sync::{MutexGuard, RwLockReadGuard, RwLockWriteGuard, WaitTimeoutResult};
+
+use serde::{Deserialize, Serialize};
+use serde_json::Error as SerdeError;
+
+use super::{Mempool, Peer, PeerError};
+
+use bitcoin::network::{
+    constants::{Network, ServiceFlags},
+    message::NetworkMessage,
+    Address,
+};
+
+/// Default address pool minimums
+const MIN_CBF_BUFFER: usize = 5;
+const MIN_NONCBF_BUFFER: usize = 5;
+
+/// A Discovery structure used by workers
+///
+/// Discovery can be initiated via a cache,
+/// Or it will start with default hardcoded seeds
+pub struct AddressDiscovery {
+    pending: VecDeque<SocketAddr>,
+    visited: HashSet<SocketAddr>,
+}
+
+impl AddressDiscovery {
+    fn new(network: Network, seeds: VecDeque<SocketAddr>) -> AddressDiscovery {
+        let mut network_seeds = AddressDiscovery::seeds(network);
+        let mut total_seeds = seeds;
+        total_seeds.append(&mut network_seeds);
+        AddressDiscovery {
+            pending: total_seeds,
+            visited: HashSet::new(),
+        }
+    }
+
+    fn add_pendings(&mut self, addresses: Vec<SocketAddr>) {
+        for addr in addresses {
+            if !self.pending.contains(&addr) && !self.visited.contains(&addr) {
+                self.pending.push_back(addr);
+            }
+        }
+    }
+
+    fn get_next(&mut self) -> Option<SocketAddr> {
+        match self.pending.pop_front() {
+            None => None,
+            Some(next) => {
+                self.visited.insert(next);
+                Some(next)
+            }
+        }
+    }
+
+    fn seeds(network: Network) -> VecDeque<SocketAddr> {
+        let mut seeds = VecDeque::new();
+
+        let port: u16 = match network {
+            Network::Bitcoin => 8333,
+            Network::Testnet => 18333,
+            Network::Regtest => 18444,
+            Network::Signet => 38333,
+        };
+
+        let seedhosts: &[&str] = match network {
+            Network::Bitcoin => &[
+                "seed.bitcoin.sipa.be",
+                "dnsseed.bluematt.me",
+                "dnsseed.bitcoin.dashjr.org",
+                "seed.bitcoinstats.com",
+                "seed.bitcoin.jonasschnelli.ch",
+                "seed.btc.petertodd.org",
+                "seed.bitcoin.sprovoost.nl",
+                "dnsseed.emzy.de",
+                "seed.bitcoin.wiz.biz",
+            ],
+            Network::Testnet => &[
+                "testnet-seed.bitcoin.jonasschnelli.ch",
+                "seed.tbtc.petertodd.org",
+                "seed.testnet.bitcoin.sprovoost.nl",
+                "testnet-seed.bluematt.me",
+            ],
+            Network::Regtest => &[],
+            Network::Signet => &[],
+        };
+
+        for seedhost in seedhosts.iter() {
+            if let Ok(lookup) = (*seedhost, port).to_socket_addrs() {
+                for host in lookup {
+                    if host.is_ipv4() {
+                        seeds.push_back(host);
+                    }
+                }
+            }
+        }
+        seeds
+    }
+}
+
+/// Crawler structure that will interface with Discovery and public bitcoin network
+///
+/// Address manager will spawn multiple crawlers in separate threads to discover new addresses.
+struct AddressWorker {
+    discovery: Arc<RwLock<AddressDiscovery>>,
+    sender: Sender<(SocketAddr, ServiceFlags)>,
+    network: Network,
+}
+
+impl AddressWorker {
+    fn new(
+        discovery: Arc<RwLock<AddressDiscovery>>,
+        sender: Sender<(SocketAddr, ServiceFlags)>,
+        network: Network,
+    ) -> AddressWorker {
+        AddressWorker {
+            discovery,
+            sender,
+            network,
+        }
+    }
+
+    fn try_receive_addr(&mut self, peer: &Peer) -> Result<(), AddressManagerError> {
+        if let Some(NetworkMessage::Addr(new_addresses)) =
+            peer.recv("addr", Some(Duration::from_secs(1)))?
+        {
+            self.consume_addr(new_addresses)?;
+        }
+
+        Ok(())
+    }
+
+    fn consume_addr(&mut self, addrs: Vec<(u32, Address)>) -> Result<(), AddressManagerError> {
+        let mut discovery_lock = self.discovery.write().map_err(PeerError::from)?;
+        let mut addresses = Vec::new();
+        for network_addrs in addrs {
+            if let Ok(socket_addrs) = network_addrs.1.socket_addr() {
+                addresses.push(socket_addrs);
+            }
+        }
+        discovery_lock.add_pendings(addresses);
+
+        Ok(())
+    }
+
+    fn work(&mut self) -> Result<(), AddressManagerError> {
+        loop {
+            let next_address = {
+                let mut address_discovery = self.discovery.write()?;
+                address_discovery.get_next()
+            };
+
+            match next_address {
+                Some(address) => {
+                    let potential_peer = Peer::connect_with_timeout(
+                        address,
+                        Duration::from_secs(1),
+                        Arc::new(Mempool::default()),
+                        self.network,
+                    );
+
+                    if let Ok(peer) = potential_peer {
+                        peer.send(NetworkMessage::GetAddr)?;
+                        self.try_receive_addr(&peer)?;
+                        self.try_receive_addr(&peer)?;
+                        self.sender.send((address, peer.get_version().services))?;
+                        // TODO: Investigate why close is being called on non existent connections
+                        // currently the errors are ignored
+                        peer.close().unwrap_or(());
+                    }
+                }
+                None => continue,
+            }
+        }
+    }
+}
+
+/// A dedicated cache structure, with cbf/non_cbf separation
+///
+/// [AddressCache] will interface with file i/o
+/// And can te turned into seeds. Generation of seed will put previously cached
+/// cbf addresses at front of the vec, to boost up cbf node findings
+#[derive(Serialize, Deserialize)]
+struct AddressCache {
+    banned_peers: HashSet<SocketAddr>,
+    cbf: HashSet<SocketAddr>,
+    non_cbf: HashSet<SocketAddr>,
+}
+
+impl AddressCache {
+    fn empty() -> Self {
+        Self {
+            banned_peers: HashSet::new(),
+            cbf: HashSet::new(),
+            non_cbf: HashSet::new(),
+        }
+    }
+
+    fn from_file(path: &str) -> Result<Option<Self>, AddressManagerError> {
+        let serialized: Result<String, _> = std::fs::read_to_string(path);
+        let serialized = match serialized {
+            Ok(contents) => contents,
+            Err(_) => return Ok(None),
+        };
+
+        let address_cache = serde_json::from_str(&serialized)?;
+        Ok(Some(address_cache))
+    }
+
+    fn write_to_file(&self, path: &str) -> Result<(), AddressManagerError> {
+        let serialized = serde_json::to_string_pretty(&self)?;
+
+        let mut cache_file = File::create(path)?;
+
+        cache_file.write_all(serialized.as_bytes())?;
+
+        Ok(())
+    }
+
+    fn make_seeds(&self) -> VecDeque<SocketAddr> {
+        self.cbf
+            .iter()
+            .chain(self.non_cbf.iter())
+            .copied()
+            .collect()
+    }
+
+    fn remove_address(&mut self, addrs: &SocketAddr, cbf: bool) -> bool {
+        if cbf {
+            self.cbf.remove(addrs)
+        } else {
+            self.non_cbf.remove(addrs)
+        }
+    }
+
+    fn add_address(&mut self, addrs: SocketAddr, cbf: bool) -> bool {
+        if cbf {
+            self.cbf.insert(addrs)
+        } else {
+            self.non_cbf.insert(addrs)
+        }
+    }
+
+    fn add_to_banlist(&mut self, addrs: SocketAddr, cbf: bool) {
+        if self.banned_peers.insert(addrs) {
+            self.remove_address(&addrs, cbf);
+        }
+    }
+}
+
+/// A Live directory maintained by [AddressManager] of freshly found cbf and non_cbf nodes by workers
+///
+/// Each instance of new [AddressManager] with have fresh [AddressDirectory]
+/// This is independent from the cache and will be an in-memory database to
+/// fetch addresses to the user.  
+struct AddressDirectory {
+    cbf_nodes: HashSet<SocketAddr>,
+    non_cbf_nodes: HashSet<SocketAddr>,
+
+    // List of addresses it has previously provided to the caller (PeerManager)
+    previously_sent: HashSet<SocketAddr>,
+}
+
+impl AddressDirectory {
+    fn new() -> AddressDirectory {
+        AddressDirectory {
+            cbf_nodes: HashSet::new(),
+            non_cbf_nodes: HashSet::new(),
+            previously_sent: HashSet::new(),
+        }
+    }
+
+    fn add_address(&mut self, addr: SocketAddr, cbf: bool) {
+        if cbf {
+            self.cbf_nodes.insert(addr);
+        } else {
+            self.non_cbf_nodes.insert(addr);
+        }
+    }
+
+    fn get_new_address(&mut self, cbf: bool) -> Option<SocketAddr> {
+        if cbf {
+            if let Some(new_addresses) = self
+                .cbf_nodes
+                .iter()
+                .filter(|item| !self.previously_sent.contains(item))
+                .collect::<Vec<&SocketAddr>>()
+                .pop()
+            {
+                self.previously_sent.insert(*new_addresses);
+                Some(*new_addresses)
+            } else {
+                None
+            }
+        } else if let Some(new_addresses) = self
+            .non_cbf_nodes
+            .iter()
+            .filter(|item| !self.previously_sent.contains(item))
+            .collect::<Vec<&SocketAddr>>()
+            .pop()
+        {
+            self.previously_sent.insert(*new_addresses);
+            Some(*new_addresses)
+        } else {
+            None
+        }
+    }
+
+    fn get_cbf_address_count(&self) -> usize {
+        self.cbf_nodes.len()
+    }
+
+    fn get_non_cbf_address_count(&self) -> usize {
+        self.non_cbf_nodes.len()
+    }
+
+    fn remove_address(&mut self, addrs: &SocketAddr, cbf: bool) {
+        if cbf {
+            self.cbf_nodes.remove(addrs);
+        } else {
+            self.non_cbf_nodes.remove(addrs);
+        }
+    }
+
+    fn get_cbf_buffer(&self) -> usize {
+        self.cbf_nodes
+            .iter()
+            .filter(|item| !self.previously_sent.contains(item))
+            .count()
+    }
+
+    fn get_non_cbf_buffer(&self) -> usize {
+        self.non_cbf_nodes
+            .iter()
+            .filter(|item| !self.previously_sent.contains(item))
+            .count()
+    }
+}
+
+/// Discovery statistics, useful for logging
+#[derive(Clone, Copy)]
+pub struct DiscoveryData {
+    queued: usize,
+    visited: usize,
+    non_cbf_count: usize,
+    cbf_count: usize,
+}
+
+/// Progress trait for discovery statistics logging
+pub trait DiscoveryProgress {
+    /// Update progress
+    fn update(&self, data: DiscoveryData);
+}
+
+/// Used when progress updates are not desired
+#[derive(Clone)]
+pub struct NoDiscoveryProgress;
+
+impl DiscoveryProgress for NoDiscoveryProgress {
+    fn update(&self, _data: DiscoveryData) {}
+}
+
+/// Used to log progress update
+#[derive(Clone)]
+pub struct LogDiscoveryProgress;
+
+impl DiscoveryProgress for LogDiscoveryProgress {
+    fn update(&self, data: DiscoveryData) {
+        log::trace!(
+            "P2P Discovery: {} queued, {} visited, {} connected, {} cbf_enabled",
+            data.queued,
+            data.visited,
+            data.non_cbf_count,
+            data.cbf_count
+        );
+
+        #[cfg(test)]
+        println!(
+            "P2P Discovery: {} queued, {} visited, {} connected, {} cbf_enabled",
+            data.queued, data.visited, data.non_cbf_count, data.cbf_count
+        );
+    }
+}
+
+/// A manager structure managing address discovery
+///
+/// Manager will try to maintain a given address buffer in its directory
+/// buffer = len(exiting addresses) - len(previously provided addresses)
+/// Manager will crawl the network until buffer criteria is satisfied
+/// Manager will bootstrap workers from a cache, to speed up discovery progress in
+/// subsequent call after the first crawl.
+/// Manager will keep track of the cache and only update it if previously
+/// unknown addresses are found.
+pub struct AddressManager<P: DiscoveryProgress> {
+    directory: AddressDirectory,
+    cache_filename: String,
+    discovery: Arc<RwLock<AddressDiscovery>>,
+    threads: usize,
+    receiver: Receiver<(SocketAddr, ServiceFlags)>,
+    sender: Sender<(SocketAddr, ServiceFlags)>,
+    network: Network,
+    cbf_buffer: usize,
+    non_cbf_buffer: usize,
+    progress: P,
+}
+
+impl<P: DiscoveryProgress> AddressManager<P> {
+    /// Create a new manager. Initiate Discovery seeds from the cache
+    /// if it exists, else start with hardcoded seeds
+    pub fn new(
+        network: Network,
+        cache_filename: String,
+        threads: usize,
+        cbf_buffer: Option<usize>,
+        non_cbf_buffer: Option<usize>,
+        progress: P,
+    ) -> Result<AddressManager<P>, AddressManagerError> {
+        let (sender, receiver) = channel();
+
+        let seeds = match AddressCache::from_file(&cache_filename)? {
+            Some(cache) => cache.make_seeds(),
+            None => VecDeque::new(),
+        };
+
+        let min_cbf = cbf_buffer.unwrap_or(MIN_CBF_BUFFER);
+
+        let min_non_cbf = non_cbf_buffer.unwrap_or(MIN_NONCBF_BUFFER);
+
+        let discovery = AddressDiscovery::new(network, seeds);
+
+        Ok(AddressManager {
+            cache_filename,
+            directory: AddressDirectory::new(),
+            discovery: Arc::new(RwLock::new(discovery)),
+            sender,
+            receiver,
+            network,
+            threads,
+            cbf_buffer: min_cbf,
+            non_cbf_buffer: min_non_cbf,
+            progress,
+        })
+    }
+
+    /// Get running address discovery progress
+    fn get_progress(&self) -> Result<DiscoveryData, AddressManagerError> {
+        let (queued_count, visited_count) = {
+            let address_discovery = self.discovery.read()?;
+            (
+                address_discovery.pending.len(),
+                address_discovery.visited.len(),
+            )
+        };
+
+        let cbf_node_count = self.directory.get_cbf_address_count();
+        let other_node_count = self.directory.get_non_cbf_address_count();
+
+        Ok(DiscoveryData {
+            queued: queued_count,
+            visited: visited_count,
+            non_cbf_count: cbf_node_count + other_node_count,
+            cbf_count: cbf_node_count,
+        })
+    }
+
+    /// Spawn [self.thread] no. of worker threads
+    fn spawn_workers(&mut self) -> Vec<JoinHandle<()>> {
+        let mut worker_handles: Vec<JoinHandle<()>> = vec![];
+        for _ in 0..self.threads {
+            let sender = self.sender.clone();
+            let discovery = self.discovery.clone();
+            let network = self.network;
+            let worker_handle = thread::spawn(move || {
+                let mut worker = AddressWorker::new(discovery, sender, network);
+                worker.work().unwrap();
+            });
+            worker_handles.push(worker_handle);
+        }
+        worker_handles
+    }
+
+    /// Crawl the Bitcoin network until required number of cbf/non_cbf nodes are found
+    ///
+    /// - This will start a bunch of crawlers.
+    /// - load up the existing cache.
+    /// - Update the cache with new found peers.
+    /// - check if address is in banlist
+    /// - run crawlers until buffer requirement is matched
+    /// - flush the current cache into disk
+    pub fn fetch(&mut self) -> Result<(), AddressManagerError> {
+        self.spawn_workers();
+
+        // Get already existing cache
+        let mut cache = match AddressCache::from_file(&self.cache_filename)? {
+            Some(cache) => cache,
+            None => AddressCache::empty(),
+        };
+
+        while self.directory.get_cbf_buffer() < self.cbf_buffer
+            || self.directory.get_non_cbf_buffer() < self.non_cbf_buffer
+        {
+            if let Ok(message) = self.receiver.recv() {
+                let (addr, flag) = message;
+                if !cache.banned_peers.contains(&addr) {
+                    let cbf = flag.has(ServiceFlags::COMPACT_FILTERS);
+                    self.directory.add_address(addr, cbf);
+                    cache.add_address(addr, cbf);
+                }
+            }
+        }
+
+        self.progress.update(self.get_progress()?);
+
+        // When completed, flush the cache
+        cache.write_to_file(&self.cache_filename)?;
+
+        Ok(())
+    }
+
+    /// Get a new addresses not previously provided
+    pub fn get_new_cbf_address(&mut self) -> Option<SocketAddr> {
+        self.directory.get_new_address(true)
+    }
+
+    /// Get a new non_cbf address
+    pub fn get_new_non_cbf_address(&mut self) -> Option<SocketAddr> {
+        self.directory.get_new_address(false)
+    }
+
+    /// Ban an address
+    pub fn ban_peer(&mut self, addrs: &SocketAddr, cbf: bool) -> Result<(), AddressManagerError> {
+        let mut cache = AddressCache::from_file(&self.cache_filename)?.ok_or_else(|| {
+            AddressManagerError::Generic("Address Cache file not found".to_string())
+        })?;
+
+        cache.add_to_banlist(*addrs, cbf);
+
+        // When completed, flush the cache
+        cache.write_to_file(&self.cache_filename).unwrap();
+
+        self.directory.remove_address(addrs, cbf);
+
+        Ok(())
+    }
+
+    /// Get all the known CBF addresses
+    pub fn get_known_cbfs(&self) -> Option<Vec<SocketAddr>> {
+        let addresses = self
+            .directory
+            .cbf_nodes
+            .iter()
+            .copied()
+            .collect::<Vec<SocketAddr>>();
+
+        match addresses.len() {
+            0 => None,
+            _ => Some(addresses),
+        }
+    }
+
+    /// Get all the known regular addresses
+    pub fn get_known_non_cbfs(&self) -> Option<Vec<SocketAddr>> {
+        let addresses = self
+            .directory
+            .non_cbf_nodes
+            .iter()
+            .copied()
+            .collect::<Vec<SocketAddr>>();
+
+        match addresses.len() {
+            0 => None,
+            _ => Some(addresses),
+        }
+    }
+
+    /// Get previously tried addresses
+    pub fn get_previously_tried(&self) -> Option<Vec<SocketAddr>> {
+        let addresses = self
+            .directory
+            .previously_sent
+            .iter()
+            .copied()
+            .collect::<Vec<SocketAddr>>();
+
+        match addresses.len() {
+            0 => None,
+            _ => Some(addresses),
+        }
+    }
+}
+
+#[derive(Debug)]
+pub enum AddressManagerError {
+    /// Std I/O Error
+    Io(std::io::Error),
+
+    /// Internal Peer error
+    Peer(PeerError),
+
+    /// Internal Mutex poisoning error
+    MutexPoisoned,
+
+    /// Internal Mutex wait timed out
+    MutexTimedOut,
+
+    /// Internal RW read lock poisoned
+    RwReadLockPoisined,
+
+    /// Internal RW write lock poisoned
+    RwWriteLockPoisoned,
+
+    /// Internal MPSC sending error
+    MpscSendError,
+
+    /// Serde Json Error
+    SerdeJson(SerdeError),
+
+    /// Generic Errors
+    Generic(String),
+}
+
+impl_error!(PeerError, Peer, AddressManagerError);
+impl_error!(std::io::Error, Io, AddressManagerError);
+impl_error!(SerdeError, SerdeJson, AddressManagerError);
+
+impl<T> From<PoisonError<MutexGuard<'_, T>>> for AddressManagerError {
+    fn from(_: PoisonError<MutexGuard<'_, T>>) -> Self {
+        AddressManagerError::MutexPoisoned
+    }
+}
+
+impl<T> From<PoisonError<RwLockWriteGuard<'_, T>>> for AddressManagerError {
+    fn from(_: PoisonError<RwLockWriteGuard<'_, T>>) -> Self {
+        AddressManagerError::RwWriteLockPoisoned
+    }
+}
+
+impl<T> From<PoisonError<RwLockReadGuard<'_, T>>> for AddressManagerError {
+    fn from(_: PoisonError<RwLockReadGuard<'_, T>>) -> Self {
+        AddressManagerError::RwReadLockPoisined
+    }
+}
+
+impl<T> From<PoisonError<(MutexGuard<'_, T>, WaitTimeoutResult)>> for AddressManagerError {
+    fn from(err: PoisonError<(MutexGuard<'_, T>, WaitTimeoutResult)>) -> Self {
+        let (_, wait_result) = err.into_inner();
+        if wait_result.timed_out() {
+            AddressManagerError::MutexTimedOut
+        } else {
+            AddressManagerError::MutexPoisoned
+        }
+    }
+}
+
+impl<T> From<SendError<T>> for AddressManagerError {
+    fn from(_: SendError<T>) -> Self {
+        AddressManagerError::MpscSendError
+    }
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+
+    #[test]
+    #[ignore]
+    fn test_address_manager() {
+        // Initiate a manager with an non existent cache file name.
+        // It will create a new cache file
+        let mut manager = AddressManager::new(
+            Network::Bitcoin,
+            "addr_cache".to_string(),
+            20,
+            None,
+            None,
+            LogDiscoveryProgress,
+        )
+        .unwrap();
+
+        // start the crawlers and time them
+        println!("Starting manager and initial fetch");
+        let start = std::time::Instant::now();
+        manager.fetch().unwrap();
+        let duration1 = start.elapsed();
+        println!("Completed Initial fetch");
+
+        // Create a new manager from existing cache and fetch again
+        let mut manager = AddressManager::new(
+            Network::Bitcoin,
+            "addr_cache".to_string(),
+            20,
+            None,
+            None,
+            LogDiscoveryProgress,
+        )
+        .unwrap();
+
+        // start the crawlers and time them
+        println!("Starting new fetch with previous cache");
+        let start = std::time::Instant::now();
+        manager.fetch().unwrap();
+        let duration2 = start.elapsed();
+        println!("Completed new fetch()");
+
+        println!("Time taken for initial crawl: {:#?}", duration1);
+        println!("Time taken for next crawl {:#?}", duration2);
+
+        // Check Buffer Management
+
+        println!("Checking buffer management");
+        // Fetch few new address and ensure buffer goes to zero
+        let mut addrs_list = Vec::new();
+        for _ in 0..5 {
+            let addr_cbf = manager.get_new_cbf_address().unwrap();
+            let addrs_non_cbf = manager.get_new_non_cbf_address().unwrap();
+
+            addrs_list.push(addr_cbf);
+
+            addrs_list.push(addrs_non_cbf);
+        }
+
+        assert_eq!(addrs_list.len(), 10);
+
+        // This should exhaust the cbf buffer
+        assert_eq!(manager.directory.get_cbf_buffer(), 0);
+
+        // Calling fetch again should start crawlers until buffer
+        // requirements are matched.
+        println!("Address buffer exhausted, starting new fetch");
+        manager.fetch().unwrap();
+        println!("Fetch Complete");
+        // It should again have a cbf buffer of 5
+        assert_eq!(manager.directory.get_cbf_buffer(), 5);
+
+        println!("Buffer management passed");
+    }
+}

src/blockchain/compact_filters/mod.rs

@@ -63,22 +63,27 @@ use bitcoin::{Network, OutPoint, Transaction, Txid};
 
 use rocksdb::{Options, SliceTransform, DB};
 
+mod address_manager;
 mod peer;
+mod peermngr;
 mod store;
 mod sync;
 
-use crate::blockchain::*;
+use super::{Blockchain, Capability, ConfigurableBlockchain, Progress};
 use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
 use crate::error::Error;
 use crate::types::{KeychainKind, LocalUtxo, TransactionDetails};
-use crate::{BlockTime, FeeRate};
+use crate::{ConfirmationTime, FeeRate};
 
 use peer::*;
 use store::*;
 use sync::*;
 
+// Only added to avoid unused warnings in addrsmngr module
+pub use address_manager::{
+    AddressManager, DiscoveryProgress, LogDiscoveryProgress, NoDiscoveryProgress,
+};
 pub use peer::{Mempool, Peer};
-
 const SYNC_HEADERS_COST: f32 = 1.0;
 const SYNC_FILTERS_COST: f32 = 11.6 * 1_000.0;
 const PROCESS_BLOCKS_COST: f32 = 20_000.0;
@@ -206,7 +211,8 @@ impl CompactFiltersBlockchain {
                 transaction: Some(tx.clone()),
                 received: incoming,
                 sent: outgoing,
-                confirmation_time: BlockTime::new(height, timestamp),
+                confirmation_time: ConfirmationTime::new(height, timestamp),
+                verified: height.is_some(),
                 fee: Some(inputs_sum.saturating_sub(outputs_sum)),
             };
 
@@ -225,38 +231,11 @@ impl Blockchain for CompactFiltersBlockchain {
         vec![Capability::FullHistory].into_iter().collect()
     }
 
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        self.peers[0].broadcast_tx(tx.clone())?;
-
-        Ok(())
-    }
-
-    fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
-        // TODO
-        Ok(FeeRate::default())
-    }
-}
-
-impl GetHeight for CompactFiltersBlockchain {
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(self.headers.get_height()? as u32)
-    }
-}
-
-impl GetTx for CompactFiltersBlockchain {
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.peers[0]
-            .get_mempool()
-            .get_tx(&Inventory::Transaction(*txid)))
-    }
-}
-
-impl WalletSync for CompactFiltersBlockchain {
     #[allow(clippy::mutex_atomic)] // Mutex is easier to understand than a CAS loop.
-    fn wallet_setup<D: BatchDatabase>(
+    fn setup<D: BatchDatabase, P: 'static + Progress>(
         &self,
         database: &mut D,
-        progress_update: Box<dyn Progress>,
+        progress_update: P,
     ) -> Result<(), Error> {
         let first_peer = &self.peers[0];
 
@@ -280,7 +259,7 @@ impl WalletSync for CompactFiltersBlockchain {
         let total_cost = headers_cost + filters_cost + PROCESS_BLOCKS_COST;
 
         if let Some(snapshot) = sync::sync_headers(
-            Arc::clone(first_peer),
+            Arc::clone(&first_peer),
             Arc::clone(&self.headers),
             |new_height| {
                 let local_headers_cost =
@@ -301,7 +280,7 @@ impl WalletSync for CompactFiltersBlockchain {
         let buried_height = synced_height.saturating_sub(sync::BURIED_CONFIRMATIONS);
         info!("Synced headers to height: {}", synced_height);
 
-        cf_sync.prepare_sync(Arc::clone(first_peer))?;
+        cf_sync.prepare_sync(Arc::clone(&first_peer))?;
 
         let all_scripts = Arc::new(
             database
@@ -320,7 +299,7 @@ impl WalletSync for CompactFiltersBlockchain {
         let mut threads = Vec::with_capacity(self.peers.len());
         for peer in &self.peers {
             let cf_sync = Arc::clone(&cf_sync);
-            let peer = Arc::clone(peer);
+            let peer = Arc::clone(&peer);
             let headers = Arc::clone(&self.headers);
             let all_scripts = Arc::clone(&all_scripts);
             let last_synced_block = Arc::clone(&last_synced_block);
@@ -397,10 +376,10 @@ impl WalletSync for CompactFiltersBlockchain {
         database.commit_batch(updates)?;
 
         match first_peer.ask_for_mempool() {
-            Err(CompactFiltersError::PeerBloomDisabled) => {
+            Err(PeerError::PeerBloomDisabled(_)) => {
                 log::warn!("Peer has BLOOM disabled, we can't ask for the mempool")
             }
-            e => e?,
+            e => e.map_err(CompactFiltersError::from)?,
         };
 
         let mut internal_max_deriv = None;
@@ -418,7 +397,12 @@ impl WalletSync for CompactFiltersBlockchain {
                 )?;
             }
         }
-        for tx in first_peer.get_mempool().iter_txs().iter() {
+        for tx in first_peer
+            .get_mempool()
+            .iter_txs()
+            .map_err(CompactFiltersError::from)?
+            .iter()
+        {
             self.process_tx(
                 database,
                 tx,
@@ -457,6 +441,30 @@ impl WalletSync for CompactFiltersBlockchain {
 
         Ok(())
     }
+
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.peers[0]
+            .get_mempool()
+            .get_tx(&Inventory::Transaction(*txid))
+            .map_err(CompactFiltersError::from)?)
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        self.peers[0]
+            .broadcast_tx(tx.clone())
+            .map_err(CompactFiltersError::from)?;
+
+        Ok(())
+    }
+
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.headers.get_height()? as u32)
+    }
+
+    fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
+        // TODO
+        Ok(FeeRate::default())
+    }
 }
 
 /// Data to connect to a Bitcoin P2P peer
@@ -477,7 +485,7 @@ pub struct CompactFiltersBlockchainConfig {
     pub peers: Vec<BitcoinPeerConfig>,
     /// Network used
     pub network: Network,
-    /// Storage dir to save partially downloaded headers and full blocks. Should be a separate directory per descriptor. Consider using [crate::wallet::wallet_name_from_descriptor] for this.
+    /// Storage dir to save partially downloaded headers and full blocks
     pub storage_dir: String,
     /// Optionally skip initial `skip_blocks` blocks (default: 0)
     pub skip_blocks: Option<usize>,
@@ -492,7 +500,8 @@ impl ConfigurableBlockchain for CompactFiltersBlockchain {
             .peers
             .iter()
             .map(|peer_conf| match &peer_conf.socks5 {
-                None => Peer::connect(&peer_conf.address, Arc::clone(&mempool), config.network),
+                None => Peer::connect(&peer_conf.address, Arc::clone(&mempool), config.network)
+                    .map_err(CompactFiltersError::from),
                 Some(proxy) => Peer::connect_proxy(
                     peer_conf.address.as_str(),
                     proxy,
@@ -502,7 +511,8 @@ impl ConfigurableBlockchain for CompactFiltersBlockchain {
                         .map(|(a, b)| (a.as_str(), b.as_str())),
                     Arc::clone(&mempool),
                     config.network,
-                ),
+                )
+                .map_err(CompactFiltersError::from),
             })
             .collect::<Result<_, _>>()?;
 
@@ -551,6 +561,9 @@ pub enum CompactFiltersError {
 
     /// Wrapper for [`crate::error::Error`]
     Global(Box<crate::error::Error>),
+
+    /// Internal Peer Error
+    Peer(PeerError),
 }
 
 impl fmt::Display for CompactFiltersError {
@@ -565,6 +578,7 @@ impl_error!(rocksdb::Error, Db, CompactFiltersError);
 impl_error!(std::io::Error, Io, CompactFiltersError);
 impl_error!(bitcoin::util::bip158::Error, Bip158, CompactFiltersError);
 impl_error!(std::time::SystemTimeError, Time, CompactFiltersError);
+impl_error!(PeerError, Peer, CompactFiltersError);
 
 impl From<crate::error::Error> for CompactFiltersError {
     fn from(err: crate::error::Error) -> Self {

src/blockchain/compact_filters/peer.rs

@@ -10,11 +10,15 @@
 // licenses.
 
 use std::collections::HashMap;
-use std::net::{TcpStream, ToSocketAddrs};
+use std::fmt;
+use std::net::{SocketAddr, TcpStream, ToSocketAddrs};
 use std::sync::{Arc, Condvar, Mutex, RwLock};
 use std::thread;
 use std::time::{Duration, SystemTime, UNIX_EPOCH};
 
+use std::sync::PoisonError;
+use std::sync::{MutexGuard, RwLockReadGuard, RwLockWriteGuard, WaitTimeoutResult};
+
 use socks::{Socks5Stream, ToTargetAddr};
 
 use rand::{thread_rng, Rng};
@@ -30,8 +34,6 @@ use bitcoin::network::stream_reader::StreamReader;
 use bitcoin::network::Address;
 use bitcoin::{Block, Network, Transaction, Txid, Wtxid};
 
-use super::CompactFiltersError;
-
 type ResponsesMap = HashMap<&'static str, Arc<(Mutex<Vec<NetworkMessage>>, Condvar)>>;
 
 pub(crate) const TIMEOUT_SECS: u64 = 30;
@@ -65,17 +67,18 @@ impl Mempool {
     ///
     /// Note that this doesn't propagate the transaction to other
     /// peers. To do that, [`broadcast`](crate::blockchain::Blockchain::broadcast) should be used.
-    pub fn add_tx(&self, tx: Transaction) {
-        let mut guard = self.0.write().unwrap();
+    pub fn add_tx(&self, tx: Transaction) -> Result<(), PeerError> {
+        let mut guard = self.0.write()?;
 
         guard.wtxids.insert(tx.wtxid(), tx.txid());
         guard.txs.insert(tx.txid(), tx);
+        Ok(())
     }
 
     /// Look-up a transaction in the mempool given an [`Inventory`] request
-    pub fn get_tx(&self, inventory: &Inventory) -> Option<Transaction> {
+    pub fn get_tx(&self, inventory: &Inventory) -> Result<Option<Transaction>, PeerError> {
         let identifer = match inventory {
-            Inventory::Error | Inventory::Block(_) | Inventory::WitnessBlock(_) => return None,
+            Inventory::Error | Inventory::Block(_) | Inventory::WitnessBlock(_) => return Ok(None),
             Inventory::Transaction(txid) => TxIdentifier::Txid(*txid),
             Inventory::WitnessTransaction(txid) => TxIdentifier::Txid(*txid),
             Inventory::WTx(wtxid) => TxIdentifier::Wtxid(*wtxid),
@@ -85,27 +88,34 @@ impl Mempool {
                     inv_type,
                     hash
                 );
-                return None;
+                return Ok(None);
             }
         };
 
         let txid = match identifer {
             TxIdentifier::Txid(txid) => Some(txid),
-            TxIdentifier::Wtxid(wtxid) => self.0.read().unwrap().wtxids.get(&wtxid).cloned(),
+            TxIdentifier::Wtxid(wtxid) => self.0.read()?.wtxids.get(&wtxid).cloned(),
         };
 
-        txid.map(|txid| self.0.read().unwrap().txs.get(&txid).cloned())
-            .flatten()
+        let result = match txid {
+            Some(txid) => {
+                let read_lock = self.0.read()?;
+                read_lock.txs.get(&txid).cloned()
+            }
+            None => None,
+        };
+
+        Ok(result)
     }
 
     /// Return whether or not the mempool contains a transaction with a given txid
-    pub fn has_tx(&self, txid: &Txid) -> bool {
-        self.0.read().unwrap().txs.contains_key(txid)
+    pub fn has_tx(&self, txid: &Txid) -> Result<bool, PeerError> {
+        Ok(self.0.read()?.txs.contains_key(txid))
     }
 
     /// Return the list of transactions contained in the mempool
-    pub fn iter_txs(&self) -> Vec<Transaction> {
-        self.0.read().unwrap().txs.values().cloned().collect()
+    pub fn iter_txs(&self) -> Result<Vec<Transaction>, PeerError> {
+        Ok(self.0.read()?.txs.values().cloned().collect())
     }
 }
 
@@ -133,12 +143,31 @@ impl Peer {
         address: A,
         mempool: Arc<Mempool>,
         network: Network,
-    ) -> Result<Self, CompactFiltersError> {
+    ) -> Result<Self, PeerError> {
         let stream = TcpStream::connect(address)?;
 
         Peer::from_stream(stream, mempool, network)
     }
 
+    /// Connect to a peer over a plaintext TCP connection with a timeout
+    ///
+    /// This function behaves exactly the same as `connect` except for two differences
+    /// 1) It assumes your ToSocketAddrs will resolve to a single address
+    /// 2) It lets you specify a connection timeout
+    pub fn connect_with_timeout<A: ToSocketAddrs>(
+        address: A,
+        timeout: Duration,
+        mempool: Arc<Mempool>,
+        network: Network,
+    ) -> Result<Self, PeerError> {
+        let socket_addr = address
+            .to_socket_addrs()?
+            .next()
+            .ok_or(PeerError::AddresseResolution)?;
+        let stream = TcpStream::connect_timeout(&socket_addr, timeout)?;
+        Peer::from_stream(stream, mempool, network)
+    }
+
     /// Connect to a peer through a SOCKS5 proxy, optionally by using some credentials, specified
     /// as a tuple of `(username, password)`
     ///
@@ -150,7 +179,7 @@ impl Peer {
         credentials: Option<(&str, &str)>,
         mempool: Arc<Mempool>,
         network: Network,
-    ) -> Result<Self, CompactFiltersError> {
+    ) -> Result<Self, PeerError> {
         let socks_stream = if let Some((username, password)) = credentials {
             Socks5Stream::connect_with_password(proxy, target, username, password)?
         } else {
@@ -165,12 +194,12 @@ impl Peer {
         stream: TcpStream,
         mempool: Arc<Mempool>,
         network: Network,
-    ) -> Result<Self, CompactFiltersError> {
+    ) -> Result<Self, PeerError> {
         let writer = Arc::new(Mutex::new(stream.try_clone()?));
         let responses: Arc<RwLock<ResponsesMap>> = Arc::new(RwLock::new(HashMap::new()));
         let connected = Arc::new(RwLock::new(true));
 
-        let mut locked_writer = writer.lock().unwrap();
+        let mut locked_writer = writer.lock()?;
 
         let reader_thread_responses = Arc::clone(&responses);
         let reader_thread_writer = Arc::clone(&writer);
@@ -185,6 +214,7 @@ impl Peer {
                 reader_thread_mempool,
                 reader_thread_connected,
             )
+            .unwrap()
         });
 
         let timestamp = SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs() as i64;
@@ -209,18 +239,20 @@ impl Peer {
                 0,
             )),
         )?;
-        let version = if let NetworkMessage::Version(version) =
-            Self::_recv(&responses, "version", None).unwrap()
-        {
-            version
-        } else {
-            return Err(CompactFiltersError::InvalidResponse);
+
+        let version = match Self::_recv(&responses, "version", Some(Duration::from_secs(1)))? {
+            Some(NetworkMessage::Version(version)) => version,
+            _ => {
+                return Err(PeerError::InvalidResponse(locked_writer.peer_addr()?));
+            }
         };
 
-        if let NetworkMessage::Verack = Self::_recv(&responses, "verack", None).unwrap() {
+        if let Some(NetworkMessage::Verack) =
+            Self::_recv(&responses, "verack", Some(Duration::from_secs(1)))?
+        {
             Self::_send(&mut locked_writer, network.magic(), NetworkMessage::Verack)?;
         } else {
-            return Err(CompactFiltersError::InvalidResponse);
+            return Err(PeerError::InvalidResponse(locked_writer.peer_addr()?));
         }
 
         std::mem::drop(locked_writer);
@@ -236,19 +268,26 @@ impl Peer {
         })
     }
 
+    /// Close the peer connection
+    // Consume Self
+    pub fn close(self) -> Result<(), PeerError> {
+        let locked_writer = self.writer.lock()?;
+        Ok((*locked_writer).shutdown(std::net::Shutdown::Both)?)
+    }
+
+    /// Get the socket address of the remote peer
+    pub fn get_address(&self) -> Result<SocketAddr, PeerError> {
+        let locked_writer = self.writer.lock()?;
+        Ok(locked_writer.peer_addr()?)
+    }
+
     /// Send a Bitcoin network message
-    fn _send(
-        writer: &mut TcpStream,
-        magic: u32,
-        payload: NetworkMessage,
-    ) -> Result<(), CompactFiltersError> {
+    fn _send(writer: &mut TcpStream, magic: u32, payload: NetworkMessage) -> Result<(), PeerError> {
         log::trace!("==> {:?}", payload);
 
         let raw_message = RawNetworkMessage { magic, payload };
 
-        raw_message
-            .consensus_encode(writer)
-            .map_err(|_| CompactFiltersError::DataCorruption)?;
+        raw_message.consensus_encode(writer)?;
 
         Ok(())
     }
@@ -258,30 +297,30 @@ impl Peer {
         responses: &Arc<RwLock<ResponsesMap>>,
         wait_for: &'static str,
         timeout: Option<Duration>,
-    ) -> Option<NetworkMessage> {
+    ) -> Result<Option<NetworkMessage>, PeerError> {
         let message_resp = {
-            let mut lock = responses.write().unwrap();
+            let mut lock = responses.write()?;
             let message_resp = lock.entry(wait_for).or_default();
-            Arc::clone(message_resp)
+            Arc::clone(&message_resp)
         };
 
         let (lock, cvar) = &*message_resp;
 
-        let mut messages = lock.lock().unwrap();
+        let mut messages = lock.lock()?;
         while messages.is_empty() {
             match timeout {
-                None => messages = cvar.wait(messages).unwrap(),
+                None => messages = cvar.wait(messages)?,
                 Some(t) => {
-                    let result = cvar.wait_timeout(messages, t).unwrap();
+                    let result = cvar.wait_timeout(messages, t)?;
                     if result.1.timed_out() {
-                        return None;
+                        return Ok(None);
                     }
                     messages = result.0;
                 }
             }
         }
 
-        messages.pop()
+        Ok(messages.pop())
     }
 
     /// Return the [`VersionMessage`] sent by the peer
@@ -300,8 +339,8 @@ impl Peer {
     }
 
     /// Return whether or not the peer is still connected
-    pub fn is_connected(&self) -> bool {
-        *self.connected.read().unwrap()
+    pub fn is_connected(&self) -> Result<bool, PeerError> {
+        Ok(*self.connected.read()?)
     }
 
     /// Internal function called once the `reader_thread` is spawned
@@ -312,14 +351,14 @@ impl Peer {
         reader_thread_writer: Arc<Mutex<TcpStream>>,
         reader_thread_mempool: Arc<Mempool>,
         reader_thread_connected: Arc<RwLock<bool>>,
-    ) {
+    ) -> Result<(), PeerError> {
         macro_rules! check_disconnect {
             ($call:expr) => {
                 match $call {
                     Ok(good) => good,
                     Err(e) => {
                         log::debug!("Error {:?}", e);
-                        *reader_thread_connected.write().unwrap() = false;
+                        *reader_thread_connected.write()? = false;
 
                         break;
                     }
@@ -328,7 +367,7 @@ impl Peer {
         }
 
         let mut reader = StreamReader::new(connection, None);
-        loop {
+        while *reader_thread_connected.read()? {
             let raw_message: RawNetworkMessage = check_disconnect!(reader.read_next());
 
             let in_message = if raw_message.magic != network.magic() {
@@ -342,7 +381,7 @@ impl Peer {
             match in_message {
                 NetworkMessage::Ping(nonce) => {
                     check_disconnect!(Self::_send(
-                        &mut reader_thread_writer.lock().unwrap(),
+                        &mut *reader_thread_writer.lock()?,
                         network.magic(),
                         NetworkMessage::Pong(nonce),
                     ));
@@ -353,19 +392,21 @@ impl Peer {
                 NetworkMessage::GetData(ref inv) => {
                     let (found, not_found): (Vec<_>, Vec<_>) = inv
                         .iter()
-                        .map(|item| (*item, reader_thread_mempool.get_tx(item)))
+                        .map(|item| (*item, reader_thread_mempool.get_tx(item).unwrap()))
                         .partition(|(_, d)| d.is_some());
                     for (_, found_tx) in found {
                         check_disconnect!(Self::_send(
-                            &mut reader_thread_writer.lock().unwrap(),
+                            &mut *reader_thread_writer.lock()?,
                             network.magic(),
-                            NetworkMessage::Tx(found_tx.unwrap()),
+                            NetworkMessage::Tx(found_tx.ok_or_else(|| PeerError::Generic(
+                                "Got None while expecting Transaction".to_string()
+                            ))?),
                         ));
                     }
 
                     if !not_found.is_empty() {
                         check_disconnect!(Self::_send(
-                            &mut reader_thread_writer.lock().unwrap(),
+                            &mut *reader_thread_writer.lock()?,
                             network.magic(),
                             NetworkMessage::NotFound(
                                 not_found.into_iter().map(|(i, _)| i).collect(),
@@ -377,21 +418,23 @@ impl Peer {
             }
 
             let message_resp = {
-                let mut lock = reader_thread_responses.write().unwrap();
+                let mut lock = reader_thread_responses.write()?;
                 let message_resp = lock.entry(in_message.cmd()).or_default();
-                Arc::clone(message_resp)
+                Arc::clone(&message_resp)
             };
 
             let (lock, cvar) = &*message_resp;
-            let mut messages = lock.lock().unwrap();
+            let mut messages = lock.lock()?;
             messages.push(in_message);
             cvar.notify_all();
         }
+
+        Ok(())
     }
 
     /// Send a raw Bitcoin message to the peer
-    pub fn send(&self, payload: NetworkMessage) -> Result<(), CompactFiltersError> {
-        let mut writer = self.writer.lock().unwrap();
+    pub fn send(&self, payload: NetworkMessage) -> Result<(), PeerError> {
+        let mut writer = self.writer.lock()?;
         Self::_send(&mut writer, self.network.magic(), payload)
     }
 
@@ -400,30 +443,27 @@ impl Peer {
         &self,
         wait_for: &'static str,
         timeout: Option<Duration>,
-    ) -> Result<Option<NetworkMessage>, CompactFiltersError> {
-        Ok(Self::_recv(&self.responses, wait_for, timeout))
+    ) -> Result<Option<NetworkMessage>, PeerError> {
+        Self::_recv(&self.responses, wait_for, timeout)
     }
 }
 
 pub trait CompactFiltersPeer {
-    fn get_cf_checkpt(
-        &self,
-        filter_type: u8,
-        stop_hash: BlockHash,
-    ) -> Result<CFCheckpt, CompactFiltersError>;
+    fn get_cf_checkpt(&self, filter_type: u8, stop_hash: BlockHash)
+        -> Result<CFCheckpt, PeerError>;
     fn get_cf_headers(
         &self,
         filter_type: u8,
         start_height: u32,
         stop_hash: BlockHash,
-    ) -> Result<CFHeaders, CompactFiltersError>;
+    ) -> Result<CFHeaders, PeerError>;
     fn get_cf_filters(
         &self,
         filter_type: u8,
         start_height: u32,
         stop_hash: BlockHash,
-    ) -> Result<(), CompactFiltersError>;
-    fn pop_cf_filter_resp(&self) -> Result<CFilter, CompactFiltersError>;
+    ) -> Result<(), PeerError>;
+    fn pop_cf_filter_resp(&self) -> Result<CFilter, PeerError>;
 }
 
 impl CompactFiltersPeer for Peer {
@@ -431,22 +471,20 @@ impl CompactFiltersPeer for Peer {
         &self,
         filter_type: u8,
         stop_hash: BlockHash,
-    ) -> Result<CFCheckpt, CompactFiltersError> {
+    ) -> Result<CFCheckpt, PeerError> {
         self.send(NetworkMessage::GetCFCheckpt(GetCFCheckpt {
             filter_type,
             stop_hash,
         }))?;
 
-        let response = self
-            .recv("cfcheckpt", Some(Duration::from_secs(TIMEOUT_SECS)))?
-            .ok_or(CompactFiltersError::Timeout)?;
+        let response = self.recv("cfcheckpt", Some(Duration::from_secs(TIMEOUT_SECS)))?;
         let response = match response {
-            NetworkMessage::CFCheckpt(response) => response,
-            _ => return Err(CompactFiltersError::InvalidResponse),
+            Some(NetworkMessage::CFCheckpt(response)) => response,
+            _ => return Err(PeerError::InvalidResponse(self.get_address()?)),
         };
 
         if response.filter_type != filter_type {
-            return Err(CompactFiltersError::InvalidResponse);
+            return Err(PeerError::InvalidResponse(self.get_address()?));
         }
 
         Ok(response)
@@ -457,35 +495,31 @@ impl CompactFiltersPeer for Peer {
         filter_type: u8,
         start_height: u32,
         stop_hash: BlockHash,
-    ) -> Result<CFHeaders, CompactFiltersError> {
+    ) -> Result<CFHeaders, PeerError> {
         self.send(NetworkMessage::GetCFHeaders(GetCFHeaders {
             filter_type,
             start_height,
             stop_hash,
         }))?;
 
-        let response = self
-            .recv("cfheaders", Some(Duration::from_secs(TIMEOUT_SECS)))?
-            .ok_or(CompactFiltersError::Timeout)?;
+        let response = self.recv("cfheaders", Some(Duration::from_secs(TIMEOUT_SECS)))?;
         let response = match response {
-            NetworkMessage::CFHeaders(response) => response,
-            _ => return Err(CompactFiltersError::InvalidResponse),
+            Some(NetworkMessage::CFHeaders(response)) => response,
+            _ => return Err(PeerError::InvalidResponse(self.get_address()?)),
         };
 
         if response.filter_type != filter_type {
-            return Err(CompactFiltersError::InvalidResponse);
+            return Err(PeerError::InvalidResponse(self.get_address()?));
         }
 
         Ok(response)
     }
 
-    fn pop_cf_filter_resp(&self) -> Result<CFilter, CompactFiltersError> {
-        let response = self
-            .recv("cfilter", Some(Duration::from_secs(TIMEOUT_SECS)))?
-            .ok_or(CompactFiltersError::Timeout)?;
+    fn pop_cf_filter_resp(&self) -> Result<CFilter, PeerError> {
+        let response = self.recv("cfilter", Some(Duration::from_secs(TIMEOUT_SECS)))?;
         let response = match response {
-            NetworkMessage::CFilter(response) => response,
-            _ => return Err(CompactFiltersError::InvalidResponse),
+            Some(NetworkMessage::CFilter(response)) => response,
+            _ => return Err(PeerError::InvalidResponse(self.get_address()?)),
         };
 
         Ok(response)
@@ -496,7 +530,7 @@ impl CompactFiltersPeer for Peer {
         filter_type: u8,
         start_height: u32,
         stop_hash: BlockHash,
-    ) -> Result<(), CompactFiltersError> {
+    ) -> Result<(), PeerError> {
         self.send(NetworkMessage::GetCFilters(GetCFilters {
             filter_type,
             start_height,
@@ -508,13 +542,13 @@ impl CompactFiltersPeer for Peer {
 }
 
 pub trait InvPeer {
-    fn get_block(&self, block_hash: BlockHash) -> Result<Option<Block>, CompactFiltersError>;
-    fn ask_for_mempool(&self) -> Result<(), CompactFiltersError>;
-    fn broadcast_tx(&self, tx: Transaction) -> Result<(), CompactFiltersError>;
+    fn get_block(&self, block_hash: BlockHash) -> Result<Option<Block>, PeerError>;
+    fn ask_for_mempool(&self) -> Result<(), PeerError>;
+    fn broadcast_tx(&self, tx: Transaction) -> Result<(), PeerError>;
 }
 
 impl InvPeer for Peer {
-    fn get_block(&self, block_hash: BlockHash) -> Result<Option<Block>, CompactFiltersError> {
+    fn get_block(&self, block_hash: BlockHash) -> Result<Option<Block>, PeerError> {
         self.send(NetworkMessage::GetData(vec![Inventory::WitnessBlock(
             block_hash,
         )]))?;
@@ -522,51 +556,126 @@ impl InvPeer for Peer {
         match self.recv("block", Some(Duration::from_secs(TIMEOUT_SECS)))? {
             None => Ok(None),
             Some(NetworkMessage::Block(response)) => Ok(Some(response)),
-            _ => Err(CompactFiltersError::InvalidResponse),
+            _ => Err(PeerError::InvalidResponse(self.get_address()?)),
         }
     }
 
-    fn ask_for_mempool(&self) -> Result<(), CompactFiltersError> {
+    fn ask_for_mempool(&self) -> Result<(), PeerError> {
         if !self.version.services.has(ServiceFlags::BLOOM) {
-            return Err(CompactFiltersError::PeerBloomDisabled);
+            return Err(PeerError::PeerBloomDisabled(self.get_address()?));
         }
 
         self.send(NetworkMessage::MemPool)?;
         let inv = match self.recv("inv", Some(Duration::from_secs(5)))? {
             None => return Ok(()), // empty mempool
             Some(NetworkMessage::Inv(inv)) => inv,
-            _ => return Err(CompactFiltersError::InvalidResponse),
+            _ => return Err(PeerError::InvalidResponse(self.get_address()?)),
         };
 
         let getdata = inv
             .iter()
             .cloned()
             .filter(
-                |item| matches!(item, Inventory::Transaction(txid) if !self.mempool.has_tx(txid)),
+                |item| matches!(item, Inventory::Transaction(txid) if !self.mempool.has_tx(txid).unwrap()),
             )
             .collect::<Vec<_>>();
         let num_txs = getdata.len();
         self.send(NetworkMessage::GetData(getdata))?;
 
         for _ in 0..num_txs {
-            let tx = self
-                .recv("tx", Some(Duration::from_secs(TIMEOUT_SECS)))?
-                .ok_or(CompactFiltersError::Timeout)?;
+            let tx = self.recv("tx", Some(Duration::from_secs(TIMEOUT_SECS)))?;
             let tx = match tx {
-                NetworkMessage::Tx(tx) => tx,
-                _ => return Err(CompactFiltersError::InvalidResponse),
+                Some(NetworkMessage::Tx(tx)) => tx,
+                _ => return Err(PeerError::InvalidResponse(self.get_address()?)),
             };
 
-            self.mempool.add_tx(tx);
+            self.mempool.add_tx(tx)?;
         }
 
         Ok(())
     }
 
-    fn broadcast_tx(&self, tx: Transaction) -> Result<(), CompactFiltersError> {
-        self.mempool.add_tx(tx.clone());
+    fn broadcast_tx(&self, tx: Transaction) -> Result<(), PeerError> {
+        self.mempool.add_tx(tx.clone())?;
         self.send(NetworkMessage::Tx(tx))?;
 
         Ok(())
     }
 }
+
+/// Peer Errors
+#[derive(Debug)]
+pub enum PeerError {
+    /// Internal I/O error
+    Io(std::io::Error),
+
+    /// Internal system time error
+    Time(std::time::SystemTimeError),
+
+    /// A peer sent an invalid or unexpected response
+    InvalidResponse(SocketAddr),
+
+    /// Peer had bloom filter disabled
+    PeerBloomDisabled(SocketAddr),
+
+    /// Internal Mutex poisoning error
+    MutexPoisoned,
+
+    /// Internal Mutex wait timed out
+    MutexTimedout,
+
+    /// Internal RW read lock poisoned
+    RwReadLockPoisined,
+
+    /// Internal RW write lock poisoned
+    RwWriteLockPoisoned,
+
+    /// Mempool Mutex poisoned
+    MempoolPoisoned,
+
+    /// Network address resolution Error
+    AddresseResolution,
+
+    /// Generic Errors
+    Generic(String),
+}
+
+impl std::fmt::Display for PeerError {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(f, "{:?}", self)
+    }
+}
+
+impl std::error::Error for PeerError {}
+
+impl_error!(std::io::Error, Io, PeerError);
+impl_error!(std::time::SystemTimeError, Time, PeerError);
+
+impl<T> From<PoisonError<MutexGuard<'_, T>>> for PeerError {
+    fn from(_: PoisonError<MutexGuard<'_, T>>) -> Self {
+        PeerError::MutexPoisoned
+    }
+}
+
+impl<T> From<PoisonError<RwLockWriteGuard<'_, T>>> for PeerError {
+    fn from(_: PoisonError<RwLockWriteGuard<'_, T>>) -> Self {
+        PeerError::RwWriteLockPoisoned
+    }
+}
+
+impl<T> From<PoisonError<RwLockReadGuard<'_, T>>> for PeerError {
+    fn from(_: PoisonError<RwLockReadGuard<'_, T>>) -> Self {
+        PeerError::RwReadLockPoisined
+    }
+}
+
+impl<T> From<PoisonError<(MutexGuard<'_, T>, WaitTimeoutResult)>> for PeerError {
+    fn from(err: PoisonError<(MutexGuard<'_, T>, WaitTimeoutResult)>) -> Self {
+        let (_, wait_result) = err.into_inner();
+        if wait_result.timed_out() {
+            PeerError::MutexTimedout
+        } else {
+            PeerError::MutexPoisoned
+        }
+    }
+}

src/blockchain/compact_filters/peermngr.rs

@@ -0,0 +1,578 @@
+use super::address_manager::{AddressManager, AddressManagerError, DiscoveryProgress};
+use super::peer::{Mempool, Peer, PeerError, TIMEOUT_SECS};
+
+use std::net::SocketAddr;
+use std::sync::Arc;
+use std::time;
+
+use bitcoin::network::constants::{Network, ServiceFlags};
+use bitcoin::network::message::NetworkMessage;
+
+use std::path::PathBuf;
+
+use std::collections::BTreeMap;
+
+// Peer Manager Configuration constants
+const MIN_CBF_PEERS: usize = 2;
+const MIN_TOTAL_PEERS: usize = 5;
+const MIN_CRAWLER_THREADS: usize = 20;
+const BAN_SCORE_THRESHOLD: usize = 100;
+const RECEIVE_TIMEOUT: time::Duration = time::Duration::from_secs(TIMEOUT_SECS);
+
+#[allow(dead_code)]
+/// An Error structure describing Peer Management errors
+#[derive(Debug)]
+pub enum PeerManagerError {
+    // Internal Peer Error
+    Peer(PeerError),
+
+    // Internal AddressManager Error
+    AddrsManager(AddressManagerError),
+
+    // Os String Error
+    OsString(std::ffi::OsString),
+
+    // Peer not found in directory
+    PeerNotFound,
+
+    // Generic Internal Error
+    Generic(String),
+}
+
+impl_error!(PeerError, Peer, PeerManagerError);
+impl_error!(AddressManagerError, AddrsManager, PeerManagerError);
+impl_error!(std::ffi::OsString, OsString, PeerManagerError);
+
+/// Peer Data stored in the manager's directory
+#[derive(Debug)]
+struct PeerData {
+    peer: Peer,
+    is_cbf: bool,
+    ban_score: usize,
+}
+
+#[allow(dead_code)]
+/// A Directory structure to hold live Peers
+/// All peers in the directory have live ongoing connection
+/// Banning a peer removes it from the directory
+#[derive(Default, Debug)]
+struct PeerDirectory {
+    peers: BTreeMap<SocketAddr, PeerData>,
+}
+
+#[allow(dead_code)]
+impl PeerDirectory {
+    fn new() -> Self {
+        Self::default()
+    }
+
+    fn get_cbf_peers(&self) -> Option<Vec<&PeerData>> {
+        let cbf_peers = self
+            .peers
+            .iter()
+            .filter(|(_, peer)| peer.is_cbf)
+            .map(|(_, peer)| peer)
+            .collect::<Vec<&PeerData>>();
+
+        match cbf_peers.len() {
+            0 => None,
+            _ => Some(cbf_peers),
+        }
+    }
+
+    fn get_cbf_addresses(&self) -> Option<Vec<SocketAddr>> {
+        let cbf_addrseses = self
+            .peers
+            .iter()
+            .filter_map(
+                |(addrs, peerdata)| {
+                    if peerdata.is_cbf {
+                        Some(addrs)
+                    } else {
+                        None
+                    }
+                },
+            )
+            .copied()
+            .collect::<Vec<SocketAddr>>();
+
+        match cbf_addrseses.len() {
+            0 => None,
+            _ => Some(cbf_addrseses),
+        }
+    }
+
+    fn get_non_cbf_peers(&self) -> Option<Vec<&PeerData>> {
+        let non_cbf_peers = self
+            .peers
+            .iter()
+            .filter(|(_, peerdata)| !peerdata.is_cbf)
+            .map(|(_, peerdata)| peerdata)
+            .collect::<Vec<&PeerData>>();
+
+        match non_cbf_peers.len() {
+            0 => None,
+            _ => Some(non_cbf_peers),
+        }
+    }
+
+    fn get_non_cbf_addresses(&self) -> Option<Vec<SocketAddr>> {
+        let addresses = self
+            .peers
+            .iter()
+            .filter_map(
+                |(addrs, peerdata)| {
+                    if !peerdata.is_cbf {
+                        Some(addrs)
+                    } else {
+                        None
+                    }
+                },
+            )
+            .copied()
+            .collect::<Vec<SocketAddr>>();
+
+        match addresses.len() {
+            0 => None,
+            _ => Some(addresses),
+        }
+    }
+
+    fn get_cbf_peers_mut(&mut self) -> Option<Vec<&mut PeerData>> {
+        let peers = self
+            .peers
+            .iter_mut()
+            .filter(|(_, peerdata)| peerdata.is_cbf)
+            .map(|(_, peerdata)| peerdata)
+            .collect::<Vec<&mut PeerData>>();
+
+        match peers.len() {
+            0 => None,
+            _ => Some(peers),
+        }
+    }
+
+    fn get_non_cbf_peers_mut(&mut self) -> Option<Vec<&mut PeerData>> {
+        let peers = self
+            .peers
+            .iter_mut()
+            .filter(|(_, peerdata)| !peerdata.is_cbf)
+            .map(|(_, peerdata)| peerdata)
+            .collect::<Vec<&mut PeerData>>();
+
+        match peers.len() {
+            0 => None,
+            _ => Some(peers),
+        }
+    }
+
+    fn get_cbf_count(&self) -> usize {
+        self.peers
+            .iter()
+            .filter(|(_, peerdata)| peerdata.is_cbf)
+            .count()
+    }
+
+    fn get_non_cbf_count(&self) -> usize {
+        self.peers
+            .iter()
+            .filter(|(_, peerdata)| !peerdata.is_cbf)
+            .count()
+    }
+
+    fn insert_peer(&mut self, peerdata: PeerData) -> Result<(), PeerManagerError> {
+        let addrs = peerdata.peer.get_address()?;
+        self.peers.entry(addrs).or_insert(peerdata);
+        Ok(())
+    }
+
+    fn remove_peer(&mut self, addrs: &SocketAddr) -> Option<PeerData> {
+        self.peers.remove(addrs)
+    }
+
+    fn get_peer_banscore(&self, addrs: &SocketAddr) -> Option<usize> {
+        self.peers.get(addrs).map(|peerdata| peerdata.ban_score)
+    }
+
+    fn get_peerdata_mut(&mut self, address: &SocketAddr) -> Option<&mut PeerData> {
+        self.peers.get_mut(address)
+    }
+
+    fn get_peerdata(&self, address: &SocketAddr) -> Option<&PeerData> {
+        self.peers.get(address)
+    }
+
+    fn is_cbf(&self, addrs: &SocketAddr) -> Option<bool> {
+        if let Some(peer) = self.peers.get(addrs) {
+            match peer.is_cbf {
+                true => Some(true),
+                false => Some(false),
+            }
+        } else {
+            None
+        }
+    }
+}
+
+#[allow(dead_code)]
+pub struct PeerManager<P: DiscoveryProgress> {
+    addrs_mngr: AddressManager<P>,
+    directory: PeerDirectory,
+    mempool: Arc<Mempool>,
+    min_cbf: usize,
+    min_total: usize,
+    network: Network,
+}
+
+#[allow(dead_code)]
+impl<P: DiscoveryProgress> PeerManager<P> {
+    pub fn init(
+        network: Network,
+        cache_dir: &str,
+        crawler_threads: Option<usize>,
+        progress: P,
+        cbf_peers: Option<usize>,
+        total_peers: Option<usize>,
+    ) -> Result<Self, PeerManagerError> {
+        let mut cache_filename = PathBuf::from(cache_dir);
+        cache_filename.push("addr_cache");
+
+        // Fetch minimum peer requirements, either by user input, or via default
+        let min_cbf = cbf_peers.unwrap_or(MIN_CBF_PEERS);
+
+        let min_total = total_peers.unwrap_or(MIN_TOTAL_PEERS);
+
+        let cbf_buff = min_cbf * 2;
+        let non_cbf_buff = (min_total - min_cbf) * 2;
+
+        // Create internal items
+        let addrs_mngr = AddressManager::new(
+            network,
+            cache_filename.into_os_string().into_string()?,
+            crawler_threads.unwrap_or(MIN_CRAWLER_THREADS),
+            Some(cbf_buff),
+            Some(non_cbf_buff),
+            progress,
+        )?;
+
+        let mempool = Arc::new(Mempool::new());
+
+        let peer_dir = PeerDirectory::new();
+
+        // Create self and update
+        let mut manager = Self {
+            addrs_mngr,
+            directory: peer_dir,
+            mempool,
+            min_cbf,
+            min_total,
+            network,
+        };
+
+        manager.update_directory()?;
+
+        Ok(manager)
+    }
+
+    fn update_directory(&mut self) -> Result<(), PeerManagerError> {
+        while self.directory.get_cbf_count() < self.min_cbf
+            || self.directory.get_non_cbf_count() < (self.min_total - self.min_cbf)
+        {
+            // First connect with cbf peers, then with non_cbf
+            let cbf_fetch = self.directory.get_cbf_count() < self.min_cbf;
+
+            // Try to get an address
+            // if not present start crawlers
+            let target_addrs = match cbf_fetch {
+                true => {
+                    if let Some(addrs) = self.addrs_mngr.get_new_cbf_address() {
+                        addrs
+                    } else {
+                        self.addrs_mngr.fetch()?;
+                        continue;
+                    }
+                }
+                false => {
+                    if let Some(addrs) = self.addrs_mngr.get_new_non_cbf_address() {
+                        addrs
+                    } else {
+                        self.addrs_mngr.fetch()?;
+                        continue;
+                    }
+                }
+            };
+
+            if let Ok(peer) = Peer::connect(target_addrs, Arc::clone(&self.mempool), self.network) {
+                let address = peer.get_address()?;
+
+                assert_eq!(address, target_addrs);
+
+                let is_cbf = peer
+                    .get_version()
+                    .services
+                    .has(ServiceFlags::COMPACT_FILTERS);
+
+                let peerdata = PeerData {
+                    peer,
+                    is_cbf,
+                    ban_score: 0,
+                };
+
+                self.directory.insert_peer(peerdata)?;
+            } else {
+                continue;
+            }
+        }
+
+        Ok(())
+    }
+
+    pub fn set_banscore(
+        &mut self,
+        increase_by: usize,
+        address: &SocketAddr,
+    ) -> Result<(), PeerManagerError> {
+        let mut current_score = if let Some(peer) = self.directory.get_peerdata_mut(address) {
+            peer.ban_score
+        } else {
+            return Err(PeerManagerError::PeerNotFound);
+        };
+
+        current_score += increase_by;
+
+        let mut banned = false;
+
+        if current_score >= BAN_SCORE_THRESHOLD {
+            match (
+                self.directory.is_cbf(address),
+                self.directory.remove_peer(address),
+            ) {
+                (Some(true), Some(_)) => {
+                    self.addrs_mngr.ban_peer(address, true)?;
+                    banned = true;
+                }
+                (Some(false), Some(_)) => {
+                    self.addrs_mngr.ban_peer(address, false)?;
+                    banned = true;
+                }
+                _ => {
+                    return Err(PeerManagerError::Generic(
+                        "data inconsistency in directory, should not happen".to_string(),
+                    ))
+                }
+            }
+        }
+
+        if banned {
+            self.update_directory()?;
+        }
+
+        Ok(())
+    }
+
+    pub fn send_to(
+        &self,
+        address: &SocketAddr,
+        message: NetworkMessage,
+    ) -> Result<(), PeerManagerError> {
+        if let Some(peerdata) = self.directory.get_peerdata(address) {
+            peerdata.peer.send(message)?;
+            Ok(())
+        } else {
+            Err(PeerManagerError::PeerNotFound)
+        }
+    }
+
+    pub fn receive_from(
+        &self,
+        address: &SocketAddr,
+        wait_for: &'static str,
+    ) -> Result<Option<NetworkMessage>, PeerManagerError> {
+        if let Some(peerdata) = self.directory.get_peerdata(address) {
+            if let Some(response) = peerdata.peer.recv(wait_for, Some(RECEIVE_TIMEOUT))? {
+                Ok(Some(response))
+            } else {
+                Ok(None)
+            }
+        } else {
+            Err(PeerManagerError::PeerNotFound)
+        }
+    }
+
+    pub fn connected_cbf_addresses(&self) -> Option<Vec<SocketAddr>> {
+        self.directory.get_cbf_addresses()
+    }
+
+    pub fn connected_non_cbf_addresses(&self) -> Option<Vec<SocketAddr>> {
+        self.directory.get_non_cbf_addresses()
+    }
+
+    pub fn known_cbf_addresses(&self) -> Option<Vec<SocketAddr>> {
+        self.addrs_mngr.get_known_cbfs()
+    }
+
+    pub fn known_non_cbf_addresses(&self) -> Option<Vec<SocketAddr>> {
+        self.addrs_mngr.get_known_non_cbfs()
+    }
+
+    pub fn previously_tried_addresses(&self) -> Option<Vec<SocketAddr>> {
+        self.addrs_mngr.get_previously_tried()
+    }
+}
+
+#[cfg(test)]
+mod test {
+    use super::super::LogDiscoveryProgress;
+    use super::*;
+
+    #[test]
+    #[ignore]
+    fn test_ban() {
+        let mut manager = PeerManager::init(
+            Network::Bitcoin,
+            ".",
+            None,
+            LogDiscoveryProgress,
+            None,
+            None,
+        )
+        .unwrap();
+
+        let connected_cbfs = manager.connected_cbf_addresses().unwrap();
+        let connected_non_cbfs = manager.connected_non_cbf_addresses().unwrap();
+
+        println!("Currently Connected CBFs: {:#?}", connected_cbfs);
+        assert_eq!(connected_cbfs.len(), 2);
+        assert_eq!(connected_non_cbfs.len(), 3);
+
+        let to_banned = &connected_cbfs[0];
+
+        println!("Banning address : {}", to_banned);
+
+        manager.set_banscore(100, to_banned).unwrap();
+
+        let newly_connected = manager.connected_cbf_addresses().unwrap();
+
+        println!("Newly Connected CBFs: {:#?}", newly_connected);
+
+        assert_eq!(newly_connected.len(), 2);
+
+        assert_ne!(newly_connected, connected_cbfs);
+    }
+
+    #[test]
+    #[ignore]
+    fn test_send_recv() {
+        let manager = PeerManager::init(
+            Network::Bitcoin,
+            ".",
+            None,
+            LogDiscoveryProgress,
+            None,
+            None,
+        )
+        .unwrap();
+
+        let target_address = manager.connected_cbf_addresses().unwrap()[0];
+
+        let ping = NetworkMessage::Ping(30);
+
+        println!("Asking peer {}", target_address);
+
+        manager.send_to(&target_address, ping).unwrap();
+
+        let response = manager
+            .receive_from(&target_address, "pong")
+            .unwrap()
+            .unwrap();
+
+        let value = match response {
+            NetworkMessage::Pong(v) => Some(v),
+            _ => None,
+        };
+
+        let value = value.unwrap();
+
+        println!("Got value {:#?}", value);
+    }
+
+    #[test]
+    #[ignore]
+    fn test_connect_all() {
+        let manager = PeerManager::init(
+            Network::Bitcoin,
+            ".",
+            None,
+            LogDiscoveryProgress,
+            None,
+            None,
+        )
+        .unwrap();
+
+        let cbf_pings = vec![100u64; manager.min_cbf];
+        let non_cbf_pings = vec![200u64; manager.min_total - manager.min_cbf];
+
+        let cbf_peers = manager.connected_cbf_addresses().unwrap();
+        let non_cbf_peers = manager.connected_non_cbf_addresses().unwrap();
+
+        let sent_cbf: Vec<bool> = cbf_pings
+            .iter()
+            .zip(cbf_peers.iter())
+            .map(|(ping, address)| {
+                let message = NetworkMessage::Ping(*ping);
+                manager.send_to(address, message).unwrap();
+                true
+            })
+            .collect();
+
+        assert_eq!(sent_cbf, vec![true; manager.min_cbf]);
+
+        println!("Sent pings to cbf peers");
+
+        let sent_noncbf: Vec<bool> = non_cbf_pings
+            .iter()
+            .zip(non_cbf_peers.iter())
+            .map(|(ping, address)| {
+                let message = NetworkMessage::Ping(*ping);
+                manager.send_to(address, message).unwrap();
+                true
+            })
+            .collect();
+
+        assert_eq!(sent_noncbf, vec![true; manager.min_total - manager.min_cbf]);
+
+        println!("Sent pings to non cbf peers");
+
+        let cbf_received: Vec<u64> = cbf_peers
+            .iter()
+            .map(|address| {
+                let response = manager.receive_from(address, "pong").unwrap().unwrap();
+
+                let value = match response {
+                    NetworkMessage::Pong(v) => Some(v),
+                    _ => None,
+                };
+
+                value.unwrap()
+            })
+            .collect();
+
+        let non_cbf_received: Vec<u64> = non_cbf_peers
+            .iter()
+            .map(|address| {
+                let response = manager.receive_from(address, "pong").unwrap().unwrap();
+
+                let value = match response {
+                    NetworkMessage::Pong(v) => Some(v),
+                    _ => None,
+                };
+
+                value.unwrap()
+            })
+            .collect();
+
+        assert_eq!(cbf_pings, cbf_received);
+
+        assert_eq!(non_cbf_pings, non_cbf_received);
+    }
+}

src/blockchain/compact_filters/store.rs

@@ -398,7 +398,7 @@ impl ChainStore<Full> {
             );
         }
 
-        // Delete full blocks overridden by snapshot
+        // Delete full blocks overriden by snapshot
         let from_key = StoreEntry::Block(Some(snaphost.min_height)).get_key();
         let to_key = StoreEntry::Block(Some(usize::MAX)).get_key();
         batch.delete_range(&from_key, &to_key);
@@ -760,7 +760,7 @@ impl CfStore {
         let cf_headers: Vec<FilterHeader> = filter_hashes
             .into_iter()
             .scan(checkpoint, |prev_header, filter_hash| {
-                let filter_header = filter_hash.filter_header(prev_header);
+                let filter_header = filter_hash.filter_header(&prev_header);
                 *prev_header = filter_header;
 
                 Some(filter_header)
@@ -801,7 +801,7 @@ impl CfStore {
             .zip(headers.into_iter())
             .scan(checkpoint, |prev_header, ((_, filter_content), header)| {
                 let filter = BlockFilter::new(&filter_content);
-                if header != filter.filter_header(prev_header) {
+                if header != filter.filter_header(&prev_header) {
                     return Some(Err(CompactFiltersError::InvalidFilter));
                 }
                 *prev_header = header;

src/blockchain/compact_filters/sync.rs

@@ -205,7 +205,7 @@ impl CfSync {
                     let block_hash = self.headers_store.get_block_hash(height)?.unwrap();
 
                     // TODO: also download random blocks?
-                    if process(&block_hash, &BlockFilter::new(filter))? {
+                    if process(&block_hash, &BlockFilter::new(&filter))? {
                         log::debug!("Downloading block {}", block_hash);
 
                         let block = peer

src/blockchain/electrum.rs

@@ -24,20 +24,20 @@
 //! # Ok::<(), bdk::Error>(())
 //! ```
 
-use std::collections::{HashMap, HashSet};
+use std::collections::HashSet;
 
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
-use bitcoin::{Transaction, Txid};
+use bitcoin::{BlockHeader, Script, Transaction, Txid};
 
 use electrum_client::{Client, ConfigBuilder, ElectrumApi, Socks5Config};
 
-use super::script_sync::Request;
+use self::utils::{ElectrumLikeSync, ElsGetHistoryRes};
 use super::*;
-use crate::database::{BatchDatabase, Database};
+use crate::database::BatchDatabase;
 use crate::error::Error;
-use crate::{BlockTime, FeeRate};
+use crate::FeeRate;
 
 /// Wrapper over an Electrum Client that implements the required blockchain traits
 ///
@@ -68,10 +68,32 @@ impl Blockchain for ElectrumBlockchain {
         .collect()
     }
 
+    fn setup<D: BatchDatabase, P: Progress>(
+        &self,
+        database: &mut D,
+        progress_update: P,
+    ) -> Result<(), Error> {
+        self.client
+            .electrum_like_setup(self.stop_gap, database, progress_update)
+    }
+
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(self.client.transaction_get(txid).map(Option::Some)?)
+    }
+
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         Ok(self.client.transaction_broadcast(tx).map(|_| ())?)
     }
 
+    fn get_height(&self) -> Result<u32, Error> {
+        // TODO: unsubscribe when added to the client, or is there a better call to use here?
+
+        Ok(self
+            .client
+            .block_headers_subscribe()
+            .map(|data| data.height as u32)?)
+    }
+
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         Ok(FeeRate::from_btc_per_kvb(
             self.client.estimate_fee(target)? as f32
@@ -79,207 +101,43 @@ impl Blockchain for ElectrumBlockchain {
     }
 }
 
-impl GetHeight for ElectrumBlockchain {
-    fn get_height(&self) -> Result<u32, Error> {
-        // TODO: unsubscribe when added to the client, or is there a better call to use here?
-
-        Ok(self
-            .client
-            .block_headers_subscribe()
-            .map(|data| data.height as u32)?)
-    }
-}
-
-impl GetTx for ElectrumBlockchain {
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.client.transaction_get(txid).map(Option::Some)?)
-    }
-}
-
-impl WalletSync for ElectrumBlockchain {
-    fn wallet_setup<D: BatchDatabase>(
+impl ElectrumLikeSync for Client {
+    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script> + Clone>(
         &self,
-        database: &mut D,
-        _progress_update: Box<dyn Progress>,
-    ) -> Result<(), Error> {
-        let mut request = script_sync::start(database, self.stop_gap)?;
-        let mut block_times = HashMap::<u32, u32>::new();
-        let mut txid_to_height = HashMap::<Txid, u32>::new();
-        let mut tx_cache = TxCache::new(database, &self.client);
-        let chunk_size = self.stop_gap;
-        // The electrum server has been inconsistent somehow in its responses during sync. For
-        // example, we do a batch request of transactions and the response contains less
-        // tranascations than in the request. This should never happen but we don't want to panic.
-        let electrum_goof = || Error::Generic("electrum server misbehaving".to_string());
-
-        let batch_update = loop {
-            request = match request {
-                Request::Script(script_req) => {
-                    let scripts = script_req.request().take(chunk_size);
-                    let txids_per_script: Vec<Vec<_>> = self
-                        .client
-                        .batch_script_get_history(scripts)
-                        .map_err(Error::Electrum)?
-                        .into_iter()
-                        .map(|txs| {
-                            txs.into_iter()
-                                .map(|tx| {
-                                    let tx_height = match tx.height {
-                                        none if none <= 0 => None,
-                                        height => {
-                                            txid_to_height.insert(tx.tx_hash, height as u32);
-                                            Some(height as u32)
-                                        }
-                                    };
-                                    (tx.tx_hash, tx_height)
-                                })
-                                .collect()
-                        })
-                        .collect();
-
-                    script_req.satisfy(txids_per_script)?
-                }
-
-                Request::Conftime(conftime_req) => {
-                    // collect up to chunk_size heights to fetch from electrum
-                    let needs_block_height = {
-                        let mut needs_block_height_iter = conftime_req
-                            .request()
-                            .filter_map(|txid| txid_to_height.get(txid).cloned())
-                            .filter(|height| block_times.get(height).is_none());
-                        let mut needs_block_height = HashSet::new();
-
-                        while needs_block_height.len() < chunk_size {
-                            match needs_block_height_iter.next() {
-                                Some(height) => needs_block_height.insert(height),
-                                None => break,
-                            };
-                        }
-                        needs_block_height
-                    };
-
-                    let new_block_headers = self
-                        .client
-                        .batch_block_header(needs_block_height.iter().cloned())?;
-
-                    for (height, header) in needs_block_height.into_iter().zip(new_block_headers) {
-                        block_times.insert(height, header.time);
-                    }
-
-                    let conftimes = conftime_req
-                        .request()
-                        .take(chunk_size)
-                        .map(|txid| {
-                            let confirmation_time = txid_to_height
-                                .get(txid)
-                                .map(|height| {
-                                    let timestamp =
-                                        *block_times.get(height).ok_or_else(electrum_goof)?;
-                                    Result::<_, Error>::Ok(BlockTime {
-                                        height: *height,
-                                        timestamp: timestamp.into(),
-                                    })
-                                })
-                                .transpose()?;
-                            Ok(confirmation_time)
-                        })
-                        .collect::<Result<_, Error>>()?;
-
-                    conftime_req.satisfy(conftimes)?
-                }
-                Request::Tx(tx_req) => {
-                    let needs_full = tx_req.request().take(chunk_size);
-                    tx_cache.save_txs(needs_full.clone())?;
-                    let full_transactions = needs_full
-                        .map(|txid| tx_cache.get(*txid).ok_or_else(electrum_goof))
-                        .collect::<Result<Vec<_>, _>>()?;
-                    let input_txs = full_transactions.iter().flat_map(|tx| {
-                        tx.input
-                            .iter()
-                            .filter(|input| !input.previous_output.is_null())
-                            .map(|input| &input.previous_output.txid)
-                    });
-                    tx_cache.save_txs(input_txs)?;
-
-                    let full_details = full_transactions
-                        .into_iter()
-                        .map(|tx| {
-                            let mut input_index = 0usize;
-                            let prev_outputs = tx
-                                .input
-                                .iter()
-                                .map(|input| {
-                                    if input.previous_output.is_null() {
-                                        return Ok(None);
-                                    }
-                                    let prev_tx = tx_cache
-                                        .get(input.previous_output.txid)
-                                        .ok_or_else(electrum_goof)?;
-                                    let txout = prev_tx
-                                        .output
-                                        .get(input.previous_output.vout as usize)
-                                        .ok_or_else(electrum_goof)?;
-                                    input_index += 1;
-                                    Ok(Some(txout.clone()))
-                                })
-                                .collect::<Result<Vec<_>, Error>>()?;
-                            Ok((prev_outputs, tx))
-                        })
-                        .collect::<Result<Vec<_>, Error>>()?;
-
-                    tx_req.satisfy(full_details)?
-                }
-                Request::Finish(batch_update) => break batch_update,
-            }
-        };
-
-        database.commit_batch(batch_update)?;
-        Ok(())
-    }
-}
-
-struct TxCache<'a, 'b, D> {
-    db: &'a D,
-    client: &'b Client,
-    cache: HashMap<Txid, Transaction>,
-}
-
-impl<'a, 'b, D: Database> TxCache<'a, 'b, D> {
-    fn new(db: &'a D, client: &'b Client) -> Self {
-        TxCache {
-            db,
-            client,
-            cache: HashMap::default(),
-        }
-    }
-    fn save_txs<'c>(&mut self, txids: impl Iterator<Item = &'c Txid>) -> Result<(), Error> {
-        let mut need_fetch = vec![];
-        for txid in txids {
-            if self.cache.get(txid).is_some() {
-                continue;
-            } else if let Some(transaction) = self.db.get_raw_tx(txid)? {
-                self.cache.insert(*txid, transaction);
-            } else {
-                need_fetch.push(txid);
-            }
-        }
-
-        if !need_fetch.is_empty() {
-            let txs = self
-                .client
-                .batch_transaction_get(need_fetch.clone())
-                .map_err(Error::Electrum)?;
-            for (tx, _txid) in txs.into_iter().zip(need_fetch) {
-                debug_assert_eq!(*_txid, tx.txid());
-                self.cache.insert(tx.txid(), tx);
-            }
-        }
-
-        Ok(())
+        scripts: I,
+    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error> {
+        self.batch_script_get_history(scripts)
+            .map(|v| {
+                v.into_iter()
+                    .map(|v| {
+                        v.into_iter()
+                            .map(
+                                |electrum_client::GetHistoryRes {
+                                     height, tx_hash, ..
+                                 }| ElsGetHistoryRes {
+                                    height,
+                                    tx_hash,
+                                },
+                            )
+                            .collect()
+                    })
+                    .collect()
+            })
+            .map_err(Error::Electrum)
     }
 
-    fn get(&self, txid: Txid) -> Option<Transaction> {
-        self.cache.get(&txid).map(Clone::clone)
+    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid> + Clone>(
+        &self,
+        txids: I,
+    ) -> Result<Vec<Transaction>, Error> {
+        self.batch_transaction_get(txids).map_err(Error::Electrum)
+    }
+
+    fn els_batch_block_header<I: IntoIterator<Item = u32> + Clone>(
+        &self,
+        heights: I,
+    ) -> Result<Vec<BlockHeader>, Error> {
+        self.batch_block_header(heights).map_err(Error::Electrum)
     }
 }
 

src/blockchain/esplora.rs

@@ -0,0 +1,433 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Esplora
+//!
+//! This module defines a [`Blockchain`] struct that can query an Esplora backend
+//! populate the wallet's [database](crate::database::Database) by
+//!
+//! ## Example
+//!
+//! ```no_run
+//! # use bdk::blockchain::esplora::EsploraBlockchain;
+//! let blockchain = EsploraBlockchain::new("https://blockstream.info/testnet/api", None, 20);
+//! # Ok::<(), bdk::Error>(())
+//! ```
+
+use std::collections::{HashMap, HashSet};
+use std::fmt;
+
+use bitcoin::consensus::{self, deserialize, serialize};
+use bitcoin::hashes::hex::{FromHex, ToHex};
+use bitcoin::hashes::{sha256, Hash};
+use bitcoin::{BlockHash, BlockHeader, Script, Transaction, Txid};
+use futures::stream::{self, FuturesOrdered, StreamExt, TryStreamExt};
+#[allow(unused_imports)]
+use log::{debug, error, info, trace};
+use reqwest::{Client, StatusCode};
+use serde::Deserialize;
+
+use crate::database::BatchDatabase;
+use crate::error::Error;
+use crate::wallet::utils::ChunksIterator;
+use crate::FeeRate;
+
+use super::*;
+
+use self::utils::{ElectrumLikeSync, ElsGetHistoryRes};
+
+const DEFAULT_CONCURRENT_REQUESTS: u8 = 4;
+
+#[derive(Debug)]
+struct UrlClient {
+    url: String,
+    // We use the async client instead of the blocking one because it automatically uses `fetch`
+    // when the target platform is wasm32.
+    client: Client,
+    concurrency: u8,
+}
+
+/// Structure that implements the logic to sync with Esplora
+///
+/// ## Example
+/// See the [`blockchain::esplora`](crate::blockchain::esplora) module for a usage example.
+#[derive(Debug)]
+pub struct EsploraBlockchain {
+    url_client: UrlClient,
+    stop_gap: usize,
+}
+
+impl std::convert::From<UrlClient> for EsploraBlockchain {
+    fn from(url_client: UrlClient) -> Self {
+        EsploraBlockchain {
+            url_client,
+            stop_gap: 20,
+        }
+    }
+}
+
+impl EsploraBlockchain {
+    /// Create a new instance of the client from a base URL
+    pub fn new(base_url: &str, concurrency: Option<u8>, stop_gap: usize) -> Self {
+        EsploraBlockchain {
+            url_client: UrlClient {
+                url: base_url.to_string(),
+                client: Client::new(),
+                concurrency: concurrency.unwrap_or(DEFAULT_CONCURRENT_REQUESTS),
+            },
+            stop_gap,
+        }
+    }
+}
+
+#[maybe_async]
+impl Blockchain for EsploraBlockchain {
+    fn get_capabilities(&self) -> HashSet<Capability> {
+        vec![
+            Capability::FullHistory,
+            Capability::GetAnyTx,
+            Capability::AccurateFees,
+        ]
+        .into_iter()
+        .collect()
+    }
+
+    fn setup<D: BatchDatabase, P: Progress>(
+        &self,
+        database: &mut D,
+        progress_update: P,
+    ) -> Result<(), Error> {
+        maybe_await!(self
+            .url_client
+            .electrum_like_setup(self.stop_gap, database, progress_update))
+    }
+
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(await_or_block!(self.url_client._get_tx(txid))?)
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        Ok(await_or_block!(self.url_client._broadcast(tx))?)
+    }
+
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(await_or_block!(self.url_client._get_height())?)
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        let estimates = await_or_block!(self.url_client._get_fee_estimates())?;
+
+        let fee_val = estimates
+            .into_iter()
+            .map(|(k, v)| Ok::<_, std::num::ParseIntError>((k.parse::<usize>()?, v)))
+            .collect::<Result<Vec<_>, _>>()
+            .map_err(|e| Error::Generic(e.to_string()))?
+            .into_iter()
+            .take_while(|(k, _)| k <= &target)
+            .map(|(_, v)| v)
+            .last()
+            .unwrap_or(1.0);
+
+        Ok(FeeRate::from_sat_per_vb(fee_val as f32))
+    }
+}
+
+impl UrlClient {
+    fn script_to_scripthash(script: &Script) -> String {
+        sha256::Hash::hash(script.as_bytes()).into_inner().to_hex()
+    }
+
+    async fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
+        let resp = self
+            .client
+            .get(&format!("{}/tx/{}/raw", self.url, txid))
+            .send()
+            .await?;
+
+        if let StatusCode::NOT_FOUND = resp.status() {
+            return Ok(None);
+        }
+
+        Ok(Some(deserialize(&resp.error_for_status()?.bytes().await?)?))
+    }
+
+    async fn _get_tx_no_opt(&self, txid: &Txid) -> Result<Transaction, EsploraError> {
+        match self._get_tx(txid).await {
+            Ok(Some(tx)) => Ok(tx),
+            Ok(None) => Err(EsploraError::TransactionNotFound(*txid)),
+            Err(e) => Err(e),
+        }
+    }
+
+    async fn _get_header(&self, block_height: u32) -> Result<BlockHeader, EsploraError> {
+        let resp = self
+            .client
+            .get(&format!("{}/block-height/{}", self.url, block_height))
+            .send()
+            .await?;
+
+        if let StatusCode::NOT_FOUND = resp.status() {
+            return Err(EsploraError::HeaderHeightNotFound(block_height));
+        }
+        let bytes = resp.bytes().await?;
+        let hash = std::str::from_utf8(&bytes)
+            .map_err(|_| EsploraError::HeaderHeightNotFound(block_height))?;
+
+        let resp = self
+            .client
+            .get(&format!("{}/block/{}/header", self.url, hash))
+            .send()
+            .await?;
+
+        let header = deserialize(&Vec::from_hex(&resp.text().await?)?)?;
+
+        Ok(header)
+    }
+
+    async fn _broadcast(&self, transaction: &Transaction) -> Result<(), EsploraError> {
+        self.client
+            .post(&format!("{}/tx", self.url))
+            .body(serialize(transaction).to_hex())
+            .send()
+            .await?
+            .error_for_status()?;
+
+        Ok(())
+    }
+
+    async fn _get_height(&self) -> Result<u32, EsploraError> {
+        let req = self
+            .client
+            .get(&format!("{}/blocks/tip/height", self.url))
+            .send()
+            .await?;
+
+        Ok(req.error_for_status()?.text().await?.parse()?)
+    }
+
+    async fn _script_get_history(
+        &self,
+        script: &Script,
+    ) -> Result<Vec<ElsGetHistoryRes>, EsploraError> {
+        let mut result = Vec::new();
+        let scripthash = Self::script_to_scripthash(script);
+
+        // Add the unconfirmed transactions first
+        result.extend(
+            self.client
+                .get(&format!(
+                    "{}/scripthash/{}/txs/mempool",
+                    self.url, scripthash
+                ))
+                .send()
+                .await?
+                .error_for_status()?
+                .json::<Vec<EsploraGetHistory>>()
+                .await?
+                .into_iter()
+                .map(|x| ElsGetHistoryRes {
+                    tx_hash: x.txid,
+                    height: x.status.block_height.unwrap_or(0) as i32,
+                }),
+        );
+
+        debug!(
+            "Found {} mempool txs for {} - {:?}",
+            result.len(),
+            scripthash,
+            script
+        );
+
+        // Then go through all the pages of confirmed transactions
+        let mut last_txid = String::new();
+        loop {
+            let response = self
+                .client
+                .get(&format!(
+                    "{}/scripthash/{}/txs/chain/{}",
+                    self.url, scripthash, last_txid
+                ))
+                .send()
+                .await?
+                .error_for_status()?
+                .json::<Vec<EsploraGetHistory>>()
+                .await?;
+            let len = response.len();
+            if let Some(elem) = response.last() {
+                last_txid = elem.txid.to_hex();
+            }
+
+            debug!("... adding {} confirmed transactions", len);
+
+            result.extend(response.into_iter().map(|x| ElsGetHistoryRes {
+                tx_hash: x.txid,
+                height: x.status.block_height.unwrap_or(0) as i32,
+            }));
+
+            if len < 25 {
+                break;
+            }
+        }
+
+        Ok(result)
+    }
+
+    async fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
+        Ok(self
+            .client
+            .get(&format!("{}/fee-estimates", self.url,))
+            .send()
+            .await?
+            .error_for_status()?
+            .json::<HashMap<String, f64>>()
+            .await?)
+    }
+}
+
+#[maybe_async]
+impl ElectrumLikeSync for UrlClient {
+    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script>>(
+        &self,
+        scripts: I,
+    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error> {
+        let future = async {
+            let mut results = vec![];
+            for chunk in ChunksIterator::new(scripts.into_iter(), self.concurrency as usize) {
+                let mut futs = FuturesOrdered::new();
+                for script in chunk {
+                    futs.push(self._script_get_history(&script));
+                }
+                let partial_results: Vec<Vec<ElsGetHistoryRes>> = futs.try_collect().await?;
+                results.extend(partial_results);
+            }
+            Ok(stream::iter(results).collect().await)
+        };
+
+        await_or_block!(future)
+    }
+
+    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid>>(
+        &self,
+        txids: I,
+    ) -> Result<Vec<Transaction>, Error> {
+        let future = async {
+            let mut results = vec![];
+            for chunk in ChunksIterator::new(txids.into_iter(), self.concurrency as usize) {
+                let mut futs = FuturesOrdered::new();
+                for txid in chunk {
+                    futs.push(self._get_tx_no_opt(&txid));
+                }
+                let partial_results: Vec<Transaction> = futs.try_collect().await?;
+                results.extend(partial_results);
+            }
+            Ok(stream::iter(results).collect().await)
+        };
+
+        await_or_block!(future)
+    }
+
+    fn els_batch_block_header<I: IntoIterator<Item = u32>>(
+        &self,
+        heights: I,
+    ) -> Result<Vec<BlockHeader>, Error> {
+        let future = async {
+            let mut results = vec![];
+            for chunk in ChunksIterator::new(heights.into_iter(), self.concurrency as usize) {
+                let mut futs = FuturesOrdered::new();
+                for height in chunk {
+                    futs.push(self._get_header(height));
+                }
+                let partial_results: Vec<BlockHeader> = futs.try_collect().await?;
+                results.extend(partial_results);
+            }
+            Ok(stream::iter(results).collect().await)
+        };
+
+        await_or_block!(future)
+    }
+}
+
+#[derive(Deserialize)]
+struct EsploraGetHistoryStatus {
+    block_height: Option<usize>,
+}
+
+#[derive(Deserialize)]
+struct EsploraGetHistory {
+    txid: Txid,
+    status: EsploraGetHistoryStatus,
+}
+
+/// Configuration for an [`EsploraBlockchain`]
+#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
+pub struct EsploraBlockchainConfig {
+    /// Base URL of the esplora service
+    ///
+    /// eg. `https://blockstream.info/api/`
+    pub base_url: String,
+    /// Number of parallel requests sent to the esplora service (default: 4)
+    pub concurrency: Option<u8>,
+    /// Stop searching addresses for transactions after finding an unused gap of this length
+    pub stop_gap: usize,
+}
+
+impl ConfigurableBlockchain for EsploraBlockchain {
+    type Config = EsploraBlockchainConfig;
+
+    fn from_config(config: &Self::Config) -> Result<Self, Error> {
+        Ok(EsploraBlockchain::new(
+            config.base_url.as_str(),
+            config.concurrency,
+            config.stop_gap,
+        ))
+    }
+}
+
+/// Errors that can happen during a sync with [`EsploraBlockchain`]
+#[derive(Debug)]
+pub enum EsploraError {
+    /// Error with the HTTP call
+    Reqwest(reqwest::Error),
+    /// Invalid number returned
+    Parsing(std::num::ParseIntError),
+    /// Invalid Bitcoin data returned
+    BitcoinEncoding(bitcoin::consensus::encode::Error),
+    /// Invalid Hex data returned
+    Hex(bitcoin::hashes::hex::Error),
+
+    /// Transaction not found
+    TransactionNotFound(Txid),
+    /// Header height not found
+    HeaderHeightNotFound(u32),
+    /// Header hash not found
+    HeaderHashNotFound(BlockHash),
+}
+
+impl fmt::Display for EsploraError {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(f, "{:?}", self)
+    }
+}
+
+impl std::error::Error for EsploraError {}
+
+impl_error!(reqwest::Error, Reqwest, EsploraError);
+impl_error!(std::num::ParseIntError, Parsing, EsploraError);
+impl_error!(consensus::encode::Error, BitcoinEncoding, EsploraError);
+impl_error!(bitcoin::hashes::hex::Error, Hex, EsploraError);
+
+#[cfg(test)]
+#[cfg(feature = "test-esplora")]
+crate::bdk_blockchain_tests! {
+    fn test_instance(test_client: &TestClient) -> EsploraBlockchain {
+        EsploraBlockchain::new(&format!("http://{}",test_client.electrsd.esplora_url.as_ref().unwrap()), None, 20)
+    }
+}

src/blockchain/esplora/api.rs

@@ -1,117 +0,0 @@
-//! structs from the esplora API
-//!
-//! see: <https://github.com/Blockstream/esplora/blob/master/API.md>
-use crate::BlockTime;
-use bitcoin::{OutPoint, Script, Transaction, TxIn, TxOut, Txid};
-
-#[derive(serde::Deserialize, Clone, Debug)]
-pub struct PrevOut {
-    pub value: u64,
-    pub scriptpubkey: Script,
-}
-
-#[derive(serde::Deserialize, Clone, Debug)]
-pub struct Vin {
-    pub txid: Txid,
-    pub vout: u32,
-    // None if coinbase
-    pub prevout: Option<PrevOut>,
-    pub scriptsig: Script,
-    #[serde(deserialize_with = "deserialize_witness", default)]
-    pub witness: Vec<Vec<u8>>,
-    pub sequence: u32,
-    pub is_coinbase: bool,
-}
-
-#[derive(serde::Deserialize, Clone, Debug)]
-pub struct Vout {
-    pub value: u64,
-    pub scriptpubkey: Script,
-}
-
-#[derive(serde::Deserialize, Clone, Debug)]
-pub struct TxStatus {
-    pub confirmed: bool,
-    pub block_height: Option<u32>,
-    pub block_time: Option<u64>,
-}
-
-#[derive(serde::Deserialize, Clone, Debug)]
-pub struct Tx {
-    pub txid: Txid,
-    pub version: i32,
-    pub locktime: u32,
-    pub vin: Vec<Vin>,
-    pub vout: Vec<Vout>,
-    pub status: TxStatus,
-    pub fee: u64,
-}
-
-impl Tx {
-    pub fn to_tx(&self) -> Transaction {
-        Transaction {
-            version: self.version,
-            lock_time: self.locktime,
-            input: self
-                .vin
-                .iter()
-                .cloned()
-                .map(|vin| TxIn {
-                    previous_output: OutPoint {
-                        txid: vin.txid,
-                        vout: vin.vout,
-                    },
-                    script_sig: vin.scriptsig,
-                    sequence: vin.sequence,
-                    witness: vin.witness,
-                })
-                .collect(),
-            output: self
-                .vout
-                .iter()
-                .cloned()
-                .map(|vout| TxOut {
-                    value: vout.value,
-                    script_pubkey: vout.scriptpubkey,
-                })
-                .collect(),
-        }
-    }
-
-    pub fn confirmation_time(&self) -> Option<BlockTime> {
-        match self.status {
-            TxStatus {
-                confirmed: true,
-                block_height: Some(height),
-                block_time: Some(timestamp),
-            } => Some(BlockTime { timestamp, height }),
-            _ => None,
-        }
-    }
-
-    pub fn previous_outputs(&self) -> Vec<Option<TxOut>> {
-        self.vin
-            .iter()
-            .cloned()
-            .map(|vin| {
-                vin.prevout.map(|po| TxOut {
-                    script_pubkey: po.scriptpubkey,
-                    value: po.value,
-                })
-            })
-            .collect()
-    }
-}
-
-fn deserialize_witness<'de, D>(d: D) -> Result<Vec<Vec<u8>>, D::Error>
-where
-    D: serde::de::Deserializer<'de>,
-{
-    use crate::serde::Deserialize;
-    use bitcoin::hashes::hex::FromHex;
-    let list = Vec::<String>::deserialize(d)?;
-    list.into_iter()
-        .map(|hex_str| Vec::<u8>::from_hex(&hex_str))
-        .collect::<Result<Vec<Vec<u8>>, _>>()
-        .map_err(serde::de::Error::custom)
-}

src/blockchain/esplora/mod.rs

@@ -1,212 +0,0 @@
-//! Esplora
-//!
-//! This module defines a [`EsploraBlockchain`] struct that can query an Esplora
-//! backend populate the wallet's [database](crate::database::Database) by:
-//!
-//! ## Example
-//!
-//! ```no_run
-//! # use bdk::blockchain::esplora::EsploraBlockchain;
-//! let blockchain = EsploraBlockchain::new("https://blockstream.info/testnet/api", 20);
-//! # Ok::<(), bdk::Error>(())
-//! ```
-//!
-//! Esplora blockchain can use either `ureq` or `reqwest` for the HTTP client
-//! depending on your needs (blocking or async respectively).
-//!
-//! Please note, to configure the Esplora HTTP client correctly use one of:
-//! Blocking:  --features='esplora,ureq'
-//! Async:     --features='async-interface,esplora,reqwest' --no-default-features
-use std::collections::HashMap;
-use std::fmt;
-use std::io;
-
-use bitcoin::consensus;
-use bitcoin::{BlockHash, Txid};
-
-use crate::error::Error;
-use crate::FeeRate;
-
-#[cfg(feature = "reqwest")]
-mod reqwest;
-
-#[cfg(feature = "reqwest")]
-pub use self::reqwest::*;
-
-#[cfg(feature = "ureq")]
-mod ureq;
-
-#[cfg(feature = "ureq")]
-pub use self::ureq::*;
-
-mod api;
-
-fn into_fee_rate(target: usize, estimates: HashMap<String, f64>) -> Result<FeeRate, Error> {
-    let fee_val = {
-        let mut pairs = estimates
-            .into_iter()
-            .filter_map(|(k, v)| Some((k.parse::<usize>().ok()?, v)))
-            .collect::<Vec<_>>();
-        pairs.sort_unstable_by_key(|(k, _)| std::cmp::Reverse(*k));
-        pairs
-            .into_iter()
-            .find(|(k, _)| k <= &target)
-            .map(|(_, v)| v)
-            .unwrap_or(1.0)
-    };
-    Ok(FeeRate::from_sat_per_vb(fee_val as f32))
-}
-
-/// Errors that can happen during a sync with [`EsploraBlockchain`]
-#[derive(Debug)]
-pub enum EsploraError {
-    /// Error during ureq HTTP request
-    #[cfg(feature = "ureq")]
-    Ureq(::ureq::Error),
-    /// Transport error during the ureq HTTP call
-    #[cfg(feature = "ureq")]
-    UreqTransport(::ureq::Transport),
-    /// Error during reqwest HTTP request
-    #[cfg(feature = "reqwest")]
-    Reqwest(::reqwest::Error),
-    /// HTTP response error
-    HttpResponse(u16),
-    /// IO error during ureq response read
-    Io(io::Error),
-    /// No header found in ureq response
-    NoHeader,
-    /// Invalid number returned
-    Parsing(std::num::ParseIntError),
-    /// Invalid Bitcoin data returned
-    BitcoinEncoding(bitcoin::consensus::encode::Error),
-    /// Invalid Hex data returned
-    Hex(bitcoin::hashes::hex::Error),
-
-    /// Transaction not found
-    TransactionNotFound(Txid),
-    /// Header height not found
-    HeaderHeightNotFound(u32),
-    /// Header hash not found
-    HeaderHashNotFound(BlockHash),
-}
-
-impl fmt::Display for EsploraError {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        write!(f, "{:?}", self)
-    }
-}
-
-/// Configuration for an [`EsploraBlockchain`]
-#[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
-pub struct EsploraBlockchainConfig {
-    /// Base URL of the esplora service
-    ///
-    /// eg. `https://blockstream.info/api/`
-    pub base_url: String,
-    /// Optional URL of the proxy to use to make requests to the Esplora server
-    ///
-    /// The string should be formatted as: `<protocol>://<user>:<password>@host:<port>`.
-    ///
-    /// Note that the format of this value and the supported protocols change slightly between the
-    /// sync version of esplora (using `ureq`) and the async version (using `reqwest`). For more
-    /// details check with the documentation of the two crates. Both of them are compiled with
-    /// the `socks` feature enabled.
-    ///
-    /// The proxy is ignored when targeting `wasm32`.
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub proxy: Option<String>,
-    /// Number of parallel requests sent to the esplora service (default: 4)
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub concurrency: Option<u8>,
-    /// Stop searching addresses for transactions after finding an unused gap of this length.
-    pub stop_gap: usize,
-    /// Socket timeout.
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub timeout: Option<u64>,
-}
-
-impl EsploraBlockchainConfig {
-    /// create a config with default values given the base url and stop gap
-    pub fn new(base_url: String, stop_gap: usize) -> Self {
-        Self {
-            base_url,
-            proxy: None,
-            timeout: None,
-            stop_gap,
-            concurrency: None,
-        }
-    }
-}
-
-impl std::error::Error for EsploraError {}
-
-#[cfg(feature = "ureq")]
-impl_error!(::ureq::Transport, UreqTransport, EsploraError);
-#[cfg(feature = "reqwest")]
-impl_error!(::reqwest::Error, Reqwest, EsploraError);
-impl_error!(io::Error, Io, EsploraError);
-impl_error!(std::num::ParseIntError, Parsing, EsploraError);
-impl_error!(consensus::encode::Error, BitcoinEncoding, EsploraError);
-impl_error!(bitcoin::hashes::hex::Error, Hex, EsploraError);
-
-#[cfg(test)]
-#[cfg(feature = "test-esplora")]
-crate::bdk_blockchain_tests! {
-    fn test_instance(test_client: &TestClient) -> EsploraBlockchain {
-        EsploraBlockchain::new(&format!("http://{}",test_client.electrsd.esplora_url.as_ref().unwrap()), 20)
-    }
-}
-
-const DEFAULT_CONCURRENT_REQUESTS: u8 = 4;
-
-#[cfg(test)]
-mod test {
-    use super::*;
-
-    #[test]
-    fn feerate_parsing() {
-        let esplora_fees = serde_json::from_str::<HashMap<String, f64>>(
-            r#"{
-  "25": 1.015,
-  "5": 2.3280000000000003,
-  "12": 2.0109999999999997,
-  "15": 1.018,
-  "17": 1.018,
-  "11": 2.0109999999999997,
-  "3": 3.01,
-  "2": 4.9830000000000005,
-  "6": 2.2359999999999998,
-  "21": 1.018,
-  "13": 1.081,
-  "7": 2.2359999999999998,
-  "8": 2.2359999999999998,
-  "16": 1.018,
-  "20": 1.018,
-  "22": 1.017,
-  "23": 1.017,
-  "504": 1,
-  "9": 2.2359999999999998,
-  "14": 1.018,
-  "10": 2.0109999999999997,
-  "24": 1.017,
-  "1008": 1,
-  "1": 4.9830000000000005,
-  "4": 2.3280000000000003,
-  "19": 1.018,
-  "144": 1,
-  "18": 1.018
-}
-"#,
-        )
-        .unwrap();
-        assert_eq!(
-            into_fee_rate(6, esplora_fees.clone()).unwrap(),
-            FeeRate::from_sat_per_vb(2.236)
-        );
-        assert_eq!(
-            into_fee_rate(26, esplora_fees).unwrap(),
-            FeeRate::from_sat_per_vb(1.015),
-            "should inherit from value for 25"
-        );
-    }
-}

src/blockchain/esplora/reqwest.rs

@@ -1,340 +0,0 @@
-// Bitcoin Dev Kit
-// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
-//
-// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
-//
-// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
-// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
-// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
-// You may not use this file except in accordance with one or both of these
-// licenses.
-
-//! Esplora by way of `reqwest` HTTP client.
-
-use std::collections::{HashMap, HashSet};
-
-use bitcoin::consensus::{deserialize, serialize};
-use bitcoin::hashes::hex::{FromHex, ToHex};
-use bitcoin::hashes::{sha256, Hash};
-use bitcoin::{BlockHeader, Script, Transaction, Txid};
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
-
-use ::reqwest::{Client, StatusCode};
-use futures::stream::{FuturesOrdered, TryStreamExt};
-
-use super::api::Tx;
-use crate::blockchain::esplora::EsploraError;
-use crate::blockchain::*;
-use crate::database::BatchDatabase;
-use crate::error::Error;
-use crate::FeeRate;
-
-#[derive(Debug)]
-struct UrlClient {
-    url: String,
-    // We use the async client instead of the blocking one because it automatically uses `fetch`
-    // when the target platform is wasm32.
-    client: Client,
-    concurrency: u8,
-}
-
-/// Structure that implements the logic to sync with Esplora
-///
-/// ## Example
-/// See the [`blockchain::esplora`](crate::blockchain::esplora) module for a usage example.
-#[derive(Debug)]
-pub struct EsploraBlockchain {
-    url_client: UrlClient,
-    stop_gap: usize,
-}
-
-impl std::convert::From<UrlClient> for EsploraBlockchain {
-    fn from(url_client: UrlClient) -> Self {
-        EsploraBlockchain {
-            url_client,
-            stop_gap: 20,
-        }
-    }
-}
-
-impl EsploraBlockchain {
-    /// Create a new instance of the client from a base URL and `stop_gap`.
-    pub fn new(base_url: &str, stop_gap: usize) -> Self {
-        EsploraBlockchain {
-            url_client: UrlClient {
-                url: base_url.to_string(),
-                client: Client::new(),
-                concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
-            },
-            stop_gap,
-        }
-    }
-
-    /// Set the concurrency to use when doing batch queries against the Esplora instance.
-    pub fn with_concurrency(mut self, concurrency: u8) -> Self {
-        self.url_client.concurrency = concurrency;
-        self
-    }
-}
-
-#[maybe_async]
-impl Blockchain for EsploraBlockchain {
-    fn get_capabilities(&self) -> HashSet<Capability> {
-        vec![
-            Capability::FullHistory,
-            Capability::GetAnyTx,
-            Capability::AccurateFees,
-        ]
-        .into_iter()
-        .collect()
-    }
-
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        Ok(await_or_block!(self.url_client._broadcast(tx))?)
-    }
-
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
-        let estimates = await_or_block!(self.url_client._get_fee_estimates())?;
-        super::into_fee_rate(target, estimates)
-    }
-}
-
-#[maybe_async]
-impl GetHeight for EsploraBlockchain {
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(await_or_block!(self.url_client._get_height())?)
-    }
-}
-
-#[maybe_async]
-impl GetTx for EsploraBlockchain {
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(await_or_block!(self.url_client._get_tx(txid))?)
-    }
-}
-
-#[maybe_async]
-impl WalletSync for EsploraBlockchain {
-    fn wallet_setup<D: BatchDatabase>(
-        &self,
-        database: &mut D,
-        _progress_update: Box<dyn Progress>,
-    ) -> Result<(), Error> {
-        use crate::blockchain::script_sync::Request;
-        let mut request = script_sync::start(database, self.stop_gap)?;
-        let mut tx_index: HashMap<Txid, Tx> = HashMap::new();
-
-        let batch_update = loop {
-            request = match request {
-                Request::Script(script_req) => {
-                    let futures: FuturesOrdered<_> = script_req
-                        .request()
-                        .take(self.url_client.concurrency as usize)
-                        .map(|script| async move {
-                            let mut related_txs: Vec<Tx> =
-                                self.url_client._scripthash_txs(script, None).await?;
-
-                            let n_confirmed =
-                                related_txs.iter().filter(|tx| tx.status.confirmed).count();
-                            // esplora pages on 25 confirmed transactions. If there's 25 or more we
-                            // keep requesting to see if there's more.
-                            if n_confirmed >= 25 {
-                                loop {
-                                    let new_related_txs: Vec<Tx> = self
-                                        .url_client
-                                        ._scripthash_txs(
-                                            script,
-                                            Some(related_txs.last().unwrap().txid),
-                                        )
-                                        .await?;
-                                    let n = new_related_txs.len();
-                                    related_txs.extend(new_related_txs);
-                                    // we've reached the end
-                                    if n < 25 {
-                                        break;
-                                    }
-                                }
-                            }
-                            Result::<_, Error>::Ok(related_txs)
-                        })
-                        .collect();
-                    let txs_per_script: Vec<Vec<Tx>> = await_or_block!(futures.try_collect())?;
-                    let mut satisfaction = vec![];
-
-                    for txs in txs_per_script {
-                        satisfaction.push(
-                            txs.iter()
-                                .map(|tx| (tx.txid, tx.status.block_height))
-                                .collect(),
-                        );
-                        for tx in txs {
-                            tx_index.insert(tx.txid, tx);
-                        }
-                    }
-
-                    script_req.satisfy(satisfaction)?
-                }
-                Request::Conftime(conftime_req) => {
-                    let conftimes = conftime_req
-                        .request()
-                        .map(|txid| {
-                            tx_index
-                                .get(txid)
-                                .expect("must be in index")
-                                .confirmation_time()
-                        })
-                        .collect();
-                    conftime_req.satisfy(conftimes)?
-                }
-                Request::Tx(tx_req) => {
-                    let full_txs = tx_req
-                        .request()
-                        .map(|txid| {
-                            let tx = tx_index.get(txid).expect("must be in index");
-                            Ok((tx.previous_outputs(), tx.to_tx()))
-                        })
-                        .collect::<Result<_, Error>>()?;
-                    tx_req.satisfy(full_txs)?
-                }
-                Request::Finish(batch_update) => break batch_update,
-            }
-        };
-
-        database.commit_batch(batch_update)?;
-
-        Ok(())
-    }
-}
-
-impl UrlClient {
-    async fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
-        let resp = self
-            .client
-            .get(&format!("{}/tx/{}/raw", self.url, txid))
-            .send()
-            .await?;
-
-        if let StatusCode::NOT_FOUND = resp.status() {
-            return Ok(None);
-        }
-
-        Ok(Some(deserialize(&resp.error_for_status()?.bytes().await?)?))
-    }
-
-    async fn _get_tx_no_opt(&self, txid: &Txid) -> Result<Transaction, EsploraError> {
-        match self._get_tx(txid).await {
-            Ok(Some(tx)) => Ok(tx),
-            Ok(None) => Err(EsploraError::TransactionNotFound(*txid)),
-            Err(e) => Err(e),
-        }
-    }
-
-    async fn _get_header(&self, block_height: u32) -> Result<BlockHeader, EsploraError> {
-        let resp = self
-            .client
-            .get(&format!("{}/block-height/{}", self.url, block_height))
-            .send()
-            .await?;
-
-        if let StatusCode::NOT_FOUND = resp.status() {
-            return Err(EsploraError::HeaderHeightNotFound(block_height));
-        }
-        let bytes = resp.bytes().await?;
-        let hash = std::str::from_utf8(&bytes)
-            .map_err(|_| EsploraError::HeaderHeightNotFound(block_height))?;
-
-        let resp = self
-            .client
-            .get(&format!("{}/block/{}/header", self.url, hash))
-            .send()
-            .await?;
-
-        let header = deserialize(&Vec::from_hex(&resp.text().await?)?)?;
-
-        Ok(header)
-    }
-
-    async fn _broadcast(&self, transaction: &Transaction) -> Result<(), EsploraError> {
-        self.client
-            .post(&format!("{}/tx", self.url))
-            .body(serialize(transaction).to_hex())
-            .send()
-            .await?
-            .error_for_status()?;
-
-        Ok(())
-    }
-
-    async fn _get_height(&self) -> Result<u32, EsploraError> {
-        let req = self
-            .client
-            .get(&format!("{}/blocks/tip/height", self.url))
-            .send()
-            .await?;
-
-        Ok(req.error_for_status()?.text().await?.parse()?)
-    }
-
-    async fn _scripthash_txs(
-        &self,
-        script: &Script,
-        last_seen: Option<Txid>,
-    ) -> Result<Vec<Tx>, EsploraError> {
-        let script_hash = sha256::Hash::hash(script.as_bytes()).into_inner().to_hex();
-        let url = match last_seen {
-            Some(last_seen) => format!(
-                "{}/scripthash/{}/txs/chain/{}",
-                self.url, script_hash, last_seen
-            ),
-            None => format!("{}/scripthash/{}/txs", self.url, script_hash),
-        };
-        Ok(self
-            .client
-            .get(url)
-            .send()
-            .await?
-            .error_for_status()?
-            .json::<Vec<Tx>>()
-            .await?)
-    }
-
-    async fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
-        Ok(self
-            .client
-            .get(&format!("{}/fee-estimates", self.url,))
-            .send()
-            .await?
-            .error_for_status()?
-            .json::<HashMap<String, f64>>()
-            .await?)
-    }
-}
-
-impl ConfigurableBlockchain for EsploraBlockchain {
-    type Config = super::EsploraBlockchainConfig;
-
-    fn from_config(config: &Self::Config) -> Result<Self, Error> {
-        let map_e = |e: reqwest::Error| Error::Esplora(Box::new(e.into()));
-
-        let mut blockchain = EsploraBlockchain::new(config.base_url.as_str(), config.stop_gap);
-        if let Some(concurrency) = config.concurrency {
-            blockchain.url_client.concurrency = concurrency;
-        }
-        let mut builder = Client::builder();
-        #[cfg(not(target_arch = "wasm32"))]
-        if let Some(proxy) = &config.proxy {
-            builder = builder.proxy(reqwest::Proxy::all(proxy).map_err(map_e)?);
-        }
-
-        #[cfg(not(target_arch = "wasm32"))]
-        if let Some(timeout) = config.timeout {
-            builder = builder.timeout(core::time::Duration::from_secs(timeout));
-        }
-
-        blockchain.url_client.client = builder.build().map_err(map_e)?;
-
-        Ok(blockchain)
-    }
-}

src/blockchain/esplora/ureq.rs

@@ -1,377 +0,0 @@
-// Bitcoin Dev Kit
-// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
-//
-// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
-//
-// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
-// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
-// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
-// You may not use this file except in accordance with one or both of these
-// licenses.
-
-//! Esplora by way of `ureq` HTTP client.
-
-use std::collections::{HashMap, HashSet};
-use std::io;
-use std::io::Read;
-use std::time::Duration;
-
-#[allow(unused_imports)]
-use log::{debug, error, info, trace};
-
-use ureq::{Agent, Proxy, Response};
-
-use bitcoin::consensus::{deserialize, serialize};
-use bitcoin::hashes::hex::{FromHex, ToHex};
-use bitcoin::hashes::{sha256, Hash};
-use bitcoin::{BlockHeader, Script, Transaction, Txid};
-
-use super::api::Tx;
-use crate::blockchain::esplora::EsploraError;
-use crate::blockchain::*;
-use crate::database::BatchDatabase;
-use crate::error::Error;
-use crate::FeeRate;
-
-#[derive(Debug, Clone)]
-struct UrlClient {
-    url: String,
-    agent: Agent,
-}
-
-/// Structure that implements the logic to sync with Esplora
-///
-/// ## Example
-/// See the [`blockchain::esplora`](crate::blockchain::esplora) module for a usage example.
-#[derive(Debug)]
-pub struct EsploraBlockchain {
-    url_client: UrlClient,
-    stop_gap: usize,
-    concurrency: u8,
-}
-
-impl EsploraBlockchain {
-    /// Create a new instance of the client from a base URL and the `stop_gap`.
-    pub fn new(base_url: &str, stop_gap: usize) -> Self {
-        EsploraBlockchain {
-            url_client: UrlClient {
-                url: base_url.to_string(),
-                agent: Agent::new(),
-            },
-            concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
-            stop_gap,
-        }
-    }
-
-    /// Set the inner `ureq` agent.
-    pub fn with_agent(mut self, agent: Agent) -> Self {
-        self.url_client.agent = agent;
-        self
-    }
-
-    /// Set the number of parallel requests the client can make.
-    pub fn with_concurrency(mut self, concurrency: u8) -> Self {
-        self.concurrency = concurrency;
-        self
-    }
-}
-
-impl Blockchain for EsploraBlockchain {
-    fn get_capabilities(&self) -> HashSet<Capability> {
-        vec![
-            Capability::FullHistory,
-            Capability::GetAnyTx,
-            Capability::AccurateFees,
-        ]
-        .into_iter()
-        .collect()
-    }
-
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        let _txid = self.url_client._broadcast(tx)?;
-        Ok(())
-    }
-
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
-        let estimates = self.url_client._get_fee_estimates()?;
-        super::into_fee_rate(target, estimates)
-    }
-}
-
-impl GetHeight for EsploraBlockchain {
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(self.url_client._get_height()?)
-    }
-}
-
-impl GetTx for EsploraBlockchain {
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.url_client._get_tx(txid)?)
-    }
-}
-
-impl WalletSync for EsploraBlockchain {
-    fn wallet_setup<D: BatchDatabase>(
-        &self,
-        database: &mut D,
-        _progress_update: Box<dyn Progress>,
-    ) -> Result<(), Error> {
-        use crate::blockchain::script_sync::Request;
-        let mut request = script_sync::start(database, self.stop_gap)?;
-        let mut tx_index: HashMap<Txid, Tx> = HashMap::new();
-        let batch_update = loop {
-            request = match request {
-                Request::Script(script_req) => {
-                    let scripts = script_req
-                        .request()
-                        .take(self.concurrency as usize)
-                        .cloned();
-
-                    let handles = scripts.map(move |script| {
-                        let client = self.url_client.clone();
-                        // make each request in its own thread.
-                        std::thread::spawn(move || {
-                            let mut related_txs: Vec<Tx> = client._scripthash_txs(&script, None)?;
-
-                            let n_confirmed =
-                                related_txs.iter().filter(|tx| tx.status.confirmed).count();
-                            // esplora pages on 25 confirmed transactions. If there's 25 or more we
-                            // keep requesting to see if there's more.
-                            if n_confirmed >= 25 {
-                                loop {
-                                    let new_related_txs: Vec<Tx> = client._scripthash_txs(
-                                        &script,
-                                        Some(related_txs.last().unwrap().txid),
-                                    )?;
-                                    let n = new_related_txs.len();
-                                    related_txs.extend(new_related_txs);
-                                    // we've reached the end
-                                    if n < 25 {
-                                        break;
-                                    }
-                                }
-                            }
-                            Result::<_, Error>::Ok(related_txs)
-                        })
-                    });
-
-                    let txs_per_script: Vec<Vec<Tx>> = handles
-                        .map(|handle| handle.join().unwrap())
-                        .collect::<Result<_, _>>()?;
-                    let mut satisfaction = vec![];
-
-                    for txs in txs_per_script {
-                        satisfaction.push(
-                            txs.iter()
-                                .map(|tx| (tx.txid, tx.status.block_height))
-                                .collect(),
-                        );
-                        for tx in txs {
-                            tx_index.insert(tx.txid, tx);
-                        }
-                    }
-
-                    script_req.satisfy(satisfaction)?
-                }
-                Request::Conftime(conftime_req) => {
-                    let conftimes = conftime_req
-                        .request()
-                        .map(|txid| {
-                            tx_index
-                                .get(txid)
-                                .expect("must be in index")
-                                .confirmation_time()
-                        })
-                        .collect();
-                    conftime_req.satisfy(conftimes)?
-                }
-                Request::Tx(tx_req) => {
-                    let full_txs = tx_req
-                        .request()
-                        .map(|txid| {
-                            let tx = tx_index.get(txid).expect("must be in index");
-                            Ok((tx.previous_outputs(), tx.to_tx()))
-                        })
-                        .collect::<Result<_, Error>>()?;
-                    tx_req.satisfy(full_txs)?
-                }
-                Request::Finish(batch_update) => break batch_update,
-            }
-        };
-
-        database.commit_batch(batch_update)?;
-
-        Ok(())
-    }
-}
-
-impl UrlClient {
-    fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
-        let resp = self
-            .agent
-            .get(&format!("{}/tx/{}/raw", self.url, txid))
-            .call();
-
-        match resp {
-            Ok(resp) => Ok(Some(deserialize(&into_bytes(resp)?)?)),
-            Err(ureq::Error::Status(code, _)) => {
-                if is_status_not_found(code) {
-                    return Ok(None);
-                }
-                Err(EsploraError::HttpResponse(code))
-            }
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }
-    }
-
-    fn _get_tx_no_opt(&self, txid: &Txid) -> Result<Transaction, EsploraError> {
-        match self._get_tx(txid) {
-            Ok(Some(tx)) => Ok(tx),
-            Ok(None) => Err(EsploraError::TransactionNotFound(*txid)),
-            Err(e) => Err(e),
-        }
-    }
-
-    fn _get_header(&self, block_height: u32) -> Result<BlockHeader, EsploraError> {
-        let resp = self
-            .agent
-            .get(&format!("{}/block-height/{}", self.url, block_height))
-            .call();
-
-        let bytes = match resp {
-            Ok(resp) => Ok(into_bytes(resp)?),
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }?;
-
-        let hash = std::str::from_utf8(&bytes)
-            .map_err(|_| EsploraError::HeaderHeightNotFound(block_height))?;
-
-        let resp = self
-            .agent
-            .get(&format!("{}/block/{}/header", self.url, hash))
-            .call();
-
-        match resp {
-            Ok(resp) => Ok(deserialize(&Vec::from_hex(&resp.into_string()?)?)?),
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }
-    }
-
-    fn _broadcast(&self, transaction: &Transaction) -> Result<(), EsploraError> {
-        let resp = self
-            .agent
-            .post(&format!("{}/tx", self.url))
-            .send_string(&serialize(transaction).to_hex());
-
-        match resp {
-            Ok(_) => Ok(()), // We do not return the txid?
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }
-    }
-
-    fn _get_height(&self) -> Result<u32, EsploraError> {
-        let resp = self
-            .agent
-            .get(&format!("{}/blocks/tip/height", self.url))
-            .call();
-
-        match resp {
-            Ok(resp) => Ok(resp.into_string()?.parse()?),
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }
-    }
-
-    fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
-        let resp = self
-            .agent
-            .get(&format!("{}/fee-estimates", self.url,))
-            .call();
-
-        let map = match resp {
-            Ok(resp) => {
-                let map: HashMap<String, f64> = resp.into_json()?;
-                Ok(map)
-            }
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }?;
-
-        Ok(map)
-    }
-
-    fn _scripthash_txs(
-        &self,
-        script: &Script,
-        last_seen: Option<Txid>,
-    ) -> Result<Vec<Tx>, EsploraError> {
-        let script_hash = sha256::Hash::hash(script.as_bytes()).into_inner().to_hex();
-        let url = match last_seen {
-            Some(last_seen) => format!(
-                "{}/scripthash/{}/txs/chain/{}",
-                self.url, script_hash, last_seen
-            ),
-            None => format!("{}/scripthash/{}/txs", self.url, script_hash),
-        };
-        Ok(self.agent.get(&url).call()?.into_json()?)
-    }
-}
-
-fn is_status_not_found(status: u16) -> bool {
-    status == 404
-}
-
-fn into_bytes(resp: Response) -> Result<Vec<u8>, io::Error> {
-    const BYTES_LIMIT: usize = 10 * 1_024 * 1_024;
-
-    let mut buf: Vec<u8> = vec![];
-    resp.into_reader()
-        .take((BYTES_LIMIT + 1) as u64)
-        .read_to_end(&mut buf)?;
-    if buf.len() > BYTES_LIMIT {
-        return Err(io::Error::new(
-            io::ErrorKind::Other,
-            "response too big for into_bytes",
-        ));
-    }
-
-    Ok(buf)
-}
-
-impl ConfigurableBlockchain for EsploraBlockchain {
-    type Config = super::EsploraBlockchainConfig;
-
-    fn from_config(config: &Self::Config) -> Result<Self, Error> {
-        let mut agent_builder = ureq::AgentBuilder::new();
-
-        if let Some(timeout) = config.timeout {
-            agent_builder = agent_builder.timeout(Duration::from_secs(timeout));
-        }
-
-        if let Some(proxy) = &config.proxy {
-            agent_builder = agent_builder
-                .proxy(Proxy::new(proxy).map_err(|e| Error::Esplora(Box::new(e.into())))?);
-        }
-
-        let mut blockchain = EsploraBlockchain::new(config.base_url.as_str(), config.stop_gap)
-            .with_agent(agent_builder.build());
-
-        if let Some(concurrency) = config.concurrency {
-            blockchain = blockchain.with_concurrency(concurrency);
-        }
-
-        Ok(blockchain)
-    }
-}
-
-impl From<ureq::Error> for EsploraError {
-    fn from(e: ureq::Error) -> Self {
-        match e {
-            ureq::Error::Status(code, _) => EsploraError::HttpResponse(code),
-            e => EsploraError::Ureq(e),
-        }
-    }
-}

src/blockchain/mod.rs

@@ -27,21 +27,12 @@ use crate::database::BatchDatabase;
 use crate::error::Error;
 use crate::FeeRate;
 
-#[cfg(any(
-    feature = "electrum",
-    feature = "esplora",
-    feature = "compact_filters",
-    feature = "rpc"
-))]
-pub mod any;
-mod script_sync;
+#[cfg(any(feature = "electrum", feature = "esplora"))]
+pub(crate) mod utils;
 
-#[cfg(any(
-    feature = "electrum",
-    feature = "esplora",
-    feature = "compact_filters",
-    feature = "rpc"
-))]
+#[cfg(any(feature = "electrum", feature = "esplora", feature = "compact_filters"))]
+pub mod any;
+#[cfg(any(feature = "electrum", feature = "esplora", feature = "compact_filters"))]
 pub use any::{AnyBlockchain, AnyBlockchainConfig};
 
 #[cfg(feature = "electrum")]
@@ -86,50 +77,28 @@ pub enum Capability {
 
 /// Trait that defines the actions that must be supported by a blockchain backend
 #[maybe_async]
-pub trait Blockchain: WalletSync + GetHeight + GetTx {
+pub trait Blockchain {
     /// Return the set of [`Capability`] supported by this backend
     fn get_capabilities(&self) -> HashSet<Capability>;
-    /// Broadcast a transaction
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error>;
-    /// Estimate the fee rate required to confirm a transaction in a given `target` of blocks
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error>;
-}
 
-/// Trait for getting the current height of the blockchain.
-#[maybe_async]
-pub trait GetHeight {
-    /// Return the current height
-    fn get_height(&self) -> Result<u32, Error>;
-}
-
-#[maybe_async]
-/// Trait for getting a transaction by txid
-pub trait GetTx {
-    /// Fetch a transaction given its txid
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
-}
-
-/// Trait for blockchains that can sync by updating the database directly.
-#[maybe_async]
-pub trait WalletSync {
     /// Setup the backend and populate the internal database for the first time
     ///
-    /// This method is the equivalent of [`Self::wallet_sync`], but it's guaranteed to only be
+    /// This method is the equivalent of [`Blockchain::sync`], but it's guaranteed to only be
     /// called once, at the first [`Wallet::sync`](crate::wallet::Wallet::sync).
     ///
     /// The rationale behind the distinction between `sync` and `setup` is that some custom backends
     /// might need to perform specific actions only the first time they are synced.
     ///
     /// For types that do not have that distinction, only this method can be implemented, since
-    /// [`WalletSync::wallet_sync`] defaults to calling this internally if not overridden.
-    /// Populate the internal database with transactions and UTXOs
-    fn wallet_setup<D: BatchDatabase>(
+    /// [`Blockchain::sync`] defaults to calling this internally if not overridden.
+    fn setup<D: BatchDatabase, P: 'static + Progress>(
         &self,
         database: &mut D,
-        progress_update: Box<dyn Progress>,
+        progress_update: P,
     ) -> Result<(), Error>;
-
-    /// If not overridden, it defaults to calling [`Self::wallet_setup`] internally.
+    /// Populate the internal database with transactions and UTXOs
+    ///
+    /// If not overridden, it defaults to calling [`Blockchain::setup`] internally.
     ///
     /// This method should implement the logic required to iterate over the list of the wallet's
     /// script_pubkeys using [`Database::iter_script_pubkeys`] and look for relevant transactions
@@ -146,13 +115,23 @@ pub trait WalletSync {
     /// [`BatchOperations::set_tx`]: crate::database::BatchOperations::set_tx
     /// [`BatchOperations::set_utxo`]: crate::database::BatchOperations::set_utxo
     /// [`BatchOperations::del_utxo`]: crate::database::BatchOperations::del_utxo
-    fn wallet_sync<D: BatchDatabase>(
+    fn sync<D: BatchDatabase, P: 'static + Progress>(
         &self,
         database: &mut D,
-        progress_update: Box<dyn Progress>,
+        progress_update: P,
     ) -> Result<(), Error> {
-        maybe_await!(self.wallet_setup(database, progress_update))
+        maybe_await!(self.setup(database, progress_update))
     }
+
+    /// Fetch a transaction from the blockchain given its txid
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
+    /// Broadcast a transaction
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error>;
+
+    /// Return the current height
+    fn get_height(&self) -> Result<u32, Error>;
+    /// Estimate the fee rate required to confirm a transaction in a given `target` of blocks
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error>;
 }
 
 /// Trait for [`Blockchain`] types that can be created given a configuration
@@ -167,9 +146,9 @@ pub trait ConfigurableBlockchain: Blockchain + Sized {
 /// Data sent with a progress update over a [`channel`]
 pub type ProgressData = (f32, Option<String>);
 
-/// Trait for types that can receive and process progress updates during [`WalletSync::wallet_sync`] and
-/// [`WalletSync::wallet_setup`]
-pub trait Progress: Send + 'static + core::fmt::Debug {
+/// Trait for types that can receive and process progress updates during [`Blockchain::sync`] and
+/// [`Blockchain::setup`]
+pub trait Progress: Send {
     /// Send a new progress update
     ///
     /// The `progress` value should be in the range 0.0 - 100.0, and the `message` value is an
@@ -194,7 +173,7 @@ impl Progress for Sender<ProgressData> {
 }
 
 /// Type that implements [`Progress`] and drops every update received
-#[derive(Clone, Copy, Default, Debug)]
+#[derive(Clone, Copy)]
 pub struct NoopProgress;
 
 /// Create a new instance of [`NoopProgress`]
@@ -209,10 +188,10 @@ impl Progress for NoopProgress {
 }
 
 /// Type that implements [`Progress`] and logs at level `INFO` every update received
-#[derive(Clone, Copy, Default, Debug)]
+#[derive(Clone, Copy)]
 pub struct LogProgress;
 
-/// Create a new instance of [`LogProgress`]
+/// Create a nwe instance of [`LogProgress`]
 pub fn log_progress() -> LogProgress {
     LogProgress
 }
@@ -235,44 +214,33 @@ impl<T: Blockchain> Blockchain for Arc<T> {
         maybe_await!(self.deref().get_capabilities())
     }
 
+    fn setup<D: BatchDatabase, P: 'static + Progress>(
+        &self,
+        database: &mut D,
+        progress_update: P,
+    ) -> Result<(), Error> {
+        maybe_await!(self.deref().setup(database, progress_update))
+    }
+
+    fn sync<D: BatchDatabase, P: 'static + Progress>(
+        &self,
+        database: &mut D,
+        progress_update: P,
+    ) -> Result<(), Error> {
+        maybe_await!(self.deref().sync(database, progress_update))
+    }
+
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        maybe_await!(self.deref().get_tx(txid))
+    }
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
         maybe_await!(self.deref().broadcast(tx))
     }
 
+    fn get_height(&self) -> Result<u32, Error> {
+        maybe_await!(self.deref().get_height())
+    }
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
         maybe_await!(self.deref().estimate_fee(target))
     }
 }
-
-#[maybe_async]
-impl<T: GetTx> GetTx for Arc<T> {
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        maybe_await!(self.deref().get_tx(txid))
-    }
-}
-
-#[maybe_async]
-impl<T: GetHeight> GetHeight for Arc<T> {
-    fn get_height(&self) -> Result<u32, Error> {
-        maybe_await!(self.deref().get_height())
-    }
-}
-
-#[maybe_async]
-impl<T: WalletSync> WalletSync for Arc<T> {
-    fn wallet_setup<D: BatchDatabase>(
-        &self,
-        database: &mut D,
-        progress_update: Box<dyn Progress>,
-    ) -> Result<(), Error> {
-        maybe_await!(self.deref().wallet_setup(database, progress_update))
-    }
-
-    fn wallet_sync<D: BatchDatabase>(
-        &self,
-        database: &mut D,
-        progress_update: Box<dyn Progress>,
-    ) -> Result<(), Error> {
-        maybe_await!(self.deref().wallet_sync(database, progress_update))
-    }
-}

src/blockchain/rpc.rs

@@ -18,12 +18,10 @@
 //! ## Example
 //!
 //! ```no_run
-//! # use bdk::blockchain::{RpcConfig, RpcBlockchain, ConfigurableBlockchain, rpc::Auth};
+//! # use bdk::blockchain::{RpcConfig, RpcBlockchain, ConfigurableBlockchain};
 //! let config = RpcConfig {
 //!     url: "127.0.0.1:18332".to_string(),
-//!     auth: Auth::Cookie {
-//!         file: "/home/user/.bitcoin/.cookie".into(),
-//!     },
+//!     auth: bitcoincore_rpc::Auth::CookieFile("/home/user/.bitcoin/.cookie".into()),
 //!     network: bdk::bitcoin::Network::Testnet,
 //!     wallet_name: "wallet_name".to_string(),
 //!     skip_blocks: None,
@@ -33,20 +31,20 @@
 
 use crate::bitcoin::consensus::deserialize;
 use crate::bitcoin::{Address, Network, OutPoint, Transaction, TxOut, Txid};
-use crate::blockchain::*;
+use crate::blockchain::{Blockchain, Capability, ConfigurableBlockchain, Progress};
 use crate::database::{BatchDatabase, DatabaseUtils};
-use crate::{BlockTime, Error, FeeRate, KeychainKind, LocalUtxo, TransactionDetails};
+use crate::descriptor::{get_checksum, IntoWalletDescriptor};
+use crate::wallet::utils::SecpCtx;
+use crate::{ConfirmationTime, Error, FeeRate, KeychainKind, LocalUtxo, TransactionDetails};
 use bitcoincore_rpc::json::{
     GetAddressInfoResultLabel, ImportMultiOptions, ImportMultiRequest,
     ImportMultiRequestScriptPubkey, ImportMultiRescanSince,
 };
 use bitcoincore_rpc::jsonrpc::serde_json::Value;
-use bitcoincore_rpc::Auth as RpcAuth;
-use bitcoincore_rpc::{Client, RpcApi};
+use bitcoincore_rpc::{Auth, Client, RpcApi};
 use log::debug;
-use serde::{Deserialize, Serialize};
+use serde::Deserialize;
 use std::collections::{HashMap, HashSet};
-use std::path::PathBuf;
 use std::str::FromStr;
 
 /// The main struct for RPC backend implementing the [crate::blockchain::Blockchain] trait
@@ -66,7 +64,7 @@ pub struct RpcBlockchain {
 }
 
 /// RpcBlockchain configuration options
-#[derive(Debug, Serialize, Deserialize, Clone, PartialEq)]
+#[derive(Debug)]
 pub struct RpcConfig {
     /// The bitcoin node url
     pub url: String,
@@ -74,45 +72,12 @@ pub struct RpcConfig {
     pub auth: Auth,
     /// The network we are using (it will be checked the bitcoin node network matches this)
     pub network: Network,
-    /// The wallet name in the bitcoin node, consider using [crate::wallet::wallet_name_from_descriptor] for this
+    /// The wallet name in the bitcoin node, consider using [wallet_name_from_descriptor] for this
     pub wallet_name: String,
     /// Skip this many blocks of the blockchain at the first rescan, if None the rescan is done from the genesis block
     pub skip_blocks: Option<u32>,
 }
 
-/// This struct is equivalent to [bitcoincore_rpc::Auth] but it implements [serde::Serialize]
-/// To be removed once upstream equivalent is implementing Serialize (json serialization format
-/// should be the same), see [rust-bitcoincore-rpc/pull/181](https://github.com/rust-bitcoin/rust-bitcoincore-rpc/pull/181)
-#[derive(Clone, Debug, Hash, Eq, PartialEq, Ord, PartialOrd, Serialize, Deserialize)]
-#[serde(rename_all = "snake_case")]
-#[serde(untagged)]
-pub enum Auth {
-    /// None authentication
-    None,
-    /// Authentication with username and password, usually [Auth::Cookie] should be preferred
-    UserPass {
-        /// Username
-        username: String,
-        /// Password
-        password: String,
-    },
-    /// Authentication with a cookie file
-    Cookie {
-        /// Cookie file
-        file: PathBuf,
-    },
-}
-
-impl From<Auth> for RpcAuth {
-    fn from(auth: Auth) -> Self {
-        match auth {
-            Auth::None => RpcAuth::None,
-            Auth::UserPass { username, password } => RpcAuth::UserPass(username, password),
-            Auth::Cookie { file } => RpcAuth::CookieFile(file),
-        }
-    }
-}
-
 impl RpcBlockchain {
     fn get_node_synced_height(&self) -> Result<u32, Error> {
         let info = self.client.get_address_info(&self._storage_address)?;
@@ -139,39 +104,10 @@ impl Blockchain for RpcBlockchain {
         self.capabilities.clone()
     }
 
-    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        Ok(self.client.send_raw_transaction(tx).map(|_| ())?)
-    }
-
-    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
-        let sat_per_kb = self
-            .client
-            .estimate_smart_fee(target as u16, None)?
-            .fee_rate
-            .ok_or(Error::FeeRateUnavailable)?
-            .as_sat() as f64;
-
-        Ok(FeeRate::from_sat_per_vb((sat_per_kb / 1000f64) as f32))
-    }
-}
-
-impl GetTx for RpcBlockchain {
-    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(Some(self.client.get_raw_transaction(txid, None)?))
-    }
-}
-
-impl GetHeight for RpcBlockchain {
-    fn get_height(&self) -> Result<u32, Error> {
-        Ok(self.client.get_blockchain_info().map(|i| i.blocks as u32)?)
-    }
-}
-
-impl WalletSync for RpcBlockchain {
-    fn wallet_setup<D: BatchDatabase>(
+    fn setup<D: BatchDatabase, P: 'static + Progress>(
         &self,
         database: &mut D,
-        progress_update: Box<dyn Progress>,
+        progress_update: P,
     ) -> Result<(), Error> {
         let mut scripts_pubkeys = database.iter_script_pubkeys(Some(KeychainKind::External))?;
         scripts_pubkeys.extend(database.iter_script_pubkeys(Some(KeychainKind::Internal))?);
@@ -183,7 +119,7 @@ impl WalletSync for RpcBlockchain {
             .iter()
             .map(|s| ImportMultiRequest {
                 timestamp: ImportMultiRescanSince::Timestamp(0),
-                script_pubkey: Some(ImportMultiRequestScriptPubkey::Script(s)),
+                script_pubkey: Some(ImportMultiRequestScriptPubkey::Script(&s)),
                 watchonly: Some(true),
                 ..Default::default()
             })
@@ -196,33 +132,30 @@ impl WalletSync for RpcBlockchain {
         //TODO maybe convenient using import_descriptor for compatible descriptor and import_multi as fallback
         self.client.import_multi(&requests, Some(&options))?;
 
-        loop {
-            let current_height = self.get_height()?;
+        let current_height = self.get_height()?;
 
-            // min because block invalidate may cause height to go down
-            let node_synced = self.get_node_synced_height()?.min(current_height);
+        // min because block invalidate may cause height to go down
+        let node_synced = self.get_node_synced_height()?.min(current_height);
 
-            let sync_up_to = node_synced.saturating_add(10_000).min(current_height);
+        //TODO call rescan in chunks (updating node_synced_height) so that in case of
+        // interruption work can be partially recovered
+        debug!(
+            "rescan_blockchain from:{} to:{}",
+            node_synced, current_height
+        );
+        self.client
+            .rescan_blockchain(Some(node_synced as usize), Some(current_height as usize))?;
+        progress_update.update(1.0, None)?;
 
-            debug!("rescan_blockchain from:{} to:{}", node_synced, sync_up_to);
-            self.client
-                .rescan_blockchain(Some(node_synced as usize), Some(sync_up_to as usize))?;
-            progress_update.update((sync_up_to as f32) / (current_height as f32), None)?;
+        self.set_node_synced_height(current_height)?;
 
-            self.set_node_synced_height(sync_up_to)?;
-
-            if sync_up_to == current_height {
-                break;
-            }
-        }
-
-        self.wallet_sync(database, progress_update)
+        self.sync(database, progress_update)
     }
 
-    fn wallet_sync<D: BatchDatabase>(
+    fn sync<D: BatchDatabase, P: 'static + Progress>(
         &self,
         db: &mut D,
-        _progress_update: Box<dyn Progress>,
+        _progress_update: P,
     ) -> Result<(), Error> {
         let mut indexes = HashMap::new();
         for keykind in &[KeychainKind::External, KeychainKind::Internal] {
@@ -257,7 +190,7 @@ impl WalletSync for RpcBlockchain {
             list_txs_ids.insert(txid);
             if let Some(mut known_tx) = known_txs.get_mut(&txid) {
                 let confirmation_time =
-                    BlockTime::new(tx_result.info.blockheight, tx_result.info.blocktime);
+                    ConfirmationTime::new(tx_result.info.blockheight, tx_result.info.blocktime);
                 if confirmation_time != known_tx.confirmation_time {
                     // reorg may change tx height
                     debug!(
@@ -265,7 +198,7 @@ impl WalletSync for RpcBlockchain {
                         txid, confirmation_time
                     );
                     known_tx.confirmation_time = confirmation_time;
-                    db.set_tx(known_tx)?;
+                    db.set_tx(&known_tx)?;
                 }
             } else {
                 //TODO check there is already the raw tx in db?
@@ -286,22 +219,21 @@ impl WalletSync for RpcBlockchain {
 
                 for input in tx.input.iter() {
                     if let Some(previous_output) = db.get_previous_output(&input.previous_output)? {
-                        if db.is_mine(&previous_output.script_pubkey)? {
-                            sent += previous_output.value;
-                        }
+                        sent += previous_output.value;
                     }
                 }
 
                 let td = TransactionDetails {
                     transaction: Some(tx),
                     txid: tx_result.info.txid,
-                    confirmation_time: BlockTime::new(
+                    confirmation_time: ConfirmationTime::new(
                         tx_result.info.blockheight,
                         tx_result.info.blocktime,
                     ),
                     received,
                     sent,
                     fee: tx_result.fee.map(|f| f.as_sat().abs() as u64),
+                    verified: true,
                 };
                 debug!(
                     "saving tx: {} tx_result.fee:{:?} td.fees:{:?}",
@@ -318,24 +250,22 @@ impl WalletSync for RpcBlockchain {
             }
         }
 
-        // Filter out trasactions that are for script pubkeys that aren't in this wallet.
-        let current_utxos = current_utxo
+        let current_utxos: HashSet<_> = current_utxo
             .into_iter()
-            .filter_map(
-                |u| match db.get_path_from_script_pubkey(&u.script_pub_key) {
-                    Err(e) => Some(Err(e)),
-                    Ok(None) => None,
-                    Ok(Some(path)) => Some(Ok(LocalUtxo {
-                        outpoint: OutPoint::new(u.txid, u.vout),
-                        keychain: path.0,
-                        txout: TxOut {
-                            value: u.amount.as_sat(),
-                            script_pubkey: u.script_pub_key,
-                        },
-                    })),
-                },
-            )
-            .collect::<Result<HashSet<_>, Error>>()?;
+            .map(|u| {
+                Ok(LocalUtxo {
+                    outpoint: OutPoint::new(u.txid, u.vout),
+                    keychain: db
+                        .get_path_from_script_pubkey(&u.script_pub_key)?
+                        .ok_or(Error::TransactionNotFound)?
+                        .0,
+                    txout: TxOut {
+                        value: u.amount.as_sat(),
+                        script_pubkey: u.script_pub_key,
+                    },
+                })
+            })
+            .collect::<Result<_, Error>>()?;
 
         let spent: HashSet<_> = known_utxos.difference(&current_utxos).collect();
         for s in spent {
@@ -355,6 +285,29 @@ impl WalletSync for RpcBlockchain {
 
         Ok(())
     }
+
+    fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
+        Ok(Some(self.client.get_raw_transaction(txid, None)?))
+    }
+
+    fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
+        Ok(self.client.send_raw_transaction(tx).map(|_| ())?)
+    }
+
+    fn get_height(&self) -> Result<u32, Error> {
+        Ok(self.client.get_blockchain_info().map(|i| i.blocks as u32)?)
+    }
+
+    fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
+        let sat_per_kb = self
+            .client
+            .estimate_smart_fee(target as u16, None)?
+            .fee_rate
+            .ok_or(Error::FeeRateUnavailable)?
+            .as_sat() as f64;
+
+        Ok(FeeRate::from_sat_per_vb((sat_per_kb / 1000f64) as f32))
+    }
 }
 
 impl ConfigurableBlockchain for RpcBlockchain {
@@ -367,7 +320,7 @@ impl ConfigurableBlockchain for RpcBlockchain {
         let wallet_url = format!("{}/wallet/{}", config.url, &wallet_name);
         debug!("connecting to {} auth:{:?}", wallet_url, config.auth);
 
-        let client = Client::new(wallet_url.as_str(), config.auth.clone().into())?;
+        let client = Client::new(wallet_url, config.auth.clone())?;
         let loaded_wallets = client.list_wallets()?;
         if loaded_wallets.contains(&wallet_name) {
             debug!("wallet already loaded {:?}", wallet_name);
@@ -422,6 +375,35 @@ impl ConfigurableBlockchain for RpcBlockchain {
     }
 }
 
+/// Deterministically generate a unique name given the descriptors defining the wallet
+pub fn wallet_name_from_descriptor<T>(
+    descriptor: T,
+    change_descriptor: Option<T>,
+    network: Network,
+    secp: &SecpCtx,
+) -> Result<String, Error>
+where
+    T: IntoWalletDescriptor,
+{
+    //TODO check descriptors contains only public keys
+    let descriptor = descriptor
+        .into_wallet_descriptor(&secp, network)?
+        .0
+        .to_string();
+    let mut wallet_name = get_checksum(&descriptor[..descriptor.find('#').unwrap()])?;
+    if let Some(change_descriptor) = change_descriptor {
+        let change_descriptor = change_descriptor
+            .into_wallet_descriptor(&secp, network)?
+            .0
+            .to_string();
+        wallet_name.push_str(
+            get_checksum(&change_descriptor[..change_descriptor.find('#').unwrap()])?.as_str(),
+        );
+    }
+
+    Ok(wallet_name)
+}
+
 /// return the wallets available in default wallet directory
 //TODO use bitcoincore_rpc method when PR #179 lands
 fn list_wallet_dir(client: &Client) -> Result<Vec<String>, Error> {
@@ -445,7 +427,7 @@ crate::bdk_blockchain_tests! {
     fn test_instance(test_client: &TestClient) -> RpcBlockchain {
         let config = RpcConfig {
             url: test_client.bitcoind.rpc_url(),
-            auth: Auth::Cookie { file: test_client.bitcoind.params.cookie_file.clone() },
+            auth: Auth::CookieFile(test_client.bitcoind.params.cookie_file.clone()),
             network: Network::Regtest,
             wallet_name: format!("client-wallet-test-{:?}", std::time::SystemTime::now() ),
             skip_blocks: None,

src/blockchain/script_sync.rs

@@ -1,407 +0,0 @@
-/*!
-This models a how a sync happens where you have a server that you send your script pubkeys to and it
-returns associated transactions i.e. electrum.
-*/
-#![allow(dead_code)]
-use crate::{
-    database::{BatchDatabase, BatchOperations, DatabaseUtils},
-    wallet::time::Instant,
-    BlockTime, Error, KeychainKind, LocalUtxo, TransactionDetails,
-};
-use bitcoin::{OutPoint, Script, Transaction, TxOut, Txid};
-use log::*;
-use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque};
-
-/// A request for on-chain information
-pub enum Request<'a, D: BatchDatabase> {
-    /// A request for transactions related to script pubkeys.
-    Script(ScriptReq<'a, D>),
-    /// A request for confirmation times for some transactions.
-    Conftime(ConftimeReq<'a, D>),
-    /// A request for full transaction details of some transactions.
-    Tx(TxReq<'a, D>),
-    /// Requests are finished here's a batch database update to reflect data gathered.
-    Finish(D::Batch),
-}
-
-/// starts a sync
-pub fn start<D: BatchDatabase>(db: &D, stop_gap: usize) -> Result<Request<'_, D>, Error> {
-    use rand::seq::SliceRandom;
-    let mut keychains = vec![KeychainKind::Internal, KeychainKind::External];
-    // shuffling improve privacy, the server doesn't know my first request is from my internal or external addresses
-    keychains.shuffle(&mut rand::thread_rng());
-    let keychain = keychains.pop().unwrap();
-    let scripts_needed = db
-        .iter_script_pubkeys(Some(keychain))?
-        .into_iter()
-        .collect();
-    let state = State::new(db);
-
-    Ok(Request::Script(ScriptReq {
-        state,
-        scripts_needed,
-        script_index: 0,
-        stop_gap,
-        keychain,
-        next_keychains: keychains,
-    }))
-}
-
-pub struct ScriptReq<'a, D: BatchDatabase> {
-    state: State<'a, D>,
-    script_index: usize,
-    scripts_needed: VecDeque<Script>,
-    stop_gap: usize,
-    keychain: KeychainKind,
-    next_keychains: Vec<KeychainKind>,
-}
-
-/// The sync starts by returning script pubkeys we are interested in.
-impl<'a, D: BatchDatabase> ScriptReq<'a, D> {
-    pub fn request(&self) -> impl Iterator<Item = &Script> + Clone {
-        self.scripts_needed.iter()
-    }
-
-    pub fn satisfy(
-        mut self,
-        // we want to know the txids assoiciated with the script and their height
-        txids: Vec<Vec<(Txid, Option<u32>)>>,
-    ) -> Result<Request<'a, D>, Error> {
-        for (txid_list, script) in txids.iter().zip(self.scripts_needed.iter()) {
-            debug!(
-                "found {} transactions for script pubkey {}",
-                txid_list.len(),
-                script
-            );
-            if !txid_list.is_empty() {
-                // the address is active
-                self.state
-                    .last_active_index
-                    .insert(self.keychain, self.script_index);
-            }
-
-            for (txid, height) in txid_list {
-                // have we seen this txid already?
-                match self.state.db.get_tx(txid, true)? {
-                    Some(mut details) => {
-                        let old_height = details.confirmation_time.as_ref().map(|x| x.height);
-                        match (old_height, height) {
-                            (None, Some(_)) => {
-                                // It looks like the tx has confirmed since we last saw it -- we
-                                // need to know the confirmation time.
-                                self.state.tx_missing_conftime.insert(*txid, details);
-                            }
-                            (Some(old_height), Some(new_height)) if old_height != *new_height => {
-                                // The height of the tx has changed !? -- It's a reorg get the new confirmation time.
-                                self.state.tx_missing_conftime.insert(*txid, details);
-                            }
-                            (Some(_), None) => {
-                                // A re-org where the tx is not in the chain anymore.
-                                details.confirmation_time = None;
-                                self.state.finished_txs.push(details);
-                            }
-                            _ => self.state.finished_txs.push(details),
-                        }
-                    }
-                    None => {
-                        // we've never seen it let's get the whole thing
-                        self.state.tx_needed.insert(*txid);
-                    }
-                };
-            }
-
-            self.script_index += 1;
-        }
-
-        for _ in txids {
-            self.scripts_needed.pop_front();
-        }
-
-        let last_active_index = self
-            .state
-            .last_active_index
-            .get(&self.keychain)
-            .map(|x| x + 1)
-            .unwrap_or(0); // so no addresses active maps to 0
-
-        Ok(
-            if self.script_index > last_active_index + self.stop_gap
-                || self.scripts_needed.is_empty()
-            {
-                debug!(
-                    "finished scanning for transactions for keychain {:?} at index {}",
-                    self.keychain, last_active_index
-                );
-                // we're done here -- check if we need to do the next keychain
-                if let Some(keychain) = self.next_keychains.pop() {
-                    self.keychain = keychain;
-                    self.script_index = 0;
-                    self.scripts_needed = self
-                        .state
-                        .db
-                        .iter_script_pubkeys(Some(keychain))?
-                        .into_iter()
-                        .collect();
-                    Request::Script(self)
-                } else {
-                    Request::Tx(TxReq { state: self.state })
-                }
-            } else {
-                Request::Script(self)
-            },
-        )
-    }
-}
-
-/// Then we get full transactions
-pub struct TxReq<'a, D> {
-    state: State<'a, D>,
-}
-
-impl<'a, D: BatchDatabase> TxReq<'a, D> {
-    pub fn request(&self) -> impl Iterator<Item = &Txid> + Clone {
-        self.state.tx_needed.iter()
-    }
-
-    pub fn satisfy(
-        mut self,
-        tx_details: Vec<(Vec<Option<TxOut>>, Transaction)>,
-    ) -> Result<Request<'a, D>, Error> {
-        let tx_details: Vec<TransactionDetails> = tx_details
-            .into_iter()
-            .zip(self.state.tx_needed.iter())
-            .map(|((vout, tx), txid)| {
-                debug!("found tx_details for {}", txid);
-                assert_eq!(tx.txid(), *txid);
-                let mut sent: u64 = 0;
-                let mut received: u64 = 0;
-                let mut inputs_sum: u64 = 0;
-                let mut outputs_sum: u64 = 0;
-
-                for (txout, (_input_index, input)) in
-                    vout.into_iter().zip(tx.input.iter().enumerate())
-                {
-                    let txout = match txout {
-                        Some(txout) => txout,
-                        None => {
-                            // skip coinbase inputs
-                            debug_assert!(
-                                input.previous_output.is_null(),
-                                "prevout should only be missing for coinbase"
-                            );
-                            continue;
-                        }
-                    };
-                    // Verify this input if requested via feature flag
-                    #[cfg(feature = "verify")]
-                    {
-                        use crate::wallet::verify::VerifyError;
-                        let serialized_tx = bitcoin::consensus::serialize(&tx);
-                        bitcoinconsensus::verify(
-                            txout.script_pubkey.to_bytes().as_ref(),
-                            txout.value,
-                            &serialized_tx,
-                            _input_index,
-                        )
-                        .map_err(VerifyError::from)?;
-                    }
-                    inputs_sum += txout.value;
-                    if self.state.db.is_mine(&txout.script_pubkey)? {
-                        sent += txout.value;
-                    }
-                }
-
-                for out in &tx.output {
-                    outputs_sum += out.value;
-                    if self.state.db.is_mine(&out.script_pubkey)? {
-                        received += out.value;
-                    }
-                }
-                // we need to saturating sub since we want coinbase txs to map to 0 fee and
-                // this subtraction will be negative for coinbase txs.
-                let fee = inputs_sum.saturating_sub(outputs_sum);
-                Result::<_, Error>::Ok(TransactionDetails {
-                    txid: *txid,
-                    transaction: Some(tx),
-                    received,
-                    sent,
-                    // we're going to fill this in later
-                    confirmation_time: None,
-                    fee: Some(fee),
-                })
-            })
-            .collect::<Result<Vec<_>, _>>()?;
-
-        for tx_detail in tx_details {
-            self.state.tx_needed.remove(&tx_detail.txid);
-            self.state
-                .tx_missing_conftime
-                .insert(tx_detail.txid, tx_detail);
-        }
-
-        if !self.state.tx_needed.is_empty() {
-            Ok(Request::Tx(self))
-        } else {
-            Ok(Request::Conftime(ConftimeReq { state: self.state }))
-        }
-    }
-}
-
-/// Final step is to get confirmation times
-pub struct ConftimeReq<'a, D> {
-    state: State<'a, D>,
-}
-
-impl<'a, D: BatchDatabase> ConftimeReq<'a, D> {
-    pub fn request(&self) -> impl Iterator<Item = &Txid> + Clone {
-        self.state.tx_missing_conftime.keys()
-    }
-
-    pub fn satisfy(
-        mut self,
-        confirmation_times: Vec<Option<BlockTime>>,
-    ) -> Result<Request<'a, D>, Error> {
-        let conftime_needed = self
-            .request()
-            .cloned()
-            .take(confirmation_times.len())
-            .collect::<Vec<_>>();
-        for (confirmation_time, txid) in confirmation_times.into_iter().zip(conftime_needed.iter())
-        {
-            debug!("confirmation time for {} was {:?}", txid, confirmation_time);
-            if let Some(mut tx_details) = self.state.tx_missing_conftime.remove(txid) {
-                tx_details.confirmation_time = confirmation_time;
-                self.state.finished_txs.push(tx_details);
-            }
-        }
-
-        if self.state.tx_missing_conftime.is_empty() {
-            Ok(Request::Finish(self.state.into_db_update()?))
-        } else {
-            Ok(Request::Conftime(self))
-        }
-    }
-}
-
-struct State<'a, D> {
-    db: &'a D,
-    last_active_index: HashMap<KeychainKind, usize>,
-    /// Transactions where we need to get the full details
-    tx_needed: BTreeSet<Txid>,
-    /// Transacitions that we know everything about
-    finished_txs: Vec<TransactionDetails>,
-    /// Transactions that discovered conftimes should be inserted into
-    tx_missing_conftime: BTreeMap<Txid, TransactionDetails>,
-    /// The start of the sync
-    start_time: Instant,
-}
-
-impl<'a, D: BatchDatabase> State<'a, D> {
-    fn new(db: &'a D) -> Self {
-        State {
-            db,
-            last_active_index: HashMap::default(),
-            finished_txs: vec![],
-            tx_needed: BTreeSet::default(),
-            tx_missing_conftime: BTreeMap::default(),
-            start_time: Instant::new(),
-        }
-    }
-    fn into_db_update(self) -> Result<D::Batch, Error> {
-        debug_assert!(self.tx_needed.is_empty() && self.tx_missing_conftime.is_empty());
-        let existing_txs = self.db.iter_txs(false)?;
-        let existing_txids: HashSet<Txid> = existing_txs.iter().map(|tx| tx.txid).collect();
-        let finished_txs = make_txs_consistent(&self.finished_txs);
-        let observed_txids: HashSet<Txid> = finished_txs.iter().map(|tx| tx.txid).collect();
-        let txids_to_delete = existing_txids.difference(&observed_txids);
-        let mut batch = self.db.begin_batch();
-
-        // Delete old txs that no longer exist
-        for txid in txids_to_delete {
-            if let Some(raw_tx) = self.db.get_raw_tx(txid)? {
-                for i in 0..raw_tx.output.len() {
-                    // Also delete any utxos from the txs that no longer exist.
-                    let _ = batch.del_utxo(&OutPoint {
-                        txid: *txid,
-                        vout: i as u32,
-                    })?;
-                }
-            } else {
-                unreachable!("we should always have the raw tx");
-            }
-            batch.del_tx(txid, true)?;
-        }
-
-        // Set every tx we observed
-        for finished_tx in &finished_txs {
-            let tx = finished_tx
-                .transaction
-                .as_ref()
-                .expect("transaction will always be present here");
-            for (i, output) in tx.output.iter().enumerate() {
-                if let Some((keychain, _)) =
-                    self.db.get_path_from_script_pubkey(&output.script_pubkey)?
-                {
-                    // add utxos we own from the new transactions we've seen.
-                    batch.set_utxo(&LocalUtxo {
-                        outpoint: OutPoint {
-                            txid: finished_tx.txid,
-                            vout: i as u32,
-                        },
-                        txout: output.clone(),
-                        keychain,
-                    })?;
-                }
-            }
-            batch.set_tx(finished_tx)?;
-        }
-
-        // we don't do this in the loop above since we may want to delete some of the utxos we
-        // just added in case there are new tranasactions that spend form each other.
-        for finished_tx in &finished_txs {
-            let tx = finished_tx
-                .transaction
-                .as_ref()
-                .expect("transaction will always be present here");
-            for input in &tx.input {
-                // Delete any spent utxos
-                batch.del_utxo(&input.previous_output)?;
-            }
-        }
-
-        for (keychain, last_active_index) in self.last_active_index {
-            batch.set_last_index(keychain, last_active_index as u32)?;
-        }
-
-        info!(
-            "finished setup, elapsed {:?}ms",
-            self.start_time.elapsed().as_millis()
-        );
-        Ok(batch)
-    }
-}
-
-/// Remove conflicting transactions -- tie breaking them by fee.
-fn make_txs_consistent(txs: &[TransactionDetails]) -> Vec<&TransactionDetails> {
-    let mut utxo_index: HashMap<OutPoint, &TransactionDetails> = HashMap::default();
-    for tx in txs {
-        for input in &tx.transaction.as_ref().unwrap().input {
-            utxo_index
-                .entry(input.previous_output)
-                .and_modify(|existing| match (tx.fee, existing.fee) {
-                    (Some(fee), Some(existing_fee)) if fee > existing_fee => *existing = tx,
-                    (Some(_), None) => *existing = tx,
-                    _ => { /* leave it the same */ }
-                })
-                .or_insert(tx);
-        }
-    }
-
-    utxo_index
-        .into_iter()
-        .map(|(_, tx)| (tx.txid, tx))
-        .collect::<HashMap<_, _>>()
-        .into_iter()
-        .map(|(_, tx)| tx)
-        .collect()
-}

src/blockchain/utils.rs

@@ -0,0 +1,388 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use std::collections::{HashMap, HashSet};
+
+#[allow(unused_imports)]
+use log::{debug, error, info, trace};
+use rand::seq::SliceRandom;
+use rand::thread_rng;
+
+use bitcoin::{BlockHeader, OutPoint, Script, Transaction, Txid};
+
+use super::*;
+use crate::database::{BatchDatabase, BatchOperations, DatabaseUtils};
+use crate::error::Error;
+use crate::types::{ConfirmationTime, KeychainKind, LocalUtxo, TransactionDetails};
+use crate::wallet::time::Instant;
+use crate::wallet::utils::ChunksIterator;
+
+#[derive(Debug)]
+pub struct ElsGetHistoryRes {
+    pub height: i32,
+    pub tx_hash: Txid,
+}
+
+/// Implements the synchronization logic for an Electrum-like client.
+#[maybe_async]
+pub trait ElectrumLikeSync {
+    fn els_batch_script_get_history<'s, I: IntoIterator<Item = &'s Script> + Clone>(
+        &self,
+        scripts: I,
+    ) -> Result<Vec<Vec<ElsGetHistoryRes>>, Error>;
+
+    fn els_batch_transaction_get<'s, I: IntoIterator<Item = &'s Txid> + Clone>(
+        &self,
+        txids: I,
+    ) -> Result<Vec<Transaction>, Error>;
+
+    fn els_batch_block_header<I: IntoIterator<Item = u32> + Clone>(
+        &self,
+        heights: I,
+    ) -> Result<Vec<BlockHeader>, Error>;
+
+    // Provided methods down here...
+
+    fn electrum_like_setup<D: BatchDatabase, P: Progress>(
+        &self,
+        stop_gap: usize,
+        db: &mut D,
+        _progress_update: P,
+    ) -> Result<(), Error> {
+        // TODO: progress
+        let start = Instant::new();
+        debug!("start setup");
+
+        let chunk_size = stop_gap;
+
+        let mut history_txs_id = HashSet::new();
+        let mut txid_height = HashMap::new();
+        let mut max_indexes = HashMap::new();
+
+        let mut wallet_chains = vec![KeychainKind::Internal, KeychainKind::External];
+        // shuffling improve privacy, the server doesn't know my first request is from my internal or external addresses
+        wallet_chains.shuffle(&mut thread_rng());
+        // download history of our internal and external script_pubkeys
+        for keychain in wallet_chains.iter() {
+            let script_iter = db.iter_script_pubkeys(Some(*keychain))?.into_iter();
+
+            for (i, chunk) in ChunksIterator::new(script_iter, stop_gap).enumerate() {
+                // TODO if i == last, should create another chunk of addresses in db
+                let call_result: Vec<Vec<ElsGetHistoryRes>> =
+                    maybe_await!(self.els_batch_script_get_history(chunk.iter()))?;
+                let max_index = call_result
+                    .iter()
+                    .enumerate()
+                    .filter_map(|(i, v)| v.first().map(|_| i as u32))
+                    .max();
+                if let Some(max) = max_index {
+                    max_indexes.insert(keychain, max + (i * chunk_size) as u32);
+                }
+                let flattened: Vec<ElsGetHistoryRes> = call_result.into_iter().flatten().collect();
+                debug!("#{} of {:?} results:{}", i, keychain, flattened.len());
+                if flattened.is_empty() {
+                    // Didn't find anything in the last `stop_gap` script_pubkeys, breaking
+                    break;
+                }
+
+                for el in flattened {
+                    // el.height = -1 means unconfirmed with unconfirmed parents
+                    // el.height =  0 means unconfirmed with confirmed parents
+                    // but we treat those tx the same
+                    if el.height <= 0 {
+                        txid_height.insert(el.tx_hash, None);
+                    } else {
+                        txid_height.insert(el.tx_hash, Some(el.height as u32));
+                    }
+                    history_txs_id.insert(el.tx_hash);
+                }
+            }
+        }
+
+        // saving max indexes
+        info!("max indexes are: {:?}", max_indexes);
+        for keychain in wallet_chains.iter() {
+            if let Some(index) = max_indexes.get(keychain) {
+                db.set_last_index(*keychain, *index)?;
+            }
+        }
+
+        // get db status
+        let txs_details_in_db: HashMap<Txid, TransactionDetails> = db
+            .iter_txs(false)?
+            .into_iter()
+            .map(|tx| (tx.txid, tx))
+            .collect();
+        let txs_raw_in_db: HashMap<Txid, Transaction> = db
+            .iter_raw_txs()?
+            .into_iter()
+            .map(|tx| (tx.txid(), tx))
+            .collect();
+        let utxos_deps = utxos_deps(db, &txs_raw_in_db)?;
+
+        // download new txs and headers
+        let new_txs = maybe_await!(self.download_and_save_needed_raw_txs(
+            &history_txs_id,
+            &txs_raw_in_db,
+            chunk_size,
+            db
+        ))?;
+        let new_timestamps = maybe_await!(self.download_needed_headers(
+            &txid_height,
+            &txs_details_in_db,
+            chunk_size
+        ))?;
+
+        let mut batch = db.begin_batch();
+
+        // save any tx details not in db but in history_txs_id or with different height/timestamp
+        for txid in history_txs_id.iter() {
+            let height = txid_height.get(txid).cloned().flatten();
+            let timestamp = new_timestamps.get(txid).cloned();
+            if let Some(tx_details) = txs_details_in_db.get(txid) {
+                // check if tx height matches, otherwise updates it. timestamp is not in the if clause
+                // because we are not asking headers for confirmed tx we know about
+                if tx_details.confirmation_time.as_ref().map(|c| c.height) != height {
+                    let confirmation_time = ConfirmationTime::new(height, timestamp);
+                    let mut new_tx_details = tx_details.clone();
+                    new_tx_details.confirmation_time = confirmation_time;
+                    batch.set_tx(&new_tx_details)?;
+                }
+            } else {
+                save_transaction_details_and_utxos(
+                    txid,
+                    db,
+                    timestamp,
+                    height,
+                    &mut batch,
+                    &utxos_deps,
+                )?;
+            }
+        }
+
+        // remove any tx details in db but not in history_txs_id
+        for txid in txs_details_in_db.keys() {
+            if !history_txs_id.contains(txid) {
+                batch.del_tx(txid, false)?;
+            }
+        }
+
+        // remove any spent utxo
+        for new_tx in new_txs.iter() {
+            for input in new_tx.input.iter() {
+                batch.del_utxo(&input.previous_output)?;
+            }
+        }
+
+        db.commit_batch(batch)?;
+        info!("finish setup, elapsed {:?}ms", start.elapsed().as_millis());
+
+        Ok(())
+    }
+
+    /// download txs identified by `history_txs_id` and theirs previous outputs if not already present in db
+    fn download_and_save_needed_raw_txs<D: BatchDatabase>(
+        &self,
+        history_txs_id: &HashSet<Txid>,
+        txs_raw_in_db: &HashMap<Txid, Transaction>,
+        chunk_size: usize,
+        db: &mut D,
+    ) -> Result<Vec<Transaction>, Error> {
+        let mut txs_downloaded = vec![];
+        let txids_raw_in_db: HashSet<Txid> = txs_raw_in_db.keys().cloned().collect();
+        let txids_to_download: Vec<&Txid> = history_txs_id.difference(&txids_raw_in_db).collect();
+        if !txids_to_download.is_empty() {
+            info!("got {} txs to download", txids_to_download.len());
+            txs_downloaded.extend(maybe_await!(self.download_and_save_in_chunks(
+                txids_to_download,
+                chunk_size,
+                db,
+            ))?);
+            let mut prev_txids = HashSet::new();
+            let mut txids_downloaded = HashSet::new();
+            for tx in txs_downloaded.iter() {
+                txids_downloaded.insert(tx.txid());
+                // add every previous input tx, but skip coinbase
+                for input in tx.input.iter().filter(|i| !i.previous_output.is_null()) {
+                    prev_txids.insert(input.previous_output.txid);
+                }
+            }
+            let already_present: HashSet<Txid> =
+                txids_downloaded.union(&txids_raw_in_db).cloned().collect();
+            let prev_txs_to_download: Vec<&Txid> =
+                prev_txids.difference(&already_present).collect();
+            info!("{} previous txs to download", prev_txs_to_download.len());
+            txs_downloaded.extend(maybe_await!(self.download_and_save_in_chunks(
+                prev_txs_to_download,
+                chunk_size,
+                db,
+            ))?);
+        }
+
+        Ok(txs_downloaded)
+    }
+
+    /// download headers at heights in `txid_height` if tx details not already present, returns a map Txid -> timestamp
+    fn download_needed_headers(
+        &self,
+        txid_height: &HashMap<Txid, Option<u32>>,
+        txs_details_in_db: &HashMap<Txid, TransactionDetails>,
+        chunk_size: usize,
+    ) -> Result<HashMap<Txid, u64>, Error> {
+        let mut txid_timestamp = HashMap::new();
+        let txid_in_db_with_conf: HashSet<_> = txs_details_in_db
+            .values()
+            .filter_map(|details| details.confirmation_time.as_ref().map(|_| details.txid))
+            .collect();
+        let needed_txid_height: HashMap<&Txid, u32> = txid_height
+            .iter()
+            .filter(|(t, _)| !txid_in_db_with_conf.contains(*t))
+            .filter_map(|(t, o)| o.map(|h| (t, h)))
+            .collect();
+        let needed_heights: HashSet<u32> = needed_txid_height.values().cloned().collect();
+        if !needed_heights.is_empty() {
+            info!("{} headers to download for timestamp", needed_heights.len());
+            let mut height_timestamp: HashMap<u32, u64> = HashMap::new();
+            for chunk in ChunksIterator::new(needed_heights.into_iter(), chunk_size) {
+                let call_result: Vec<BlockHeader> =
+                    maybe_await!(self.els_batch_block_header(chunk.clone()))?;
+                height_timestamp.extend(
+                    chunk
+                        .into_iter()
+                        .zip(call_result.iter().map(|h| h.time as u64)),
+                );
+            }
+            for (txid, height) in needed_txid_height {
+                let timestamp = height_timestamp
+                    .get(&height)
+                    .ok_or_else(|| Error::Generic("timestamp missing".to_string()))?;
+                txid_timestamp.insert(*txid, *timestamp);
+            }
+        }
+
+        Ok(txid_timestamp)
+    }
+
+    fn download_and_save_in_chunks<D: BatchDatabase>(
+        &self,
+        to_download: Vec<&Txid>,
+        chunk_size: usize,
+        db: &mut D,
+    ) -> Result<Vec<Transaction>, Error> {
+        let mut txs_downloaded = vec![];
+        for chunk in ChunksIterator::new(to_download.into_iter(), chunk_size) {
+            let call_result: Vec<Transaction> =
+                maybe_await!(self.els_batch_transaction_get(chunk))?;
+            let mut batch = db.begin_batch();
+            for new_tx in call_result.iter() {
+                batch.set_raw_tx(new_tx)?;
+            }
+            db.commit_batch(batch)?;
+            txs_downloaded.extend(call_result);
+        }
+
+        Ok(txs_downloaded)
+    }
+}
+
+fn save_transaction_details_and_utxos<D: BatchDatabase>(
+    txid: &Txid,
+    db: &mut D,
+    timestamp: Option<u64>,
+    height: Option<u32>,
+    updates: &mut dyn BatchOperations,
+    utxo_deps: &HashMap<OutPoint, OutPoint>,
+) -> Result<(), Error> {
+    let tx = db.get_raw_tx(txid)?.ok_or(Error::TransactionNotFound)?;
+
+    let mut incoming: u64 = 0;
+    let mut outgoing: u64 = 0;
+
+    let mut inputs_sum: u64 = 0;
+    let mut outputs_sum: u64 = 0;
+
+    // look for our own inputs
+    for input in tx.input.iter() {
+        // skip coinbase inputs
+        if input.previous_output.is_null() {
+            continue;
+        }
+
+        // We already downloaded all previous output txs in the previous step
+        if let Some(previous_output) = db.get_previous_output(&input.previous_output)? {
+            inputs_sum += previous_output.value;
+
+            if db.is_mine(&previous_output.script_pubkey)? {
+                outgoing += previous_output.value;
+            }
+        } else {
+            // The input is not ours, but we still need to count it for the fees
+            let tx = db
+                .get_raw_tx(&input.previous_output.txid)?
+                .ok_or(Error::TransactionNotFound)?;
+            inputs_sum += tx.output[input.previous_output.vout as usize].value;
+        }
+
+        // removes conflicting UTXO if any (generated from same inputs, like for example RBF)
+        if let Some(outpoint) = utxo_deps.get(&input.previous_output) {
+            updates.del_utxo(outpoint)?;
+        }
+    }
+
+    for (i, output) in tx.output.iter().enumerate() {
+        // to compute the fees later
+        outputs_sum += output.value;
+
+        // this output is ours, we have a path to derive it
+        if let Some((keychain, _child)) = db.get_path_from_script_pubkey(&output.script_pubkey)? {
+            debug!("{} output #{} is mine, adding utxo", txid, i);
+            updates.set_utxo(&LocalUtxo {
+                outpoint: OutPoint::new(tx.txid(), i as u32),
+                txout: output.clone(),
+                keychain,
+            })?;
+
+            incoming += output.value;
+        }
+    }
+
+    let tx_details = TransactionDetails {
+        txid: tx.txid(),
+        transaction: Some(tx),
+        received: incoming,
+        sent: outgoing,
+        confirmation_time: ConfirmationTime::new(height, timestamp),
+        fee: Some(inputs_sum.saturating_sub(outputs_sum)), /* if the tx is a coinbase, fees would be negative */
+        verified: height.is_some(),
+    };
+    updates.set_tx(&tx_details)?;
+
+    Ok(())
+}
+
+/// returns utxo dependency as the inputs needed for the utxo to exist
+/// `tx_raw_in_db` must contains utxo's generating txs or errors witt [crate::Error::TransactionNotFound]
+fn utxos_deps<D: BatchDatabase>(
+    db: &mut D,
+    tx_raw_in_db: &HashMap<Txid, Transaction>,
+) -> Result<HashMap<OutPoint, OutPoint>, Error> {
+    let utxos = db.iter_utxos()?;
+    let mut utxos_deps = HashMap::new();
+    for utxo in utxos {
+        let from_tx = tx_raw_in_db
+            .get(&utxo.outpoint.txid)
+            .ok_or(Error::TransactionNotFound)?;
+        for input in from_tx.input.iter() {
+            utxos_deps.insert(input.previous_output, utxo.outpoint);
+        }
+    }
+    Ok(utxos_deps)
+}

src/database/any.rs

@@ -23,12 +23,12 @@
 //! # use bdk::database::{AnyDatabase, MemoryDatabase};
 //! # use bdk::{Wallet};
 //! let memory = MemoryDatabase::default();
-//! let wallet_memory = Wallet::new("...", None, Network::Testnet, memory)?;
+//! let wallet_memory = Wallet::new_offline("...", None, Network::Testnet, memory)?;
 //!
 //! # #[cfg(feature = "key-value-db")]
 //! # {
 //! let sled = sled::open("my-database")?.open_tree("default_tree")?;
-//! let wallet_sled = Wallet::new("...", None, Network::Testnet, sled)?;
+//! let wallet_sled = Wallet::new_offline("...", None, Network::Testnet, sled)?;
 //! # }
 //! # Ok::<(), bdk::Error>(())
 //! ```
@@ -42,7 +42,7 @@
 //! # use bdk::{Wallet};
 //! let config = serde_json::from_str("...")?;
 //! let database = AnyDatabase::from_config(&config)?;
-//! let wallet = Wallet::new("...", None, Network::Testnet, database)?;
+//! let wallet = Wallet::new_offline("...", None, Network::Testnet, database)?;
 //! # Ok::<(), bdk::Error>(())
 //! ```
 
@@ -65,8 +65,6 @@ macro_rules! impl_inner_method {
             $enum_name::Memory(inner) => inner.$name( $($args, )* ),
             #[cfg(feature = "key-value-db")]
             $enum_name::Sled(inner) => inner.$name( $($args, )* ),
-            #[cfg(feature = "sqlite")]
-            $enum_name::Sqlite(inner) => inner.$name( $($args, )* ),
         }
     }
 }
@@ -84,15 +82,10 @@ pub enum AnyDatabase {
     #[cfg_attr(docsrs, doc(cfg(feature = "key-value-db")))]
     /// Simple key-value embedded database based on [`sled`]
     Sled(sled::Tree),
-    #[cfg(feature = "sqlite")]
-    #[cfg_attr(docsrs, doc(cfg(feature = "sqlite")))]
-    /// Sqlite embedded database using [`rusqlite`]
-    Sqlite(sqlite::SqliteDatabase),
 }
 
 impl_from!(memory::MemoryDatabase, AnyDatabase, Memory,);
 impl_from!(sled::Tree, AnyDatabase, Sled, #[cfg(feature = "key-value-db")]);
-impl_from!(sqlite::SqliteDatabase, AnyDatabase, Sqlite, #[cfg(feature = "sqlite")]);
 
 /// Type that contains any of the [`BatchDatabase::Batch`] types defined by the library
 pub enum AnyBatch {
@@ -102,10 +95,6 @@ pub enum AnyBatch {
     #[cfg_attr(docsrs, doc(cfg(feature = "key-value-db")))]
     /// Simple key-value embedded database based on [`sled`]
     Sled(<sled::Tree as BatchDatabase>::Batch),
-    #[cfg(feature = "sqlite")]
-    #[cfg_attr(docsrs, doc(cfg(feature = "sqlite")))]
-    /// Sqlite embedded database using [`rusqlite`]
-    Sqlite(<sqlite::SqliteDatabase as BatchDatabase>::Batch),
 }
 
 impl_from!(
@@ -114,7 +103,6 @@ impl_from!(
     Memory,
 );
 impl_from!(<sled::Tree as BatchDatabase>::Batch, AnyBatch, Sled, #[cfg(feature = "key-value-db")]);
-impl_from!(<sqlite::SqliteDatabase as BatchDatabase>::Batch, AnyBatch, Sqlite, #[cfg(feature = "sqlite")]);
 
 impl BatchOperations for AnyDatabase {
     fn set_script_pubkey(
@@ -144,9 +132,6 @@ impl BatchOperations for AnyDatabase {
     fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error> {
         impl_inner_method!(AnyDatabase, self, set_last_index, keychain, value)
     }
-    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error> {
-        impl_inner_method!(AnyDatabase, self, set_sync_time, sync_time)
-    }
 
     fn del_script_pubkey_from_path(
         &mut self,
@@ -183,9 +168,6 @@ impl BatchOperations for AnyDatabase {
     fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         impl_inner_method!(AnyDatabase, self, del_last_index, keychain)
     }
-    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
-        impl_inner_method!(AnyDatabase, self, del_sync_time)
-    }
 }
 
 impl Database for AnyDatabase {
@@ -247,17 +229,10 @@ impl Database for AnyDatabase {
     fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         impl_inner_method!(AnyDatabase, self, get_last_index, keychain)
     }
-    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
-        impl_inner_method!(AnyDatabase, self, get_sync_time)
-    }
 
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
         impl_inner_method!(AnyDatabase, self, increment_last_index, keychain)
     }
-
-    fn flush(&mut self) -> Result<(), Error> {
-        impl_inner_method!(AnyDatabase, self, flush)
-    }
 }
 
 impl BatchOperations for AnyBatch {
@@ -281,9 +256,6 @@ impl BatchOperations for AnyBatch {
     fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error> {
         impl_inner_method!(AnyBatch, self, set_last_index, keychain, value)
     }
-    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error> {
-        impl_inner_method!(AnyBatch, self, set_sync_time, sync_time)
-    }
 
     fn del_script_pubkey_from_path(
         &mut self,
@@ -314,9 +286,6 @@ impl BatchOperations for AnyBatch {
     fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error> {
         impl_inner_method!(AnyBatch, self, del_last_index, keychain)
     }
-    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
-        impl_inner_method!(AnyBatch, self, del_sync_time)
-    }
 }
 
 impl BatchDatabase for AnyDatabase {
@@ -327,26 +296,19 @@ impl BatchDatabase for AnyDatabase {
             AnyDatabase::Memory(inner) => inner.begin_batch().into(),
             #[cfg(feature = "key-value-db")]
             AnyDatabase::Sled(inner) => inner.begin_batch().into(),
-            #[cfg(feature = "sqlite")]
-            AnyDatabase::Sqlite(inner) => inner.begin_batch().into(),
         }
     }
     fn commit_batch(&mut self, batch: Self::Batch) -> Result<(), Error> {
         match self {
             AnyDatabase::Memory(db) => match batch {
                 AnyBatch::Memory(batch) => db.commit_batch(batch),
-                #[cfg(any(feature = "key-value-db", feature = "sqlite"))]
-                _ => unimplemented!("Other batch shouldn't be used with Memory db."),
+                #[cfg(feature = "key-value-db")]
+                _ => unimplemented!("Sled batch shouldn't be used with Memory db."),
             },
             #[cfg(feature = "key-value-db")]
             AnyDatabase::Sled(db) => match batch {
                 AnyBatch::Sled(batch) => db.commit_batch(batch),
-                _ => unimplemented!("Other batch shouldn't be used with Sled db."),
-            },
-            #[cfg(feature = "sqlite")]
-            AnyDatabase::Sqlite(db) => match batch {
-                AnyBatch::Sqlite(batch) => db.commit_batch(batch),
-                _ => unimplemented!("Other batch shouldn't be used with Sqlite db."),
+                _ => unimplemented!("Memory batch shouldn't be used with Sled db."),
             },
         }
     }
@@ -371,23 +333,6 @@ impl ConfigurableDatabase for sled::Tree {
     }
 }
 
-/// Configuration type for a [`sqlite::SqliteDatabase`] database
-#[cfg(feature = "sqlite")]
-#[derive(Debug, serde::Serialize, serde::Deserialize)]
-pub struct SqliteDbConfiguration {
-    /// Main directory of the db
-    pub path: String,
-}
-
-#[cfg(feature = "sqlite")]
-impl ConfigurableDatabase for sqlite::SqliteDatabase {
-    type Config = SqliteDbConfiguration;
-
-    fn from_config(config: &Self::Config) -> Result<Self, Error> {
-        Ok(sqlite::SqliteDatabase::new(config.path.clone()))
-    }
-}
-
 /// Type that can contain any of the database configurations defined by the library
 ///
 /// This allows storing a single configuration that can be loaded into an [`AnyDatabase`]
@@ -401,10 +346,6 @@ pub enum AnyDatabaseConfig {
     #[cfg_attr(docsrs, doc(cfg(feature = "key-value-db")))]
     /// Simple key-value embedded database based on [`sled`]
     Sled(SledDbConfiguration),
-    #[cfg(feature = "sqlite")]
-    #[cfg_attr(docsrs, doc(cfg(feature = "sqlite")))]
-    /// Sqlite embedded database using [`rusqlite`]
-    Sqlite(SqliteDbConfiguration),
 }
 
 impl ConfigurableDatabase for AnyDatabase {
@@ -417,14 +358,9 @@ impl ConfigurableDatabase for AnyDatabase {
             }
             #[cfg(feature = "key-value-db")]
             AnyDatabaseConfig::Sled(inner) => AnyDatabase::Sled(sled::Tree::from_config(inner)?),
-            #[cfg(feature = "sqlite")]
-            AnyDatabaseConfig::Sqlite(inner) => {
-                AnyDatabase::Sqlite(sqlite::SqliteDatabase::from_config(inner)?)
-            }
         })
     }
 }
 
 impl_from!((), AnyDatabaseConfig, Memory,);
 impl_from!(SledDbConfiguration, AnyDatabaseConfig, Sled, #[cfg(feature = "key-value-db")]);
-impl_from!(SqliteDbConfiguration, AnyDatabaseConfig, Sqlite, #[cfg(feature = "sqlite")]);

src/database/keyvalue.rs

@@ -18,7 +18,7 @@ use bitcoin::hash_types::Txid;
 use bitcoin::{OutPoint, Script, Transaction};
 
 use crate::database::memory::MapKey;
-use crate::database::{BatchDatabase, BatchOperations, Database, SyncTime};
+use crate::database::{BatchDatabase, BatchOperations, Database};
 use crate::error::Error;
 use crate::types::*;
 
@@ -82,13 +82,6 @@ macro_rules! impl_batch_operations {
             Ok(())
         }
 
-        fn set_sync_time(&mut self, data: SyncTime) -> Result<(), Error> {
-            let key = MapKey::SyncTime.as_map_key();
-            self.insert(key, serde_json::to_vec(&data)?)$($after_insert)*;
-
-            Ok(())
-        }
-
         fn del_script_pubkey_from_path(&mut self, keychain: KeychainKind, path: u32) -> Result<Option<Script>, Error> {
             let key = MapKey::Path((Some(keychain), Some(path))).as_map_key();
             let res = self.remove(key);
@@ -175,14 +168,6 @@ macro_rules! impl_batch_operations {
                 }
             }
         }
-
-        fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
-            let key = MapKey::SyncTime.as_map_key();
-            let res = self.remove(key);
-            let res = $process_delete!(res);
-
-            Ok(res.map(|b| serde_json::from_slice(&b)).transpose()?)
-        }
     }
 }
 
@@ -357,14 +342,6 @@ impl Database for Tree {
             .transpose()
     }
 
-    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
-        let key = MapKey::SyncTime.as_map_key();
-        Ok(self
-            .get(key)?
-            .map(|b| serde_json::from_slice(&b))
-            .transpose()?)
-    }
-
     // inserts 0 if not present
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
         let key = MapKey::LastIndex(keychain).as_map_key();
@@ -390,10 +367,6 @@ impl Database for Tree {
             Ok(val)
         })
     }
-
-    fn flush(&mut self) -> Result<(), Error> {
-        Ok(Tree::flush(self).map(|_| ())?)
-    }
 }
 
 impl BatchDatabase for Tree {
@@ -493,9 +466,4 @@ mod test {
     fn test_last_index() {
         crate::database::test::test_last_index(get_tree());
     }
-
-    #[test]
-    fn test_sync_time() {
-        crate::database::test::test_sync_time(get_tree());
-    }
 }

src/database/memory.rs

@@ -14,7 +14,6 @@
 //! This module defines an in-memory database type called [`MemoryDatabase`] that is based on a
 //! [`BTreeMap`].
 
-use std::any::Any;
 use std::collections::BTreeMap;
 use std::ops::Bound::{Excluded, Included};
 
@@ -22,7 +21,7 @@ use bitcoin::consensus::encode::{deserialize, serialize};
 use bitcoin::hash_types::Txid;
 use bitcoin::{OutPoint, Script, Transaction};
 
-use crate::database::{BatchDatabase, BatchOperations, ConfigurableDatabase, Database, SyncTime};
+use crate::database::{BatchDatabase, BatchOperations, ConfigurableDatabase, Database};
 use crate::error::Error;
 use crate::types::*;
 
@@ -33,7 +32,6 @@ use crate::types::*;
 // transactions         t<txid> -> tx details
 // deriv indexes        c{i,e} -> u32
 // descriptor checksum  d{i,e} -> vec<u8>
-// last sync time       l -> { height, timestamp }
 
 pub(crate) enum MapKey<'a> {
     Path((Option<KeychainKind>, Option<u32>)),
@@ -42,7 +40,6 @@ pub(crate) enum MapKey<'a> {
     RawTx(Option<&'a Txid>),
     Transaction(Option<&'a Txid>),
     LastIndex(KeychainKind),
-    SyncTime,
     DescriptorChecksum(KeychainKind),
 }
 
@@ -61,7 +58,6 @@ impl MapKey<'_> {
             MapKey::RawTx(_) => b"r".to_vec(),
             MapKey::Transaction(_) => b"t".to_vec(),
             MapKey::LastIndex(st) => [b"c", st.as_ref()].concat(),
-            MapKey::SyncTime => b"l".to_vec(),
             MapKey::DescriptorChecksum(st) => [b"d", st.as_ref()].concat(),
         }
     }
@@ -109,12 +105,12 @@ fn after(key: &[u8]) -> Vec<u8> {
 /// Once it's dropped its content will be lost.
 ///
 /// If you are looking for a permanent storage solution, you can try with the default key-value
-/// database called [`sled`]. See the [`database`] module documentation for more details.
+/// database called [`sled`]. See the [`database`] module documentation for more defailts.
 ///
 /// [`database`]: crate::database
 #[derive(Debug, Default)]
 pub struct MemoryDatabase {
-    map: BTreeMap<Vec<u8>, Box<dyn Any + Send + Sync>>,
+    map: BTreeMap<Vec<u8>, Box<dyn std::any::Any>>,
     deleted_keys: Vec<Vec<u8>>,
 }
 
@@ -183,12 +179,6 @@ impl BatchOperations for MemoryDatabase {
 
         Ok(())
     }
-    fn set_sync_time(&mut self, data: SyncTime) -> Result<(), Error> {
-        let key = MapKey::SyncTime.as_map_key();
-        self.map.insert(key, Box::new(data));
-
-        Ok(())
-    }
 
     fn del_script_pubkey_from_path(
         &mut self,
@@ -279,13 +269,6 @@ impl BatchOperations for MemoryDatabase {
             Some(b) => Ok(Some(*b.downcast_ref().unwrap())),
         }
     }
-    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error> {
-        let key = MapKey::SyncTime.as_map_key();
-        let res = self.map.remove(&key);
-        self.deleted_keys.push(key);
-
-        Ok(res.map(|b| b.downcast_ref().cloned().unwrap()))
-    }
 }
 
 impl Database for MemoryDatabase {
@@ -423,14 +406,6 @@ impl Database for MemoryDatabase {
         Ok(self.map.get(&key).map(|b| *b.downcast_ref().unwrap()))
     }
 
-    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error> {
-        let key = MapKey::SyncTime.as_map_key();
-        Ok(self
-            .map
-            .get(&key)
-            .map(|b| b.downcast_ref().cloned().unwrap()))
-    }
-
     // inserts 0 if not present
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error> {
         let key = MapKey::LastIndex(keychain).as_map_key();
@@ -444,10 +419,6 @@ impl Database for MemoryDatabase {
 
         Ok(*value)
     }
-
-    fn flush(&mut self) -> Result<(), Error> {
-        Ok(())
-    }
 }
 
 impl BatchDatabase for MemoryDatabase {
@@ -481,21 +452,20 @@ impl ConfigurableDatabase for MemoryDatabase {
 /// don't have `test` set.
 macro_rules! populate_test_db {
     ($db:expr, $tx_meta:expr, $current_height:expr$(,)?) => {{
-        use std::str::FromStr;
         use $crate::database::BatchOperations;
         let mut db = $db;
         let tx_meta = $tx_meta;
         let current_height: Option<u32> = $current_height;
-        let tx = $crate::bitcoin::Transaction {
+        let tx = Transaction {
             version: 1,
             lock_time: 0,
             input: vec![],
             output: tx_meta
                 .output
                 .iter()
-                .map(|out_meta| $crate::bitcoin::TxOut {
+                .map(|out_meta| bitcoin::TxOut {
                     value: out_meta.value,
-                    script_pubkey: $crate::bitcoin::Address::from_str(&out_meta.to_address)
+                    script_pubkey: bitcoin::Address::from_str(&out_meta.to_address)
                         .unwrap()
                         .script_pubkey(),
                 })
@@ -503,29 +473,30 @@ macro_rules! populate_test_db {
         };
 
         let txid = tx.txid();
-        let confirmation_time = tx_meta.min_confirmations.map(|conf| $crate::BlockTime {
+        let confirmation_time = tx_meta.min_confirmations.map(|conf| ConfirmationTime {
             height: current_height.unwrap().checked_sub(conf as u32).unwrap(),
             timestamp: 0,
         });
 
-        let tx_details = $crate::TransactionDetails {
+        let tx_details = TransactionDetails {
             transaction: Some(tx.clone()),
             txid,
             fee: Some(0),
             received: 0,
             sent: 0,
             confirmation_time,
+            verified: current_height.is_some(),
         };
 
         db.set_tx(&tx_details).unwrap();
         for (vout, out) in tx.output.iter().enumerate() {
-            db.set_utxo(&$crate::LocalUtxo {
+            db.set_utxo(&LocalUtxo {
                 txout: out.clone(),
-                outpoint: $crate::bitcoin::OutPoint {
+                outpoint: OutPoint {
                     txid,
                     vout: vout as u32,
                 },
-                keychain: $crate::KeychainKind::External,
+                keychain: KeychainKind::External,
             })
             .unwrap();
         }
@@ -554,7 +525,7 @@ macro_rules! doctest_wallet {
             Some(100),
         );
 
-        $crate::Wallet::new(
+        $crate::Wallet::new_offline(
             &descriptors.0,
             descriptors.1.as_ref(),
             Network::Regtest,
@@ -611,9 +582,4 @@ mod test {
     fn test_last_index() {
         crate::database::test::test_last_index(get_tree());
     }
-
-    #[test]
-    fn test_sync_time() {
-        crate::database::test::test_sync_time(get_tree());
-    }
 }

src/database/mod.rs

@@ -24,8 +24,6 @@
 //!
 //! [`Wallet`]: crate::wallet::Wallet
 
-use serde::{Deserialize, Serialize};
-
 use bitcoin::hash_types::Txid;
 use bitcoin::{OutPoint, Script, Transaction, TxOut};
 
@@ -38,23 +36,9 @@ pub use any::{AnyDatabase, AnyDatabaseConfig};
 #[cfg(feature = "key-value-db")]
 pub(crate) mod keyvalue;
 
-#[cfg(feature = "sqlite")]
-pub(crate) mod sqlite;
-#[cfg(feature = "sqlite")]
-pub use sqlite::SqliteDatabase;
-
 pub mod memory;
 pub use memory::MemoryDatabase;
 
-/// Blockchain state at the time of syncing
-///
-/// Contains only the block time and height at the moment
-#[derive(Clone, Debug, Serialize, Deserialize)]
-pub struct SyncTime {
-    /// Block timestamp and height at the time of sync
-    pub block_time: BlockTime,
-}
-
 /// Trait for operations that can be batched
 ///
 /// This trait defines the list of operations that must be implemented on the [`Database`] type and
@@ -75,8 +59,6 @@ pub trait BatchOperations {
     fn set_tx(&mut self, transaction: &TransactionDetails) -> Result<(), Error>;
     /// Store the last derivation index for a given keychain.
     fn set_last_index(&mut self, keychain: KeychainKind, value: u32) -> Result<(), Error>;
-    /// Store the sync time
-    fn set_sync_time(&mut self, sync_time: SyncTime) -> Result<(), Error>;
 
     /// Delete a script_pubkey given the keychain and its child number.
     fn del_script_pubkey_from_path(
@@ -102,10 +84,6 @@ pub trait BatchOperations {
     ) -> Result<Option<TransactionDetails>, Error>;
     /// Delete the last derivation index for a keychain.
     fn del_last_index(&mut self, keychain: KeychainKind) -> Result<Option<u32>, Error>;
-    /// Reset the sync time to `None`
-    ///
-    /// Returns the removed value
-    fn del_sync_time(&mut self) -> Result<Option<SyncTime>, Error>;
 }
 
 /// Trait for reading data from a database
@@ -149,18 +127,13 @@ pub trait Database: BatchOperations {
     fn get_raw_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error>;
     /// Fetch the transaction metadata and optionally also the raw transaction
     fn get_tx(&self, txid: &Txid, include_raw: bool) -> Result<Option<TransactionDetails>, Error>;
-    /// Return the last derivation index for a keychain.
+    /// Return the last defivation index for a keychain.
     fn get_last_index(&self, keychain: KeychainKind) -> Result<Option<u32>, Error>;
-    /// Return the sync time, if present
-    fn get_sync_time(&self) -> Result<Option<SyncTime>, Error>;
 
     /// Increment the last derivation index for a keychain and return it
     ///
     /// It should insert and return `0` if not present in the database
     fn increment_last_index(&mut self, keychain: KeychainKind) -> Result<u32, Error>;
-
-    /// Force changes to be written to disk
-    fn flush(&mut self) -> Result<(), Error>;
 }
 
 /// Trait for a database that supports batch operations
@@ -344,10 +317,11 @@ pub mod test {
             received: 1337,
             sent: 420420,
             fee: Some(140),
-            confirmation_time: Some(BlockTime {
+            confirmation_time: Some(ConfirmationTime {
                 timestamp: 123456,
                 height: 1000,
             }),
+            verified: true,
         };
 
         tree.set_tx(&tx_details).unwrap();
@@ -395,25 +369,5 @@ pub mod test {
         );
     }
 
-    pub fn test_sync_time<D: Database>(mut tree: D) {
-        assert!(tree.get_sync_time().unwrap().is_none());
-
-        tree.set_sync_time(SyncTime {
-            block_time: BlockTime {
-                height: 100,
-                timestamp: 1000,
-            },
-        })
-        .unwrap();
-
-        let extracted = tree.get_sync_time().unwrap();
-        assert!(extracted.is_some());
-        assert_eq!(extracted.as_ref().unwrap().block_time.height, 100);
-        assert_eq!(extracted.as_ref().unwrap().block_time.timestamp, 1000);
-
-        tree.del_sync_time().unwrap();
-        assert!(tree.get_sync_time().unwrap().is_none());
-    }
-
     // TODO: more tests...
 }

src/descriptor/dsl.rs

@@ -84,7 +84,7 @@ macro_rules! impl_leaf_opcode {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_miniscript()?;
+            minisc.check_minsicript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -108,7 +108,7 @@ macro_rules! impl_leaf_opcode_value {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_miniscript()?;
+            minisc.check_minsicript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -132,7 +132,7 @@ macro_rules! impl_leaf_opcode_value_two {
         )
         .map_err($crate::descriptor::DescriptorError::Miniscript)
         .and_then(|minisc| {
-            minisc.check_miniscript()?;
+            minisc.check_minsicript()?;
             Ok(minisc)
         })
         .map(|minisc| {
@@ -165,7 +165,7 @@ macro_rules! impl_node_opcode_two {
                     std::sync::Arc::new(b_minisc),
                 ))?;
 
-                minisc.check_miniscript()?;
+                minisc.check_minsicript()?;
 
                 Ok((minisc, a_keymap, $crate::keys::merge_networks(&a_networks, &b_networks)))
             })
@@ -197,7 +197,7 @@ macro_rules! impl_node_opcode_three {
                     std::sync::Arc::new(c_minisc),
                 ))?;
 
-                minisc.check_miniscript()?;
+                minisc.check_minsicript()?;
 
                 Ok((minisc, a_keymap, networks))
             })
@@ -243,7 +243,7 @@ macro_rules! apply_modifier {
                     ),
                 )?;
 
-                minisc.check_miniscript()?;
+                minisc.check_minsicript()?;
 
                 Ok((minisc, keymap, networks))
             })
@@ -521,7 +521,7 @@ macro_rules! fragment_internal {
     // three operands it's (X, (X, (X, ()))), etc.
     //
     // To check that the right number of arguments has been passed we can "cast" those tuples to
-    // more convenient structures like `TupleTwo`. If the conversion succeeds, the right number of
+    // more convenient structures like `TupleTwo`. If the conversion succedes, the right number of
     // args was passed. Otherwise the compilation fails entirely.
     ( @t $op:ident ( $( $args:tt )* ) $( $tail:tt )* ) => ({
         ($crate::fragment!( $op ( $( $args )* ) ), $crate::fragment_internal!( @t $( $tail )* ))
@@ -571,9 +571,8 @@ macro_rules! fragment {
     ( pk ( $key:expr ) ) => ({
         $crate::fragment!(c:pk_k ( $key ))
     });
-    ( pk_h ( $key:expr ) ) => ({
-        let secp = $crate::bitcoin::secp256k1::Secp256k1::new();
-        $crate::keys::make_pkh($key, &secp)
+    ( pk_h ( $key_hash:expr ) ) => ({
+        $crate::impl_leaf_opcode_value!(PkH, $key_hash)
     });
     ( after ( $value:expr ) ) => ({
         $crate::impl_leaf_opcode_value!(After, $value)
@@ -602,9 +601,6 @@ macro_rules! fragment {
     ( and_or ( $( $inner:tt )* ) ) => ({
         $crate::impl_node_opcode_three!(AndOr, $( $inner )*)
     });
-    ( andor ( $( $inner:tt )* ) ) => ({
-        $crate::impl_node_opcode_three!(AndOr, $( $inner )*)
-    });
     ( or_b ( $( $inner:tt )* ) ) => ({
         $crate::impl_node_opcode_two!(OrB, $( $inner )*)
     });

src/descriptor/mod.rs

@@ -17,13 +17,13 @@
 use std::collections::{BTreeMap, HashMap, HashSet};
 use std::ops::Deref;
 
-use bitcoin::util::bip32::{ChildNumber, DerivationPath, ExtendedPubKey, Fingerprint, KeySource};
+use bitcoin::util::bip32::{
+    ChildNumber, DerivationPath, ExtendedPrivKey, ExtendedPubKey, Fingerprint, KeySource,
+};
 use bitcoin::util::psbt;
 use bitcoin::{Network, PublicKey, Script, TxOut};
 
-use miniscript::descriptor::{
-    DescriptorPublicKey, DescriptorType, DescriptorXKey, InnerXKey, Wildcard,
-};
+use miniscript::descriptor::{DescriptorPublicKey, DescriptorType, DescriptorXKey, Wildcard};
 pub use miniscript::{descriptor::KeyMap, Descriptor, Legacy, Miniscript, ScriptContext, Segwitv0};
 use miniscript::{DescriptorTrait, ForEachKey, TranslatePk};
 
@@ -238,13 +238,13 @@ pub(crate) fn into_wallet_descriptor_checked<T: IntoWalletDescriptor>(
 #[doc(hidden)]
 /// Used internally mainly by the `descriptor!()` and `fragment!()` macros
 pub trait CheckMiniscript<Ctx: miniscript::ScriptContext> {
-    fn check_miniscript(&self) -> Result<(), miniscript::Error>;
+    fn check_minsicript(&self) -> Result<(), miniscript::Error>;
 }
 
 impl<Ctx: miniscript::ScriptContext, Pk: miniscript::MiniscriptKey> CheckMiniscript<Ctx>
     for miniscript::Miniscript<Pk, Ctx>
 {
-    fn check_miniscript(&self) -> Result<(), miniscript::Error> {
+    fn check_minsicript(&self) -> Result<(), miniscript::Error> {
         Ctx::check_global_validity(self)?;
 
         Ok(())
@@ -267,10 +267,41 @@ pub(crate) trait XKeyUtils {
     fn root_fingerprint(&self, secp: &SecpCtx) -> Fingerprint;
 }
 
-impl<T> XKeyUtils for DescriptorXKey<T>
-where
-    T: InnerXKey,
-{
+// FIXME: `InnerXKey` was made private in rust-miniscript, so we have to implement this manually on
+// both `ExtendedPubKey` and `ExtendedPrivKey`.
+//
+// Revert back to using the trait once https://github.com/rust-bitcoin/rust-miniscript/pull/230 is
+// released
+impl XKeyUtils for DescriptorXKey<ExtendedPubKey> {
+    fn full_path(&self, append: &[ChildNumber]) -> DerivationPath {
+        let full_path = match self.origin {
+            Some((_, ref path)) => path
+                .into_iter()
+                .chain(self.derivation_path.into_iter())
+                .cloned()
+                .collect(),
+            None => self.derivation_path.clone(),
+        };
+
+        if self.wildcard != Wildcard::None {
+            full_path
+                .into_iter()
+                .chain(append.iter())
+                .cloned()
+                .collect()
+        } else {
+            full_path
+        }
+    }
+
+    fn root_fingerprint(&self, _: &SecpCtx) -> Fingerprint {
+        match self.origin {
+            Some((fingerprint, _)) => fingerprint,
+            None => self.xkey.fingerprint(),
+        }
+    }
+}
+impl XKeyUtils for DescriptorXKey<ExtendedPrivKey> {
     fn full_path(&self, append: &[ChildNumber]) -> DerivationPath {
         let full_path = match self.origin {
             Some((_, ref path)) => path
@@ -295,7 +326,7 @@ where
     fn root_fingerprint(&self, secp: &SecpCtx) -> Fingerprint {
         match self.origin {
             Some((fingerprint, _)) => fingerprint,
-            None => self.xkey.xkey_fingerprint(secp),
+            None => self.xkey.fingerprint(secp),
         }
     }
 }
@@ -636,7 +667,7 @@ mod test {
 
         // make a descriptor out of it
         let desc = crate::descriptor!(wpkh(key)).unwrap();
-        // this should convert the key that supports "any_network" to the right network (testnet)
+        // this should conver the key that supports "any_network" to the right network (testnet)
         let (wallet_desc, _) = desc
             .into_wallet_descriptor(&secp, Network::Testnet)
             .unwrap();

src/descriptor/policy.rs

@@ -47,12 +47,14 @@ use bitcoin::util::bip32::Fingerprint;
 use bitcoin::PublicKey;
 
 use miniscript::descriptor::{DescriptorPublicKey, ShInner, SortedMultiVec, WshInner};
-use miniscript::{Descriptor, Miniscript, MiniscriptKey, Satisfier, ScriptContext, Terminal};
+use miniscript::{
+    Descriptor, Miniscript, MiniscriptKey, Satisfier, ScriptContext, Terminal, ToPublicKey,
+};
 
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
-use crate::descriptor::ExtractPolicy;
+use crate::descriptor::{DerivedDescriptorKey, ExtractPolicy};
 use crate::wallet::signer::{SignerId, SignersContainer};
 use crate::wallet::utils::{self, After, Older, SecpCtx};
 
@@ -86,6 +88,13 @@ impl PkOrF {
             },
         }
     }
+
+    fn from_key_hash(k: hash160::Hash) -> Self {
+        PkOrF {
+            pubkey_hash: Some(k),
+            ..Default::default()
+        }
+    }
 }
 
 /// An item that needs to be satisfied
@@ -354,9 +363,9 @@ impl Satisfaction {
                 indexes
                     .into_iter()
                     // .inspect(|x| println!("--- orig --- {:?}", x))
-                    // we map each of the combinations of elements into a tuple of ([chosen items], [conditions]). unfortunately, those items have potentially more than one
+                    // we map each of the combinations of elements into a tuple of ([choosen items], [conditions]). unfortunately, those items have potentially more than one
                     // condition (think about ORs), so we also use `mix` to expand those, i.e. [[0], [1, 2]] becomes [[0, 1], [0, 2]]. This is necessary to make sure that we
-                    // consider every possible options and check whether or not they are compatible.
+                    // consider every possibile options and check whether or not they are compatible.
                     .map(|i_vec| {
                         mix(i_vec
                             .iter()
@@ -770,6 +779,25 @@ fn signature_in_psbt(psbt: &Psbt, key: &DescriptorPublicKey, secp: &SecpCtx) ->
     })
 }
 
+fn signature_key(
+    key: &<DescriptorPublicKey as MiniscriptKey>::Hash,
+    signers: &SignersContainer,
+    secp: &SecpCtx,
+) -> Policy {
+    let key_hash = DerivedDescriptorKey::new(key.clone(), secp)
+        .to_public_key()
+        .to_pubkeyhash();
+    let mut policy: Policy = SatisfiableItem::Signature(PkOrF::from_key_hash(key_hash)).into();
+
+    if signers.find(SignerId::PkHash(key_hash)).is_some() {
+        policy.contribution = Satisfaction::Complete {
+            condition: Default::default(),
+        }
+    }
+
+    policy
+}
+
 impl<Ctx: ScriptContext> ExtractPolicy for Miniscript<DescriptorPublicKey, Ctx> {
     fn extract_policy(
         &self,
@@ -781,7 +809,7 @@ impl<Ctx: ScriptContext> ExtractPolicy for Miniscript<DescriptorPublicKey, Ctx>
             // Leaves
             Terminal::True | Terminal::False => None,
             Terminal::PkK(pubkey) => Some(signature(pubkey, signers, build_sat, secp)),
-            Terminal::PkH(pubkey_hash) => Some(signature(pubkey_hash, signers, build_sat, secp)),
+            Terminal::PkH(pubkey_hash) => Some(signature_key(pubkey_hash, signers, secp)),
             Terminal::After(value) => {
                 let mut policy: Policy = SatisfiableItem::AbsoluteTimelock { value: *value }.into();
                 policy.contribution = Satisfaction::Complete {
@@ -1416,7 +1444,6 @@ mod test {
 
     const ALICE_TPRV_STR:&str = "tprv8ZgxMBicQKsPf6T5X327efHnvJDr45Xnb8W4JifNWtEoqXu9MRYS4v1oYe6DFcMVETxy5w3bqpubYRqvcVTqovG1LifFcVUuJcbwJwrhYzP";
     const BOB_TPRV_STR:&str = "tprv8ZgxMBicQKsPeinZ155cJAn117KYhbaN6MV3WeG6sWhxWzcvX1eg1awd4C9GpUN1ncLEM2rzEvunAg3GizdZD4QPPCkisTz99tXXB4wZArp";
-    const CAROL_TPRV_STR:&str = "tprv8ZgxMBicQKsPdC3CicFifuLCEyVVdXVUNYorxUWj3iGZ6nimnLAYAY9SYB7ib8rKzRxrCKFcEytCt6szwd2GHnGPRCBLAEAoSVDefSNk4Bt";
     const ALICE_BOB_PATH: &str = "m/0'";
 
     #[test]
@@ -1575,28 +1602,4 @@ mod test {
         );
         //println!("{}", serde_json::to_string(&policy_expired_signed).unwrap());
     }
-
-    #[test]
-    fn test_extract_pkh() {
-        let secp = Secp256k1::new();
-
-        let (prvkey_alice, _, _) = setup_keys(ALICE_TPRV_STR, ALICE_BOB_PATH, &secp);
-        let (prvkey_bob, _, _) = setup_keys(BOB_TPRV_STR, ALICE_BOB_PATH, &secp);
-        let (prvkey_carol, _, _) = setup_keys(CAROL_TPRV_STR, ALICE_BOB_PATH, &secp);
-
-        let desc = descriptor!(wsh(c: andor(
-            pk(prvkey_alice),
-            pk_k(prvkey_bob),
-            pk_h(prvkey_carol),
-        )))
-        .unwrap();
-
-        let (wallet_desc, keymap) = desc
-            .into_wallet_descriptor(&secp, Network::Testnet)
-            .unwrap();
-        let signers_container = Arc::new(SignersContainer::from(keymap));
-
-        let policy = wallet_desc.extract_policy(&signers_container, BuildSatisfaction::None, &secp);
-        assert!(policy.is_ok());
-    }
 }

src/descriptor/template.rs

@@ -79,7 +79,7 @@ impl<T: DescriptorTemplate> IntoWalletDescriptor for T {
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new(
+/// let wallet = Wallet::new_offline(
 ///     P2Pkh(key),
 ///     None,
 ///     Network::Testnet,
@@ -113,7 +113,7 @@ impl<K: IntoDescriptorKey<Legacy>> DescriptorTemplate for P2Pkh<K> {
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new(
+/// let wallet = Wallet::new_offline(
 ///     P2Wpkh_P2Sh(key),
 ///     None,
 ///     Network::Testnet,
@@ -148,7 +148,7 @@ impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh_P2Sh<K> {
 ///
 /// let key =
 ///     bitcoin::PrivateKey::from_wif("cTc4vURSzdx6QE6KVynWGomDbLaA75dNALMNyfjh3p8DRRar84Um")?;
-/// let wallet = Wallet::new(
+/// let wallet = Wallet::new_offline(
 ///     P2Wpkh(key),
 ///     None,
 ///     Network::Testnet,
@@ -186,7 +186,7 @@ impl<K: IntoDescriptorKey<Segwitv0>> DescriptorTemplate for P2Wpkh<K> {
 /// use bdk::template::Bip44;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new(
+/// let wallet = Wallet::new_offline(
 ///     Bip44(key.clone(), KeychainKind::External),
 ///     Some(Bip44(key, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -226,7 +226,7 @@ impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44<K> {
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDDDzQ31JkZB7VxUr9bjvBivDdqoFLrDPyLWtLapArAi51ftfmCb2DPxwLQzX65iNcXz1DGaVvyvo6JQ6rTU73r2gqdEo8uov9QKRb7nKCSU")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new(
+/// let wallet = Wallet::new_offline(
 ///     Bip44Public(key.clone(), fingerprint, KeychainKind::External),
 ///     Some(Bip44Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -262,7 +262,7 @@ impl<K: DerivableKey<Legacy>> DescriptorTemplate for Bip44Public<K> {
 /// use bdk::template::Bip49;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new(
+/// let wallet = Wallet::new_offline(
 ///     Bip49(key.clone(), KeychainKind::External),
 ///     Some(Bip49(key, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -302,7 +302,7 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49<K> {
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC49r947KGK52X5rBWS4BLs5m9SRY3pYHnvRrm7HcybZ3BfdEsGFyzCMzayi1u58eT82ZeyFZwH7DD6Q83E3fM9CpfMtmnTygnLfP59jL9L")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new(
+/// let wallet = Wallet::new_offline(
 ///     Bip49Public(key.clone(), fingerprint, KeychainKind::External),
 ///     Some(Bip49Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -338,7 +338,7 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip49Public<K> {
 /// use bdk::template::Bip84;
 ///
 /// let key = bitcoin::util::bip32::ExtendedPrivKey::from_str("tprv8ZgxMBicQKsPeZRHk4rTG6orPS2CRNFX3njhUXx5vj9qGog5ZMH4uGReDWN5kCkY3jmWEtWause41CDvBRXD1shKknAMKxT99o9qUTRVC6m")?;
-/// let wallet = Wallet::new(
+/// let wallet = Wallet::new_offline(
 ///     Bip84(key.clone(), KeychainKind::External),
 ///     Some(Bip84(key, KeychainKind::Internal)),
 ///     Network::Testnet,
@@ -378,7 +378,7 @@ impl<K: DerivableKey<Segwitv0>> DescriptorTemplate for Bip84<K> {
 ///
 /// let key = bitcoin::util::bip32::ExtendedPubKey::from_str("tpubDC2Qwo2TFsaNC4ju8nrUJ9mqVT3eSgdmy1yPqhgkjwmke3PRXutNGRYAUo6RCHTcVQaDR3ohNU9we59brGHuEKPvH1ags2nevW5opEE9Z5Q")?;
 /// let fingerprint = bitcoin::util::bip32::Fingerprint::from_str("c55b303f")?;
-/// let wallet = Wallet::new(
+/// let wallet = Wallet::new_offline(
 ///     Bip84Public(key.clone(), fingerprint, KeychainKind::External),
 ///     Some(Bip84Public(key, fingerprint, KeychainKind::Internal)),
 ///     Network::Testnet,

src/error.rs

@@ -115,7 +115,7 @@ pub enum Error {
     Hex(bitcoin::hashes::hex::Error),
     /// Partially signed bitcoin transaction error
     Psbt(bitcoin::util::psbt::Error),
-    /// Partially signed bitcoin transaction parse error
+    /// Partially signed bitcoin transaction parseerror
     PsbtParse(bitcoin::util::psbt::PsbtParseError),
 
     //KeyMismatch(bitcoin::secp256k1::PublicKey, bitcoin::secp256k1::PublicKey),
@@ -130,7 +130,7 @@ pub enum Error {
     Electrum(electrum_client::Error),
     #[cfg(feature = "esplora")]
     /// Esplora client error
-    Esplora(Box<crate::blockchain::esplora::EsploraError>),
+    Esplora(crate::blockchain::esplora::EsploraError),
     #[cfg(feature = "compact_filters")]
     /// Compact filters client error)
     CompactFilters(crate::blockchain::compact_filters::CompactFiltersError),
@@ -140,9 +140,6 @@ pub enum Error {
     #[cfg(feature = "rpc")]
     /// Rpc client error
     Rpc(bitcoincore_rpc::Error),
-    #[cfg(feature = "sqlite")]
-    /// Rusqlite client error
-    Rusqlite(rusqlite::Error),
 }
 
 impl fmt::Display for Error {
@@ -193,12 +190,12 @@ impl_error!(bitcoin::util::psbt::PsbtParseError, PsbtParse);
 
 #[cfg(feature = "electrum")]
 impl_error!(electrum_client::Error, Electrum);
+#[cfg(feature = "esplora")]
+impl_error!(crate::blockchain::esplora::EsploraError, Esplora);
 #[cfg(feature = "key-value-db")]
 impl_error!(sled::Error, Sled);
 #[cfg(feature = "rpc")]
 impl_error!(bitcoincore_rpc::Error, Rpc);
-#[cfg(feature = "sqlite")]
-impl_error!(rusqlite::Error, Rusqlite);
 
 #[cfg(feature = "compact_filters")]
 impl From<crate::blockchain::compact_filters::CompactFiltersError> for Error {
@@ -219,10 +216,3 @@ impl From<crate::wallet::verify::VerifyError> for Error {
         }
     }
 }
-
-#[cfg(feature = "esplora")]
-impl From<crate::blockchain::esplora::EsploraError> for Error {
-    fn from(other: crate::blockchain::esplora::EsploraError) -> Self {
-        Error::Esplora(Box::new(other))
-    }
-}

src/keys/bip39.rs

@@ -19,23 +19,7 @@ use bitcoin::Network;
 
 use miniscript::ScriptContext;
 
-pub use bip39::{Language, Mnemonic};
-
-type Seed = [u8; 64];
-
-/// Type describing entropy length (aka word count) in the mnemonic
-pub enum WordCount {
-    /// 12 words mnemonic (128 bits entropy)
-    Words12 = 128,
-    /// 15 words mnemonic (160 bits entropy)
-    Words15 = 160,
-    /// 18 words mnemonic (192 bits entropy)
-    Words18 = 192,
-    /// 21 words mnemonic (224 bits entropy)
-    Words21 = 224,
-    /// 24 words mnemonic (256 bits entropy)
-    Words24 = 256,
-}
+pub use bip39::{Language, Mnemonic, MnemonicType, Seed};
 
 use super::{
     any_network, DerivableKey, DescriptorKey, ExtendedKey, GeneratableKey, GeneratedKey, KeyError,
@@ -56,7 +40,7 @@ pub type MnemonicWithPassphrase = (Mnemonic, Option<String>);
 #[cfg_attr(docsrs, doc(cfg(feature = "keys-bip39")))]
 impl<Ctx: ScriptContext> DerivableKey<Ctx> for Seed {
     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
-        Ok(bip32::ExtendedPrivKey::new_master(Network::Bitcoin, &self[..])?.into())
+        Ok(bip32::ExtendedPrivKey::new_master(Network::Bitcoin, &self.as_bytes())?.into())
     }
 
     fn into_descriptor_key(
@@ -76,7 +60,7 @@ impl<Ctx: ScriptContext> DerivableKey<Ctx> for Seed {
 impl<Ctx: ScriptContext> DerivableKey<Ctx> for MnemonicWithPassphrase {
     fn into_extended_key(self) -> Result<ExtendedKey<Ctx>, KeyError> {
         let (mnemonic, passphrase) = self;
-        let seed: Seed = mnemonic.to_seed(passphrase.as_deref().unwrap_or(""));
+        let seed = Seed::new(&mnemonic, passphrase.as_deref().unwrap_or(""));
 
         seed.into_extended_key()
     }
@@ -117,15 +101,15 @@ impl<Ctx: ScriptContext> DerivableKey<Ctx> for Mnemonic {
 impl<Ctx: ScriptContext> GeneratableKey<Ctx> for Mnemonic {
     type Entropy = [u8; 32];
 
-    type Options = (WordCount, Language);
-    type Error = Option<bip39::Error>;
+    type Options = (MnemonicType, Language);
+    type Error = Option<bip39::ErrorKind>;
 
     fn generate_with_entropy(
-        (word_count, language): Self::Options,
+        (mnemonic_type, language): Self::Options,
         entropy: Self::Entropy,
     ) -> Result<GeneratedKey<Self, Ctx>, Self::Error> {
-        let entropy = &entropy.as_ref()[..(word_count as usize / 8)];
-        let mnemonic = Mnemonic::from_entropy_in(language, entropy)?;
+        let entropy = &entropy.as_ref()[..(mnemonic_type.entropy_bits() / 8)];
+        let mnemonic = Mnemonic::from_entropy(entropy, language).map_err(|e| e.downcast().ok())?;
 
         Ok(GeneratedKey::new(mnemonic, any_network()))
     }
@@ -137,17 +121,15 @@ mod test {
 
     use bitcoin::util::bip32;
 
-    use bip39::{Language, Mnemonic};
+    use bip39::{Language, Mnemonic, MnemonicType};
 
     use crate::keys::{any_network, GeneratableKey, GeneratedKey};
 
-    use super::WordCount;
-
     #[test]
     fn test_keys_bip39_mnemonic() {
         let mnemonic =
             "aim bunker wash balance finish force paper analyst cabin spoon stable organ";
-        let mnemonic = Mnemonic::parse_in(Language::English, mnemonic).unwrap();
+        let mnemonic = Mnemonic::from_phrase(mnemonic, Language::English).unwrap();
         let path = bip32::DerivationPath::from_str("m/44'/0'/0'/0").unwrap();
 
         let key = (mnemonic, path);
@@ -161,7 +143,7 @@ mod test {
     fn test_keys_bip39_mnemonic_passphrase() {
         let mnemonic =
             "aim bunker wash balance finish force paper analyst cabin spoon stable organ";
-        let mnemonic = Mnemonic::parse_in(Language::English, mnemonic).unwrap();
+        let mnemonic = Mnemonic::from_phrase(mnemonic, Language::English).unwrap();
         let path = bip32::DerivationPath::from_str("m/44'/0'/0'/0").unwrap();
 
         let key = ((mnemonic, Some("passphrase".into())), path);
@@ -175,7 +157,7 @@ mod test {
     fn test_keys_generate_bip39() {
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
             Mnemonic::generate_with_entropy(
-                (WordCount::Words12, Language::English),
+                (MnemonicType::Words12, Language::English),
                 crate::keys::test::TEST_ENTROPY,
             )
             .unwrap();
@@ -187,7 +169,7 @@ mod test {
 
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
             Mnemonic::generate_with_entropy(
-                (WordCount::Words24, Language::English),
+                (MnemonicType::Words24, Language::English),
                 crate::keys::test::TEST_ENTROPY,
             )
             .unwrap();
@@ -198,11 +180,11 @@ mod test {
     #[test]
     fn test_keys_generate_bip39_random() {
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
-            Mnemonic::generate((WordCount::Words12, Language::English)).unwrap();
+            Mnemonic::generate((MnemonicType::Words12, Language::English)).unwrap();
         assert_eq!(generated_mnemonic.valid_networks, any_network());
 
         let generated_mnemonic: GeneratedKey<_, miniscript::Segwitv0> =
-            Mnemonic::generate((WordCount::Words24, Language::English)).unwrap();
+            Mnemonic::generate((MnemonicType::Words24, Language::English)).unwrap();
         assert_eq!(generated_mnemonic.valid_networks, any_network());
     }
 }

src/keys/mod.rs

@@ -319,7 +319,6 @@ impl<Ctx: ScriptContext> ExtendedKey<Ctx> {
         match self {
             ExtendedKey::Private((mut xprv, _)) => {
                 xprv.network = network;
-                xprv.private_key.network = network;
                 Some(xprv)
             }
             ExtendedKey::Public(_) => None,
@@ -357,7 +356,7 @@ impl<Ctx: ScriptContext> From<bip32::ExtendedPrivKey> for ExtendedKey<Ctx> {
 
 /// Trait for keys that can be derived.
 ///
-/// When extra metadata are provided, a [`DerivableKey`] can be transformed into a
+/// When extra metadata are provided, a [`DerivableKey`] can be transofrmed into a
 /// [`DescriptorKey`]: the trait [`IntoDescriptorKey`] is automatically implemented
 /// for `(DerivableKey, DerivationPath)` and
 /// `(DerivableKey, KeySource, DerivationPath)` tuples.
@@ -461,9 +460,9 @@ use bdk::keys::bip39::{Mnemonic, Language};
 
 # fn main() -> Result<(), Box<dyn std::error::Error>> {
 let xkey: ExtendedKey =
-    Mnemonic::parse_in(
-        Language::English,
+    Mnemonic::from_phrase(
         "jelly crash boy whisper mouse ecology tuna soccer memory million news short",
+        Language::English
     )?
     .into_extended_key()?;
 let xprv = xkey.into_xprv(Network::Bitcoin).unwrap();
@@ -749,21 +748,7 @@ pub fn make_pk<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
     let (key, key_map, valid_networks) = descriptor_key.into_descriptor_key()?.extract(secp)?;
     let minisc = Miniscript::from_ast(Terminal::PkK(key))?;
 
-    minisc.check_miniscript()?;
-
-    Ok((minisc, key_map, valid_networks))
-}
-
-// Used internally by `bdk::fragment!` to build `pk_h()` fragments
-#[doc(hidden)]
-pub fn make_pkh<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
-    descriptor_key: Pk,
-    secp: &SecpCtx,
-) -> Result<(Miniscript<DescriptorPublicKey, Ctx>, KeyMap, ValidNetworks), DescriptorError> {
-    let (key, key_map, valid_networks) = descriptor_key.into_descriptor_key()?.extract(secp)?;
-    let minisc = Miniscript::from_ast(Terminal::PkH(key))?;
-
-    minisc.check_miniscript()?;
+    minisc.check_minsicript()?;
 
     Ok((minisc, key_map, valid_networks))
 }
@@ -778,7 +763,7 @@ pub fn make_multi<Pk: IntoDescriptorKey<Ctx>, Ctx: ScriptContext>(
     let (pks, key_map, valid_networks) = expand_multi_keys(pks, secp)?;
     let minisc = Miniscript::from_ast(Terminal::Multi(thresh, pks))?;
 
-    minisc.check_miniscript()?;
+    minisc.check_minsicript()?;
 
     Ok((minisc, key_map, valid_networks))
 }
@@ -932,43 +917,4 @@ pub mod test {
             "L2wTu6hQrnDMiFNWA5na6jB12ErGQqtXwqpSL7aWquJaZG8Ai3ch"
         );
     }
-
-    #[test]
-    fn test_keys_wif_network() {
-        // test mainnet wif
-        let generated_xprv: GeneratedKey<_, miniscript::Segwitv0> =
-            bip32::ExtendedPrivKey::generate_with_entropy_default(TEST_ENTROPY).unwrap();
-        let xkey = generated_xprv.into_extended_key().unwrap();
-
-        let network = Network::Bitcoin;
-        let xprv = xkey.into_xprv(network).unwrap();
-        let wif = PrivateKey::from_wif(&xprv.private_key.to_wif()).unwrap();
-        assert_eq!(wif.network, network);
-
-        // test testnet wif
-        let generated_xprv: GeneratedKey<_, miniscript::Segwitv0> =
-            bip32::ExtendedPrivKey::generate_with_entropy_default(TEST_ENTROPY).unwrap();
-        let xkey = generated_xprv.into_extended_key().unwrap();
-
-        let network = Network::Testnet;
-        let xprv = xkey.into_xprv(network).unwrap();
-        let wif = PrivateKey::from_wif(&xprv.private_key.to_wif()).unwrap();
-        assert_eq!(wif.network, network);
-    }
-
-    #[cfg(feature = "keys-bip39")]
-    #[test]
-    fn test_keys_wif_network_bip39() {
-        let xkey: ExtendedKey = bip39::Mnemonic::parse_in(
-            bip39::Language::English,
-            "jelly crash boy whisper mouse ecology tuna soccer memory million news short",
-        )
-        .unwrap()
-        .into_extended_key()
-        .unwrap();
-        let xprv = xkey.into_xprv(Network::Testnet).unwrap();
-        let wif = PrivateKey::from_wif(&xprv.private_key.to_wif()).unwrap();
-
-        assert_eq!(wif.network, Network::Testnet);
-    }
 }

src/lib.rs

@@ -14,10 +14,6 @@
 // only enables the `doc_cfg` feature when
 // the `docsrs` configuration attribute is defined
 #![cfg_attr(docsrs, feature(doc_cfg))]
-#![cfg_attr(
-    docsrs,
-    doc(html_logo_url = "https://github.com/bitcoindevkit/bdk/raw/master/static/bdk.png")
-)]
 
 //! A modern, lightweight, descriptor-based wallet library written in Rust.
 //!
@@ -44,7 +40,7 @@
 //! interact with the bitcoin P2P network.
 //!
 //! ```toml
-//! bdk = "0.17.0"
+//! bdk = "0.9.0"
 //! ```
 #![cfg_attr(
     feature = "electrum",
@@ -53,22 +49,22 @@
 
 ### Example
 ```no_run
-use bdk::{Wallet, SyncOptions};
+use bdk::Wallet;
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::ElectrumBlockchain;
+use bdk::blockchain::{noop_progress, ElectrumBlockchain};
 use bdk::electrum_client::Client;
 
 fn main() -> Result<(), bdk::Error> {
     let client = Client::new("ssl://electrum.blockstream.info:60002")?;
-    let blockchain = ElectrumBlockchain::from(client);
     let wallet = Wallet::new(
         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
+        ElectrumBlockchain::from(client)
     )?;
 
-    wallet.sync(&blockchain, SyncOptions::default())?;
+    wallet.sync(noop_progress(), None)?;
 
     println!("Descriptor balance: {} SAT", wallet.get_balance()?);
 
@@ -87,7 +83,7 @@ fn main() -> Result<(), bdk::Error> {
 //! use bdk::wallet::AddressIndex::New;
 //!
 //! fn main() -> Result<(), bdk::Error> {
-//! let wallet = Wallet::new(
+//! let wallet = Wallet::new_offline(
 //!         "wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/0/*)",
 //!         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
 //!         bitcoin::Network::Testnet,
@@ -108,9 +104,9 @@ fn main() -> Result<(), bdk::Error> {
 
 ### Example
 ```no_run
-use bdk::{FeeRate, Wallet, SyncOptions};
+use bdk::{FeeRate, Wallet};
 use bdk::database::MemoryDatabase;
-use bdk::blockchain::ElectrumBlockchain;
+use bdk::blockchain::{noop_progress, ElectrumBlockchain};
 use bdk::electrum_client::Client;
 
 use bitcoin::consensus::serialize;
@@ -123,10 +119,10 @@ fn main() -> Result<(), bdk::Error> {
         Some("wpkh([c258d2e4/84h/1h/0h]tpubDDYkZojQFQjht8Tm4jsS3iuEmKjTiEGjG6KnuFNKKJb5A6ZUCUZKdvLdSDWofKi4ToRCwb9poe1XdqfUnP4jaJjCB2Zwv11ZLgSbnZSNecE/1/*)"),
         bitcoin::Network::Testnet,
         MemoryDatabase::default(),
+        ElectrumBlockchain::from(client)
     )?;
-    let blockchain = ElectrumBlockchain::from(client);
 
-    wallet.sync(&blockchain, SyncOptions::default())?;
+    wallet.sync(noop_progress(), None)?;
 
     let send_to = wallet.get_address(New)?;
     let (psbt, details) = {
@@ -160,7 +156,7 @@ fn main() -> Result<(), bdk::Error> {
 //! use bdk::database::MemoryDatabase;
 //!
 //! fn main() -> Result<(), bdk::Error> {
-//!     let wallet = Wallet::new(
+//!     let wallet = Wallet::new_offline(
 //!         "wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/0/*)",
 //!         Some("wpkh([c258d2e4/84h/1h/0h]tprv8griRPhA7342zfRyB6CqeKF8CJDXYu5pgnj1cjL1u2ngKcJha5jjTRimG82ABzJQ4MQe71CV54xfn25BbhCNfEGGJZnxvCDQCd6JkbvxW6h/1/*)"),
 //!         bitcoin::Network::Testnet,
@@ -209,24 +205,11 @@ extern crate serde;
 #[macro_use]
 extern crate serde_json;
 
-#[cfg(all(feature = "reqwest", feature = "ureq"))]
-compile_error!("Features reqwest and ureq are mutually exclusive and cannot be enabled together");
-
 #[cfg(all(feature = "async-interface", feature = "electrum"))]
 compile_error!(
     "Features async-interface and electrum are mutually exclusive and cannot be enabled together"
 );
 
-#[cfg(all(feature = "async-interface", feature = "ureq"))]
-compile_error!(
-    "Features async-interface and ureq are mutually exclusive and cannot be enabled together"
-);
-
-#[cfg(all(feature = "async-interface", feature = "compact_filters"))]
-compile_error!(
-    "Features async-interface and compact_filters are mutually exclusive and cannot be enabled together"
-);
-
 #[cfg(feature = "keys-bip39")]
 extern crate bip39;
 
@@ -245,12 +228,12 @@ pub extern crate bitcoincore_rpc;
 #[cfg(feature = "electrum")]
 pub extern crate electrum_client;
 
+#[cfg(feature = "esplora")]
+pub extern crate reqwest;
+
 #[cfg(feature = "key-value-db")]
 pub extern crate sled;
 
-#[cfg(feature = "sqlite")]
-pub extern crate rusqlite;
-
 #[allow(unused_imports)]
 #[macro_use]
 pub(crate) mod error;
@@ -272,7 +255,6 @@ pub use wallet::address_validator;
 pub use wallet::signer;
 pub use wallet::signer::SignOptions;
 pub use wallet::tx_builder::TxBuilder;
-pub use wallet::SyncOptions;
 pub use wallet::Wallet;
 
 /// Get the version of BDK at runtime
@@ -280,7 +262,7 @@ pub fn version() -> &'static str {
     env!("CARGO_PKG_VERSION", "unknown")
 }
 
-// We should consider putting this under a feature flag but we need the macro in doctests so we need
+// We should consider putting this under a feature flag but we need the macro in doctets so we need
 // to wait until https://github.com/rust-lang/rust/issues/67295 is fixed.
 //
 // Stuff in here is too rough to document atm

src/psbt/mod.rs

@@ -43,8 +43,8 @@ impl PsbtUtils for Psbt {
 mod test {
     use crate::bitcoin::TxIn;
     use crate::psbt::Psbt;
+    use crate::wallet::test::{get_funded_wallet, get_test_wpkh};
     use crate::wallet::AddressIndex;
-    use crate::wallet::{get_funded_wallet, test::get_test_wpkh};
     use crate::SignOptions;
     use std::str::FromStr;
 

src/testutils/blockchain_tests.rs

@@ -10,7 +10,7 @@ use electrsd::bitcoind::BitcoinD;
 use electrsd::{bitcoind, ElectrsD};
 pub use electrum_client::{Client as ElectrumClient, ElectrumApi};
 #[allow(unused_imports)]
-use log::{debug, error, info, log_enabled, trace, Level};
+use log::{debug, error, info, trace};
 use std::collections::HashMap;
 use std::env;
 use std::ops::Deref;
@@ -24,16 +24,11 @@ pub struct TestClient {
 impl TestClient {
     pub fn new(bitcoind_exe: String, electrs_exe: String) -> Self {
         debug!("launching {} and {}", &bitcoind_exe, &electrs_exe);
+        let bitcoind = BitcoinD::new(bitcoind_exe).unwrap();
 
-        let mut conf = bitcoind::Conf::default();
-        conf.view_stdout = log_enabled!(Level::Debug);
-        let bitcoind = BitcoinD::with_conf(bitcoind_exe, &conf).unwrap();
+        let http_enabled = cfg!(feature = "test-esplora");
 
-        let mut conf = electrsd::Conf::default();
-        conf.view_stderr = log_enabled!(Level::Debug);
-        conf.http_enabled = cfg!(feature = "test-esplora");
-
-        let electrsd = ElectrsD::with_conf(electrs_exe, &bitcoind, &conf).unwrap();
+        let electrsd = ElectrsD::new(electrs_exe, &bitcoind, false, http_enabled).unwrap();
 
         let node_address = bitcoind.client.get_new_address(None, None).unwrap();
         bitcoind
@@ -90,19 +85,13 @@ impl TestClient {
             map.insert(out.to_address.clone(), Amount::from_sat(out.value));
         }
 
-        let input: Vec<_> = meta_tx
-            .input
-            .into_iter()
-            .map(|x| x.into_raw_tx_input())
-            .collect();
-
         if self.get_balance(None, None).unwrap() < Amount::from_sat(required_balance) {
             panic!("Insufficient funds in bitcoind. Please generate a few blocks with: `bitcoin-cli generatetoaddress 10 {}`", self.get_new_address(None, None).unwrap());
         }
 
         // FIXME: core can't create a tx with two outputs to the same address
         let tx = self
-            .create_raw_transaction_hex(&input, &map, meta_tx.locktime, meta_tx.replaceable)
+            .create_raw_transaction_hex(&[], &map, meta_tx.locktime, meta_tx.replaceable)
             .unwrap();
         let tx = self.fund_raw_transaction(tx, None, None).unwrap();
         let mut tx: Transaction = deserialize(&tx.hex).unwrap();
@@ -151,7 +140,9 @@ impl TestClient {
 
         let bumped: serde_json::Value = self.call("bumpfee", &[txid.to_string().into()]).unwrap();
         let new_txid = Txid::from_str(&bumped["txid"].as_str().unwrap().to_string()).unwrap();
-        let monitor_script = Script::from_hex(&mut tx.vout[0].script_pub_key.hex.to_hex()).unwrap();
+
+        let monitor_script =
+            tx.vout[0].script_pub_key.addresses.as_ref().unwrap()[0].script_pubkey();
         self.wait_for_tx(new_txid, &monitor_script);
 
         debug!("Bumped {}, new txid {}", txid, new_txid);
@@ -359,12 +350,12 @@ macro_rules! bdk_blockchain_tests {
      fn $_fn_name:ident ( $( $test_client:ident : &TestClient )? $(,)? ) -> $blockchain:ty $block:block) => {
         #[cfg(test)]
         mod bdk_blockchain_tests {
-            use $crate::bitcoin::{Transaction, Network};
+            use $crate::bitcoin::Network;
             use $crate::testutils::blockchain_tests::TestClient;
-            use $crate::blockchain::Blockchain;
+            use $crate::blockchain::noop_progress;
             use $crate::database::MemoryDatabase;
             use $crate::types::KeychainKind;
-            use $crate::{Wallet, FeeRate, SyncOptions};
+            use $crate::{Wallet, FeeRate};
             use $crate::testutils;
 
             use super::*;
@@ -375,11 +366,11 @@ macro_rules! bdk_blockchain_tests {
                 $block
             }
 
-            fn get_wallet_from_descriptors(descriptors: &(String, Option<String>)) -> Wallet<MemoryDatabase> {
-                Wallet::new(&descriptors.0.to_string(), descriptors.1.as_ref(), Network::Regtest, MemoryDatabase::new()).unwrap()
+            fn get_wallet_from_descriptors(descriptors: &(String, Option<String>), test_client: &TestClient) -> Wallet<$blockchain, MemoryDatabase> {
+                Wallet::new(&descriptors.0.to_string(), descriptors.1.as_ref(), Network::Regtest, MemoryDatabase::new(), get_blockchain(test_client)).unwrap()
             }
 
-            fn init_single_sig() -> (Wallet<MemoryDatabase>, $blockchain, (String, Option<String>), TestClient) {
+            fn init_single_sig() -> (Wallet<$blockchain, MemoryDatabase>, (String, Option<String>), TestClient) {
                 let _ = env_logger::try_init();
 
                 let descriptors = testutils! {
@@ -387,22 +378,18 @@ macro_rules! bdk_blockchain_tests {
                 };
 
                 let test_client = TestClient::default();
-                let blockchain = get_blockchain(&test_client);
-                let wallet = get_wallet_from_descriptors(&descriptors);
+                let wallet = get_wallet_from_descriptors(&descriptors, &test_client);
 
                 // rpc need to call import_multi before receiving any tx, otherwise will not see tx in the mempool
                 #[cfg(feature = "test-rpc")]
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
 
-                (wallet, blockchain, descriptors, test_client)
+                (wallet, descriptors, test_client)
             }
 
             #[test]
             fn test_sync_simple() {
-                use std::ops::Deref;
-                use crate::database::Database;
-
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                let (wallet, descriptors, mut test_client) = init_single_sig();
 
                 let tx = testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
@@ -410,13 +397,7 @@ macro_rules! bdk_blockchain_tests {
                 println!("{:?}", tx);
                 let txid = test_client.receive(tx);
 
-                // the RPC blockchain needs to call `sync()` during initialization to import the
-                // addresses (see `init_single_sig()`), so we skip this assertion
-                #[cfg(not(feature = "test-rpc"))]
-                assert!(wallet.database().deref().get_sync_time().unwrap().is_none(), "initial sync_time not none");
-
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-                assert!(wallet.database().deref().get_sync_time().unwrap().is_some(), "sync_time hasn't been updated");
+                wallet.sync(noop_progress(), None).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
                 assert_eq!(wallet.list_unspent().unwrap()[0].keychain, KeychainKind::External, "incorrect keychain kind");
@@ -430,7 +411,7 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_stop_gap_20() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                let (wallet, descriptors, mut test_client) = init_single_sig();
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 5) => 50_000 )
@@ -439,7 +420,7 @@ macro_rules! bdk_blockchain_tests {
                     @tx ( (@external descriptors, 25) => 50_000 )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 100_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 2, "incorrect number of txs");
@@ -447,16 +428,16 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_before_and_after_receive() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                let (wallet, descriptors, mut test_client) = init_single_sig();
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 0);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 1, "incorrect number of txs");
@@ -464,13 +445,13 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_multiple_outputs_same_tx() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                let (wallet, descriptors, mut test_client) = init_single_sig();
 
                 let txid = test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000, (@external descriptors, 5) => 30_000 )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 105_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 1, "incorrect number of txs");
@@ -485,7 +466,7 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_receive_multi() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                let (wallet, descriptors, mut test_client) = init_single_sig();
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
@@ -494,7 +475,7 @@ macro_rules! bdk_blockchain_tests {
                     @tx ( (@external descriptors, 5) => 25_000 )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 75_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 2, "incorrect number of txs");
@@ -503,32 +484,32 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_address_reuse() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                let (wallet, descriptors, mut test_client) = init_single_sig();
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 25_000 )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 75_000, "incorrect balance");
             }
 
             #[test]
             fn test_sync_receive_rbf_replaced() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                let (wallet, descriptors, mut test_client) = init_single_sig();
 
                 let txid = test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 ) ( @replaceable true )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 1, "incorrect number of txs");
@@ -542,7 +523,7 @@ macro_rules! bdk_blockchain_tests {
 
                 let new_txid = test_client.bump_fee(&txid);
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance after bump");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 1, "incorrect number of txs after bump");
@@ -560,13 +541,13 @@ macro_rules! bdk_blockchain_tests {
             #[cfg(not(feature = "esplora"))]
             #[test]
             fn test_sync_reorg_block() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
+                let (wallet, descriptors, mut test_client) = init_single_sig();
 
                 let txid = test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 ) ( @confirmations 1 ) ( @replaceable true )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 1, "incorrect number of txs");
@@ -579,7 +560,7 @@ macro_rules! bdk_blockchain_tests {
                 // Invalidate 1 block
                 test_client.invalidate(1);
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
 
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance after invalidate");
 
@@ -590,15 +571,15 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_after_send() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                                println!("{}", descriptors.0);
+                let (wallet, descriptors, mut test_client) = init_single_sig();
+                println!("{}", descriptors.0);
                 let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -608,93 +589,25 @@ macro_rules! bdk_blockchain_tests {
                 assert!(finalized, "Cannot finalize transaction");
                 let tx = psbt.extract_tx();
                 println!("{}", bitcoin::consensus::encode::serialize_hex(&tx));
-                blockchain.broadcast(&tx).unwrap();
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.broadcast(tx).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), details.received, "incorrect balance after send");
 
                 assert_eq!(wallet.list_transactions(false).unwrap().len(), 2, "incorrect number of txs");
                 assert_eq!(wallet.list_unspent().unwrap().len(), 1, "incorrect number of unspents");
             }
 
-            /// Send two conflicting transactions to the same address twice in a row.
-            /// The coins should only be received once!
-            #[test]
-            fn test_sync_double_receive() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                let receiver_wallet = get_wallet_from_descriptors(&("wpkh(cVpPVruEDdmutPzisEsYvtST1usBR3ntr8pXSyt6D2YYqXRyPcFW)".to_string(), None));
-                // need to sync so rpc can start watching
-                receiver_wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-
-                test_client.receive(testutils! {
-                    @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000 ) (@confirmations 1)
-                });
-
-                wallet.sync(&blockchain, SyncOptions::default()).expect("sync");
-                assert_eq!(wallet.get_balance().unwrap(), 75_000, "incorrect balance");
-                let target_addr = receiver_wallet.get_address($crate::wallet::AddressIndex::New).unwrap().address;
-
-                let tx1 = {
-                    let mut builder = wallet.build_tx();
-                    builder.add_recipient(target_addr.script_pubkey(), 49_000).enable_rbf();
-                    let (mut psbt, _details) = builder.finish().expect("building first tx");
-                    let finalized = wallet.sign(&mut psbt, Default::default()).expect("signing first tx");
-                    assert!(finalized, "Cannot finalize transaction");
-                    psbt.extract_tx()
-                };
-
-                let tx2 = {
-                    let mut builder = wallet.build_tx();
-                    builder.add_recipient(target_addr.script_pubkey(), 49_000).enable_rbf().fee_rate(FeeRate::from_sat_per_vb(5.0));
-                    let (mut psbt, _details) = builder.finish().expect("building replacement tx");
-                    let finalized = wallet.sign(&mut psbt, Default::default()).expect("signing replacement tx");
-                    assert!(finalized, "Cannot finalize transaction");
-                    psbt.extract_tx()
-                };
-
-                blockchain.broadcast(&tx1).expect("broadcasting first");
-                blockchain.broadcast(&tx2).expect("broadcasting replacement");
-
-                receiver_wallet.sync(&blockchain, SyncOptions::default()).expect("syncing receiver");
-                assert_eq!(receiver_wallet.get_balance().expect("balance"), 49_000, "should have received coins once and only once");
-            }
-
-            #[test]
-            fn test_sync_many_sends_to_a_single_address() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-
-                for _ in 0..4 {
-                    // split this up into multiple blocks so rpc doesn't get angry
-                    for _ in 0..20 {
-                        test_client.receive(testutils! {
-                            @tx ( (@external descriptors, 0) => 1_000 )
-                        });
-                    }
-                    test_client.generate(1, None);
-                }
-
-                // add some to the mempool as well.
-                for _ in 0..20 {
-                    test_client.receive(testutils! {
-                        @tx ( (@external descriptors, 0) => 1_000 )
-                    });
-                }
-
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-
-                assert_eq!(wallet.get_balance().unwrap(), 100_000);
-            }
-
             #[test]
             fn test_update_confirmation_time_after_generate() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                                println!("{}", descriptors.0);
+                let (wallet, descriptors, mut test_client) = init_single_sig();
+                println!("{}", descriptors.0);
                 let node_addr = test_client.get_node_address(None);
 
                 let received_txid = test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let tx_map = wallet.list_transactions(false).unwrap().into_iter().map(|tx| (tx.txid, tx)).collect::<std::collections::HashMap<_, _>>();
@@ -702,7 +615,7 @@ macro_rules! bdk_blockchain_tests {
                 assert!(details.confirmation_time.is_none());
 
                 test_client.generate(1, Some(node_addr));
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
 
                 let tx_map = wallet.list_transactions(false).unwrap().into_iter().map(|tx| (tx.txid, tx)).collect::<std::collections::HashMap<_, _>>();
                 let details = tx_map.get(&received_txid).unwrap();
@@ -712,13 +625,13 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_outgoing_from_scratch() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                                let node_addr = test_client.get_node_address(None);
+                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let node_addr = test_client.get_node_address(None);
                 let received_txid = test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -727,26 +640,25 @@ macro_rules! bdk_blockchain_tests {
 
                 let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                let sent_tx = psbt.extract_tx();
-                blockchain.broadcast(&sent_tx).unwrap();
+                let sent_txid = wallet.broadcast(psbt.extract_tx()).unwrap();
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), details.received, "incorrect balance after receive");
 
                 // empty wallet
-                let wallet = get_wallet_from_descriptors(&descriptors);
+                let wallet = get_wallet_from_descriptors(&descriptors, &test_client);
 
                 #[cfg(feature = "rpc")]  // rpc cannot see mempool tx before importmulti
                 test_client.generate(1, Some(node_addr));
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 let tx_map = wallet.list_transactions(false).unwrap().into_iter().map(|tx| (tx.txid, tx)).collect::<std::collections::HashMap<_, _>>();
 
                 let received = tx_map.get(&received_txid).unwrap();
                 assert_eq!(received.received, 50_000, "incorrect received from receiver");
                 assert_eq!(received.sent, 0, "incorrect sent from receiver");
 
-                let sent = tx_map.get(&sent_tx.txid()).unwrap();
+                let sent = tx_map.get(&sent_txid).unwrap();
                 assert_eq!(sent.received, details.received, "incorrect received from sender");
                 assert_eq!(sent.sent, details.sent, "incorrect sent from sender");
                 assert_eq!(sent.fee.unwrap_or(0), details.fee.unwrap_or(0), "incorrect fees from sender");
@@ -754,14 +666,14 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_long_change_chain() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                                let node_addr = test_client.get_node_address(None);
+                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 )
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let mut total_sent = 0;
@@ -771,38 +683,38 @@ macro_rules! bdk_blockchain_tests {
                     let (mut psbt, details) = builder.finish().unwrap();
                     let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                     assert!(finalized, "Cannot finalize transaction");
-                    blockchain.broadcast(&psbt.extract_tx()).unwrap();
+                    wallet.broadcast(psbt.extract_tx()).unwrap();
 
-                    wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                    wallet.sync(noop_progress(), None).unwrap();
 
                     total_sent += 5_000 + details.fee.unwrap_or(0);
                 }
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000 - total_sent, "incorrect balance after chain");
 
                 // empty wallet
 
-                let wallet = get_wallet_from_descriptors(&descriptors);
+                let wallet = get_wallet_from_descriptors(&descriptors, &test_client);
 
                 #[cfg(feature = "rpc")]  // rpc cannot see mempool tx before importmulti
                 test_client.generate(1, Some(node_addr));
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000 - total_sent, "incorrect balance empty wallet");
 
             }
 
             #[test]
             fn test_sync_bump_fee_basic() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                                let node_addr = test_client.get_node_address(None);
+                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 ) (@confirmations 1)
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -810,18 +722,18 @@ macro_rules! bdk_blockchain_tests {
                 let (mut psbt, details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                blockchain.broadcast(&psbt.extract_tx()).unwrap();
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.broadcast(psbt.extract_tx()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000 - details.fee.unwrap_or(0) - 5_000, "incorrect balance from fees");
                 assert_eq!(wallet.get_balance().unwrap(), details.received, "incorrect balance from received");
 
                 let mut builder = wallet.build_fee_bump(details.txid).unwrap();
                 builder.fee_rate(FeeRate::from_sat_per_vb(2.1));
-                let (mut new_psbt, new_details) = builder.finish().expect("fee bump tx");
+                let (mut new_psbt, new_details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut new_psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                blockchain.broadcast(&new_psbt.extract_tx()).unwrap();
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.broadcast(new_psbt.extract_tx()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000 - new_details.fee.unwrap_or(0) - 5_000, "incorrect balance from fees after bump");
                 assert_eq!(wallet.get_balance().unwrap(), new_details.received, "incorrect balance from received after bump");
 
@@ -830,14 +742,14 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_bump_fee_remove_change() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                                let node_addr = test_client.get_node_address(None);
+                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000 ) (@confirmations 1)
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -845,18 +757,18 @@ macro_rules! bdk_blockchain_tests {
                 let (mut psbt, details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                blockchain.broadcast(&psbt.extract_tx()).unwrap();
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.broadcast(psbt.extract_tx()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 1_000 - details.fee.unwrap_or(0), "incorrect balance after send");
                 assert_eq!(wallet.get_balance().unwrap(), details.received, "incorrect received after send");
 
                 let mut builder = wallet.build_fee_bump(details.txid).unwrap();
-                builder.fee_rate(FeeRate::from_sat_per_vb(5.1));
+                builder.fee_rate(FeeRate::from_sat_per_vb(5.0));
                 let (mut new_psbt, new_details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut new_psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                blockchain.broadcast(&new_psbt.extract_tx()).unwrap();
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.broadcast(new_psbt.extract_tx()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 0, "incorrect balance after change removal");
                 assert_eq!(new_details.received, 0, "incorrect received after change removal");
 
@@ -865,14 +777,14 @@ macro_rules! bdk_blockchain_tests {
 
             #[test]
             fn test_sync_bump_fee_add_input_simple() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                                let node_addr = test_client.get_node_address(None);
+                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000 ) (@confirmations 1)
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 75_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -880,8 +792,8 @@ macro_rules! bdk_blockchain_tests {
                 let (mut psbt, details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                blockchain.broadcast(&psbt.extract_tx()).unwrap();
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.broadcast(psbt.extract_tx()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 26_000 - details.fee.unwrap_or(0), "incorrect balance after send");
                 assert_eq!(details.received, 1_000 - details.fee.unwrap_or(0), "incorrect received after send");
 
@@ -890,22 +802,22 @@ macro_rules! bdk_blockchain_tests {
                 let (mut new_psbt, new_details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut new_psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                blockchain.broadcast(&new_psbt.extract_tx()).unwrap();
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.broadcast(new_psbt.extract_tx()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(new_details.sent, 75_000, "incorrect sent");
                 assert_eq!(wallet.get_balance().unwrap(), new_details.received, "incorrect balance after add input");
             }
 
             #[test]
             fn test_sync_bump_fee_add_input_no_change() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                                let node_addr = test_client.get_node_address(None);
+                let (wallet, descriptors, mut test_client) = init_single_sig();
+                let node_addr = test_client.get_node_address(None);
 
                 test_client.receive(testutils! {
                     @tx ( (@external descriptors, 0) => 50_000, (@external descriptors, 1) => 25_000 ) (@confirmations 1)
                 });
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 75_000, "incorrect balance");
 
                 let mut builder = wallet.build_tx();
@@ -913,8 +825,8 @@ macro_rules! bdk_blockchain_tests {
                 let (mut psbt, details) = builder.finish().unwrap();
                 let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                blockchain.broadcast(&psbt.extract_tx()).unwrap();
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.broadcast(psbt.extract_tx()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 26_000 - details.fee.unwrap_or(0), "incorrect balance after send");
                 assert_eq!(details.received, 1_000 - details.fee.unwrap_or(0), "incorrect received after send");
 
@@ -925,51 +837,20 @@ macro_rules! bdk_blockchain_tests {
 
                 let finalized = wallet.sign(&mut new_psbt, Default::default()).unwrap();
                 assert!(finalized, "Cannot finalize transaction");
-                blockchain.broadcast(&new_psbt.extract_tx()).unwrap();
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.broadcast(new_psbt.extract_tx()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(new_details.sent, 75_000, "incorrect sent");
                 assert_eq!(wallet.get_balance().unwrap(), 0, "incorrect balance after add input");
                 assert_eq!(new_details.received, 0, "incorrect received after add input");
             }
 
-
-            #[test]
-            fn test_add_data() {
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-                                let node_addr = test_client.get_node_address(None);
-                let _ = test_client.receive(testutils! {
-                    @tx ( (@external descriptors, 0) => 50_000 )
-                });
-
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-                assert_eq!(wallet.get_balance().unwrap(), 50_000, "incorrect balance");
-
-                let mut builder = wallet.build_tx();
-                let data = [42u8;80];
-                builder.add_data(&data);
-                let (mut psbt, details) = builder.finish().unwrap();
-
-                let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
-                assert!(finalized, "Cannot finalize transaction");
-                let tx = psbt.extract_tx();
-                let serialized_tx = bitcoin::consensus::encode::serialize(&tx);
-                assert!(serialized_tx.windows(data.len()).any(|e| e==data), "cannot find op_return data in transaction");
-                blockchain.broadcast(&tx).unwrap();
-                test_client.generate(1, Some(node_addr));
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-                assert_eq!(wallet.get_balance().unwrap(), 50_000 - details.fee.unwrap_or(0), "incorrect balance after send");
-
-                let tx_map = wallet.list_transactions(false).unwrap().into_iter().map(|tx| (tx.txid, tx)).collect::<std::collections::HashMap<_, _>>();
-                let _ = tx_map.get(&tx.txid()).unwrap();
-            }
-
             #[test]
             fn test_sync_receive_coinbase() {
-                let (wallet, blockchain, _, mut test_client) = init_single_sig();
+                let (wallet, _, mut test_client) = init_single_sig();
 
                 let wallet_addr = wallet.get_address($crate::wallet::AddressIndex::New).unwrap().address;
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert_eq!(wallet.get_balance().unwrap(), 0, "incorrect balance");
 
                 test_client.generate(1, Some(wallet_addr));
@@ -982,178 +863,9 @@ macro_rules! bdk_blockchain_tests {
                 }
 
 
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
+                wallet.sync(noop_progress(), None).unwrap();
                 assert!(wallet.get_balance().unwrap() > 0, "incorrect balance after receiving coinbase");
             }
-
-            #[test]
-            fn test_send_to_bech32m_addr() {
-                use std::str::FromStr;
-                use serde;
-                use serde_json;
-                use serde::Serialize;
-                use bitcoincore_rpc::jsonrpc::serde_json::Value;
-                use bitcoincore_rpc::{Auth, Client, RpcApi};
-
-                let (wallet, blockchain, descriptors, mut test_client) = init_single_sig();
-
-                // TODO remove once rust-bitcoincore-rpc with PR 199 released
-                // https://github.com/rust-bitcoin/rust-bitcoincore-rpc/pull/199
-                /// Import Descriptor Request
-                #[derive(Serialize, Clone, PartialEq, Eq, Debug)]
-                pub struct ImportDescriptorRequest {
-                    pub active: bool,
-                    #[serde(rename = "desc")]
-                    pub descriptor: String,
-                    pub range: [i64; 2],
-                    pub next_index: i64,
-                    pub timestamp: String,
-                    pub internal: bool,
-                }
-
-                // TODO remove once rust-bitcoincore-rpc with PR 199 released
-                impl ImportDescriptorRequest {
-                    /// Create a new Import Descriptor request providing just the descriptor and internal flags
-                    pub fn new(descriptor: &str, internal: bool) -> Self {
-                        ImportDescriptorRequest {
-                            descriptor: descriptor.to_string(),
-                            internal,
-                            active: true,
-                            range: [0, 100],
-                            next_index: 0,
-                            timestamp: "now".to_string(),
-                        }
-                    }
-                }
-
-                // 1. Create and add descriptors to a test bitcoind node taproot wallet
-
-                // TODO replace once rust-bitcoincore-rpc with PR 174 released
-                // https://github.com/rust-bitcoin/rust-bitcoincore-rpc/pull/174
-                let _createwallet_result: Value = test_client.bitcoind.client.call("createwallet", &["taproot_wallet".into(),false.into(),true.into(),serde_json::to_value("").unwrap(), false.into(), true.into()]).unwrap();
-
-                // TODO replace once bitcoind released with support for rust-bitcoincore-rpc PR 174
-                let taproot_wallet_client = Client::new(&test_client.bitcoind.rpc_url_with_wallet("taproot_wallet"), Auth::CookieFile(test_client.bitcoind.params.cookie_file.clone())).unwrap();
-
-                let wallet_descriptor = "tr(tprv8ZgxMBicQKsPdBtxmEMPnNq58KGusNAimQirKFHqX2yk2D8q1v6pNLiKYVAdzDHy2w3vF4chuGfMvNtzsbTTLVXBcdkCA1rje1JG6oksWv8/86h/1h/0h/0/*)#y283ssmn";
-                let change_descriptor = "tr(tprv8ZgxMBicQKsPdBtxmEMPnNq58KGusNAimQirKFHqX2yk2D8q1v6pNLiKYVAdzDHy2w3vF4chuGfMvNtzsbTTLVXBcdkCA1rje1JG6oksWv8/86h/1h/0h/1/*)#47zsd9tt";
-
-                let tr_descriptors = vec![
-                            ImportDescriptorRequest::new(wallet_descriptor, false),
-                            ImportDescriptorRequest::new(change_descriptor, false),
-                        ];
-
-                // TODO replace once rust-bitcoincore-rpc with PR 199 released
-                let _import_result: Value = taproot_wallet_client.call("importdescriptors", &[serde_json::to_value(tr_descriptors).unwrap()]).unwrap();
-
-                // 2. Get a new bech32m address from test bitcoind node taproot wallet
-
-                // TODO replace once rust-bitcoincore-rpc with PR 199 released
-                let node_addr: bitcoin::Address = taproot_wallet_client.call("getnewaddress", &["test address".into(), "bech32m".into()]).unwrap();
-                assert_eq!(node_addr, bitcoin::Address::from_str("bcrt1pj5y3f0fu4y7g98k4v63j9n0xvj3lmln0cpwhsjzknm6nt0hr0q7qnzwsy9").unwrap());
-
-                // 3. Send 50_000 sats from test bitcoind node to test BDK wallet
-
-                test_client.receive(testutils! {
-                    @tx ( (@external descriptors, 0) => 50_000 )
-                });
-
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-                assert_eq!(wallet.get_balance().unwrap(), 50_000, "wallet has incorrect balance");
-
-                // 4. Send 25_000 sats from test BDK wallet to test bitcoind node taproot wallet
-
-                let mut builder = wallet.build_tx();
-                builder.add_recipient(node_addr.script_pubkey(), 25_000);
-                let (mut psbt, details) = builder.finish().unwrap();
-                let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
-                assert!(finalized, "wallet cannot finalize transaction");
-                let tx = psbt.extract_tx();
-                blockchain.broadcast(&tx).unwrap();
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-                assert_eq!(wallet.get_balance().unwrap(), details.received, "wallet has incorrect balance after send");
-                assert_eq!(wallet.list_transactions(false).unwrap().len(), 2, "wallet has incorrect number of txs");
-                assert_eq!(wallet.list_unspent().unwrap().len(), 1, "wallet has incorrect number of unspents");
-                test_client.generate(1, None);
-
-                // 5. Verify 25_000 sats are received by test bitcoind node taproot wallet
-
-                let taproot_balance = taproot_wallet_client.get_balance(None, None).unwrap();
-                assert_eq!(taproot_balance.as_sat(), 25_000, "node has incorrect taproot wallet balance");
-            }
-
-            #[test]
-            fn test_tx_chain() {
-                use bitcoincore_rpc::RpcApi;
-                use bitcoin::consensus::encode::deserialize;
-                use $crate::wallet::AddressIndex;
-
-                // Here we want to test that we set correctly the send and receive
-                // fields in the transaction object. For doing so, we create two
-                // different txs, the second one spending from the first:
-                // 1.
-                // Core (#1) -> Core (#2)
-                //           -> Us   (#3)
-                // 2.
-                // Core (#2) -> Us   (#4)
-
-                let (wallet, blockchain, _, mut test_client) = init_single_sig();
-                let bdk_address = wallet.get_address(AddressIndex::New).unwrap().address;
-                let core_address = test_client.get_new_address(None, None).unwrap();
-                let tx = testutils! {
-                    @tx ( (@addr bdk_address.clone()) => 50_000, (@addr core_address.clone()) => 40_000 )
-                };
-
-                // Tx one: from Core #1 to Core #2 and Us #3.
-                let txid_1 = test_client.receive(tx);
-                let tx_1: Transaction = deserialize(&test_client.get_transaction(&txid_1, None).unwrap().hex).unwrap();
-                let vout_1 = tx_1.output.into_iter().position(|o| o.script_pubkey == core_address.script_pubkey()).unwrap() as u32;
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-                let tx_1 = wallet.list_transactions(false).unwrap().into_iter().find(|tx| tx.txid == txid_1).unwrap();
-                assert_eq!(tx_1.received, 50_000);
-                assert_eq!(tx_1.sent, 0);
-
-                // Tx two: from Core #2 to Us #4.
-                let tx = testutils! {
-                    @tx ( (@addr bdk_address) => 10_000 ) ( @inputs (txid_1,vout_1))
-                };
-                let txid_2 = test_client.receive(tx);
-
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-                let tx_2 = wallet.list_transactions(false).unwrap().into_iter().find(|tx| tx.txid == txid_2).unwrap();
-                assert_eq!(tx_2.received, 10_000);
-                assert_eq!(tx_2.sent, 0);
-            }
-
-            #[test]
-            fn test_send_receive_pkh() {
-                let descriptors = ("pkh(cVpPVruEDdmutPzisEsYvtST1usBR3ntr8pXSyt6D2YYqXRyPcFW)".to_string(), None);
-                let mut test_client = TestClient::default();
-                let blockchain = get_blockchain(&test_client);
-
-                let wallet = get_wallet_from_descriptors(&descriptors);
-                #[cfg(feature = "test-rpc")]
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-
-                let _ = test_client.receive(testutils! {
-                    @tx ( (@external descriptors, 0)   => 50_000 )
-                });
-
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-
-                assert_eq!(wallet.get_balance().unwrap(), 50_000);
-
-                let tx = {
-                    let mut builder = wallet.build_tx();
-                    builder.add_recipient(test_client.get_node_address(None).script_pubkey(), 25_000);
-                    let (mut psbt, _details) = builder.finish().unwrap();
-                    wallet.sign(&mut psbt, Default::default()).unwrap();
-                    psbt.extract_tx()
-                };
-                blockchain.broadcast(&tx).unwrap();
-
-                wallet.sync(&blockchain, SyncOptions::default()).unwrap();
-            }
         }
     };
 

src/testutils/mod.rs

@@ -15,37 +15,11 @@
 pub mod blockchain_tests;
 
 use bitcoin::secp256k1::{Secp256k1, Verification};
-use bitcoin::{Address, PublicKey, Txid};
+use bitcoin::{Address, PublicKey};
 
 use miniscript::descriptor::DescriptorPublicKey;
 use miniscript::{Descriptor, MiniscriptKey, TranslatePk};
 
-#[derive(Clone, Debug)]
-pub struct TestIncomingInput {
-    pub txid: Txid,
-    pub vout: u32,
-    pub sequence: Option<u32>,
-}
-
-impl TestIncomingInput {
-    pub fn new(txid: Txid, vout: u32, sequence: Option<u32>) -> Self {
-        Self {
-            txid,
-            vout,
-            sequence,
-        }
-    }
-
-    #[cfg(feature = "test-blockchains")]
-    pub fn into_raw_tx_input(self) -> bitcoincore_rpc::json::CreateRawTransactionInput {
-        bitcoincore_rpc::json::CreateRawTransactionInput {
-            txid: self.txid,
-            vout: self.vout,
-            sequence: self.sequence,
-        }
-    }
-}
-
 #[derive(Clone, Debug)]
 pub struct TestIncomingOutput {
     pub value: u64,
@@ -63,7 +37,6 @@ impl TestIncomingOutput {
 
 #[derive(Clone, Debug)]
 pub struct TestIncomingTx {
-    pub input: Vec<TestIncomingInput>,
     pub output: Vec<TestIncomingOutput>,
     pub min_confirmations: Option<u64>,
     pub locktime: Option<i64>,
@@ -72,14 +45,12 @@ pub struct TestIncomingTx {
 
 impl TestIncomingTx {
     pub fn new(
-        input: Vec<TestIncomingInput>,
         output: Vec<TestIncomingOutput>,
         min_confirmations: Option<u64>,
         locktime: Option<i64>,
         replaceable: Option<bool>,
     ) -> Self {
         Self {
-            input,
             output,
             min_confirmations,
             locktime,
@@ -87,10 +58,6 @@ impl TestIncomingTx {
         }
     }
 
-    pub fn add_input(&mut self, input: TestIncomingInput) {
-        self.input.push(input);
-    }
-
     pub fn add_output(&mut self, output: TestIncomingOutput) {
         self.output.push(output);
     }
@@ -133,8 +100,8 @@ impl TranslateDescriptor for Descriptor<DescriptorPublicKey> {
 #[macro_export]
 macro_rules! testutils {
     ( @external $descriptors:expr, $child:expr ) => ({
-        use $crate::bitcoin::secp256k1::Secp256k1;
-        use $crate::miniscript::descriptor::{Descriptor, DescriptorPublicKey, DescriptorTrait};
+        use bitcoin::secp256k1::Secp256k1;
+        use miniscript::descriptor::{Descriptor, DescriptorPublicKey, DescriptorTrait};
 
         use $crate::testutils::TranslateDescriptor;
 
@@ -144,33 +111,28 @@ macro_rules! testutils {
         parsed.derive_translated(&secp, $child).address(bitcoin::Network::Regtest).expect("No address form")
     });
     ( @internal $descriptors:expr, $child:expr ) => ({
-        use $crate::bitcoin::secp256k1::Secp256k1;
-        use $crate::miniscript::descriptor::{Descriptor, DescriptorPublicKey, DescriptorTrait};
+        use bitcoin::secp256k1::Secp256k1;
+        use miniscript::descriptor::{Descriptor, DescriptorPublicKey, DescriptorTrait};
 
         use $crate::testutils::TranslateDescriptor;
 
         let secp = Secp256k1::new();
 
         let parsed = Descriptor::<DescriptorPublicKey>::parse_descriptor(&secp, &$descriptors.1.expect("Missing internal descriptor")).expect("Failed to parse descriptor in `testutils!(@internal)`").0;
-        parsed.derive_translated(&secp, $child).address($crate::bitcoin::Network::Regtest).expect("No address form")
+        parsed.derive_translated(&secp, $child).address(bitcoin::Network::Regtest).expect("No address form")
     });
     ( @e $descriptors:expr, $child:expr ) => ({ testutils!(@external $descriptors, $child) });
     ( @i $descriptors:expr, $child:expr ) => ({ testutils!(@internal $descriptors, $child) });
-    ( @addr $addr:expr ) => ({ $addr });
 
-    ( @tx ( $( ( $( $addr:tt )* ) => $amount:expr ),+ ) $( ( @inputs $( ($txid:expr, $vout:expr) ),+ ) )? $( ( @locktime $locktime:expr ) )? $( ( @confirmations $confirmations:expr ) )? $( ( @replaceable $replaceable:expr ) )? ) => ({
+    ( @tx ( $( ( $( $addr:tt )* ) => $amount:expr ),+ ) $( ( @locktime $locktime:expr ) )? $( ( @confirmations $confirmations:expr ) )? $( ( @replaceable $replaceable:expr ) )? ) => ({
         let outs = vec![$( $crate::testutils::TestIncomingOutput::new($amount, testutils!( $($addr)* ))),+];
-        let _ins: Vec<$crate::testutils::TestIncomingInput> = vec![];
-        $(
-            let _ins = vec![$( $crate::testutils::TestIncomingInput { txid: $txid, vout: $vout, sequence: None }),+];
-        )?
 
         let locktime = None::<i64>$(.or(Some($locktime)))?;
 
         let min_confirmations = None::<u64>$(.or(Some($confirmations)))?;
         let replaceable = None::<bool>$(.or(Some($replaceable)))?;
 
-        $crate::testutils::TestIncomingTx::new(_ins, outs, min_confirmations, locktime, replaceable)
+        $crate::testutils::TestIncomingTx::new(outs, min_confirmations, locktime, replaceable)
     });
 
     ( @literal $key:expr ) => ({
@@ -183,8 +145,8 @@ macro_rules! testutils {
         let mut seed = [0u8; 32];
         rand::thread_rng().fill(&mut seed[..]);
 
-        let key = $crate::bitcoin::util::bip32::ExtendedPrivKey::new_master(
-            $crate::bitcoin::Network::Testnet,
+        let key = bitcoin::util::bip32::ExtendedPrivKey::new_master(
+            bitcoin::Network::Testnet,
             &seed,
         );
 
@@ -196,13 +158,13 @@ macro_rules! testutils {
     ( @generate_wif ) => ({
         use rand::Rng;
 
-        let mut key = [0u8; $crate::bitcoin::secp256k1::constants::SECRET_KEY_SIZE];
+        let mut key = [0u8; bitcoin::secp256k1::constants::SECRET_KEY_SIZE];
         rand::thread_rng().fill(&mut key[..]);
 
-        ($crate::bitcoin::PrivateKey {
+        (bitcoin::PrivateKey {
             compressed: true,
-            network: $crate::bitcoin::Network::Testnet,
-            key: $crate::bitcoin::secp256k1::SecretKey::from_slice(&key).unwrap(),
+            network: bitcoin::Network::Testnet,
+            key: bitcoin::secp256k1::SecretKey::from_slice(&key).unwrap(),
         }.to_string(), None::<String>, None::<String>)
     });
 
@@ -219,8 +181,8 @@ macro_rules! testutils {
     ( @descriptors ( $external_descriptor:expr ) $( ( $internal_descriptor:expr ) )? $( ( @keys $( $keys:tt )* ) )* ) => ({
         use std::str::FromStr;
         use std::collections::HashMap;
-        use $crate::miniscript::descriptor::Descriptor;
-        use $crate::miniscript::TranslatePk;
+        use miniscript::descriptor::Descriptor;
+        use miniscript::TranslatePk;
 
         #[allow(unused_assignments, unused_mut)]
         let mut keys: HashMap<&'static str, (String, Option<String>, Option<String>)> = HashMap::new();

src/types.rs

@@ -10,7 +10,6 @@
 // licenses.
 
 use std::convert::AsRef;
-use std::ops::Sub;
 
 use bitcoin::blockdata::transaction::{OutPoint, Transaction, TxOut};
 use bitcoin::{hash_types::Txid, util::psbt};
@@ -66,31 +65,10 @@ impl FeeRate {
         FeeRate(1.0)
     }
 
-    /// Calculate fee rate from `fee` and weight units (`wu`).
-    pub fn from_wu(fee: u64, wu: usize) -> FeeRate {
-        Self::from_vb(fee, wu.vbytes())
-    }
-
-    /// Calculate fee rate from `fee` and `vbytes`.
-    pub fn from_vb(fee: u64, vbytes: usize) -> FeeRate {
-        let rate = fee as f32 / vbytes as f32;
-        Self::from_sat_per_vb(rate)
-    }
-
     /// Return the value as satoshi/vbyte
     pub fn as_sat_vb(&self) -> f32 {
         self.0
     }
-
-    /// Calculate absolute fee in Satoshis using size in weight units.
-    pub fn fee_wu(&self, wu: usize) -> u64 {
-        self.fee_vb(wu.vbytes())
-    }
-
-    /// Calculate absolute fee in Satoshis using size in virtual bytes.
-    pub fn fee_vb(&self, vbytes: usize) -> u64 {
-        (self.as_sat_vb() * vbytes as f32).ceil() as u64
-    }
 }
 
 impl std::default::Default for FeeRate {
@@ -99,27 +77,6 @@ impl std::default::Default for FeeRate {
     }
 }
 
-impl Sub for FeeRate {
-    type Output = Self;
-
-    fn sub(self, other: FeeRate) -> Self::Output {
-        FeeRate(self.0 - other.0)
-    }
-}
-
-/// Trait implemented by types that can be used to measure weight units.
-pub trait Vbytes {
-    /// Convert weight units to virtual bytes.
-    fn vbytes(self) -> usize;
-}
-
-impl Vbytes for usize {
-    fn vbytes(self) -> usize {
-        // ref: https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki#transaction-size-calculations
-        (self as f32 / 4.0).ceil() as usize
-    }
-}
-
 /// An unspent output owned by a [`Wallet`].
 ///
 /// [`Wallet`]: crate::Wallet
@@ -210,29 +167,32 @@ pub struct TransactionDetails {
     pub fee: Option<u64>,
     /// If the transaction is confirmed, contains height and timestamp of the block containing the
     /// transaction, unconfirmed transaction contains `None`.
-    pub confirmation_time: Option<BlockTime>,
+    pub confirmation_time: Option<ConfirmationTime>,
+    /// Whether the tx has been verified against the consensus rules
+    ///
+    /// Confirmed txs are considered "verified" by default, while unconfirmed txs are checked to
+    /// ensure an unstrusted [`Blockchain`](crate::blockchain::Blockchain) backend can't trick the
+    /// wallet into using an invalid tx as an RBF template.
+    ///
+    /// The check is only perfomed when the `verify` feature is enabled.
+    #[serde(default = "bool::default")] // default to `false` if not specified
+    pub verified: bool,
 }
 
-/// Block height and timestamp of a block
+/// Block height and timestamp of the block containing the confirmed transaction
 #[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, Default)]
-pub struct BlockTime {
+pub struct ConfirmationTime {
     /// confirmation block height
     pub height: u32,
     /// confirmation block timestamp
     pub timestamp: u64,
 }
 
-/// **DEPRECATED**: Confirmation time of a transaction
-///
-/// The structure has been renamed to `BlockTime`
-#[deprecated(note = "This structure has been renamed to `BlockTime`")]
-pub type ConfirmationTime = BlockTime;
-
-impl BlockTime {
-    /// Returns `Some` `BlockTime` if both `height` and `timestamp` are `Some`
+impl ConfirmationTime {
+    /// Returns `Some` `ConfirmationTime` if both `height` and `timestamp` are `Some`
     pub fn new(height: Option<u32>, timestamp: Option<u64>) -> Option<Self> {
         match (height, timestamp) {
-            (Some(height), Some(timestamp)) => Some(BlockTime { height, timestamp }),
+            (Some(height), Some(timestamp)) => Some(ConfirmationTime { height, timestamp }),
             _ => None,
         }
     }

src/wallet/address_validator.rs

@@ -55,7 +55,7 @@
 //! }
 //!
 //! let descriptor = "wpkh(tpubD6NzVbkrYhZ4Xferm7Pz4VnjdcDPFyjVu5K4iZXQ4pVN8Cks4pHVowTBXBKRhX64pkRyJZJN5xAKj4UDNnLPb5p2sSKXhewoYx5GbTdUFWq/*)";
-//! let mut wallet = Wallet::new(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
+//! let mut wallet = Wallet::new_offline(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
 //! wallet.add_address_validator(Arc::new(PrintAddressAndContinue));
 //!
 //! let address = wallet.get_address(New)?;
@@ -115,8 +115,8 @@ mod test {
     use std::sync::Arc;
 
     use super::*;
+    use crate::wallet::test::{get_funded_wallet, get_test_wpkh};
     use crate::wallet::AddressIndex::New;
-    use crate::wallet::{get_funded_wallet, test::get_test_wpkh};
 
     #[derive(Debug)]
     struct TestValidator;

src/wallet/coin_selection.rs

@@ -26,7 +26,7 @@
 //! ```
 //! # use std::str::FromStr;
 //! # use bitcoin::*;
-//! # use bdk::wallet::{self, coin_selection::*};
+//! # use bdk::wallet::coin_selection::*;
 //! # use bdk::database::Database;
 //! # use bdk::*;
 //! # const TXIN_BASE_WEIGHT: usize = (32 + 4 + 4 + 1) * 4;
@@ -41,7 +41,7 @@
 //!         optional_utxos: Vec<WeightedUtxo>,
 //!         fee_rate: FeeRate,
 //!         amount_needed: u64,
-//!         fee_amount: u64,
+//!         fee_amount: f32,
 //!     ) -> Result<CoinSelectionResult, bdk::Error> {
 //!         let mut selected_amount = 0;
 //!         let mut additional_weight = 0;
@@ -57,8 +57,9 @@
 //!                 },
 //!             )
 //!             .collect::<Vec<_>>();
-//!         let additional_fees = fee_rate.fee_wu(additional_weight);
-//!         let amount_needed_with_fees = (fee_amount + additional_fees) + amount_needed;
+//!         let additional_fees = additional_weight as f32 * fee_rate.as_sat_vb() / 4.0;
+//!         let amount_needed_with_fees =
+//!             (fee_amount + additional_fees).ceil() as u64 + amount_needed;
 //!         if amount_needed_with_fees > selected_amount {
 //!             return Err(bdk::Error::InsufficientFunds {
 //!                 needed: amount_needed_with_fees,
@@ -89,6 +90,7 @@
 //! ```
 
 use crate::types::FeeRate;
+use crate::wallet::Vbytes;
 use crate::{database::Database, WeightedUtxo};
 use crate::{error::Error, Utxo};
 
@@ -116,7 +118,7 @@ pub struct CoinSelectionResult {
     /// List of outputs selected for use as inputs
     pub selected: Vec<Utxo>,
     /// Total fee amount in satoshi
-    pub fee_amount: u64,
+    pub fee_amount: f32,
 }
 
 impl CoinSelectionResult {
@@ -163,7 +165,7 @@ pub trait CoinSelectionAlgorithm<D: Database>: std::fmt::Debug {
         optional_utxos: Vec<WeightedUtxo>,
         fee_rate: FeeRate,
         amount_needed: u64,
-        fee_amount: u64,
+        fee_amount: f32,
     ) -> Result<CoinSelectionResult, Error>;
 }
 
@@ -182,8 +184,10 @@ impl<D: Database> CoinSelectionAlgorithm<D> for LargestFirstCoinSelection {
         mut optional_utxos: Vec<WeightedUtxo>,
         fee_rate: FeeRate,
         amount_needed: u64,
-        mut fee_amount: u64,
+        mut fee_amount: f32,
     ) -> Result<CoinSelectionResult, Error> {
+        let calc_fee_bytes = |wu| (wu as f32) * fee_rate.as_sat_vb() / 4.0;
+
         log::debug!(
             "amount_needed = `{}`, fee_amount = `{}`, fee_rate = `{:?}`",
             amount_needed,
@@ -208,9 +212,9 @@ impl<D: Database> CoinSelectionAlgorithm<D> for LargestFirstCoinSelection {
             .scan(
                 (&mut selected_amount, &mut fee_amount),
                 |(selected_amount, fee_amount), (must_use, weighted_utxo)| {
-                    if must_use || **selected_amount < amount_needed + **fee_amount {
+                    if must_use || **selected_amount < amount_needed + (fee_amount.ceil() as u64) {
                         **fee_amount +=
-                            fee_rate.fee_wu(TXIN_BASE_WEIGHT + weighted_utxo.satisfaction_weight);
+                            calc_fee_bytes(TXIN_BASE_WEIGHT + weighted_utxo.satisfaction_weight);
                         **selected_amount += weighted_utxo.utxo.txout().value;
 
                         log::debug!(
@@ -227,7 +231,7 @@ impl<D: Database> CoinSelectionAlgorithm<D> for LargestFirstCoinSelection {
             )
             .collect::<Vec<_>>();
 
-        let amount_needed_with_fees = amount_needed + fee_amount;
+        let amount_needed_with_fees = amount_needed + (fee_amount.ceil() as u64);
         if selected_amount < amount_needed_with_fees {
             return Err(Error::InsufficientFunds {
                 needed: amount_needed_with_fees,
@@ -247,15 +251,16 @@ impl<D: Database> CoinSelectionAlgorithm<D> for LargestFirstCoinSelection {
 struct OutputGroup {
     weighted_utxo: WeightedUtxo,
     // Amount of fees for spending a certain utxo, calculated using a certain FeeRate
-    fee: u64,
+    fee: f32,
     // The effective value of the UTXO, i.e., the utxo value minus the fee for spending it
     effective_value: i64,
 }
 
 impl OutputGroup {
     fn new(weighted_utxo: WeightedUtxo, fee_rate: FeeRate) -> Self {
-        let fee = fee_rate.fee_wu(TXIN_BASE_WEIGHT + weighted_utxo.satisfaction_weight);
-        let effective_value = weighted_utxo.utxo.txout().value as i64 - fee as i64;
+        let fee =
+            (TXIN_BASE_WEIGHT + weighted_utxo.satisfaction_weight).vbytes() * fee_rate.as_sat_vb();
+        let effective_value = weighted_utxo.utxo.txout().value as i64 - fee.ceil() as i64;
         OutputGroup {
             weighted_utxo,
             fee,
@@ -298,7 +303,7 @@ impl<D: Database> CoinSelectionAlgorithm<D> for BranchAndBoundCoinSelection {
         optional_utxos: Vec<WeightedUtxo>,
         fee_rate: FeeRate,
         amount_needed: u64,
-        fee_amount: u64,
+        fee_amount: f32,
     ) -> Result<CoinSelectionResult, Error> {
         // Mapping every (UTXO, usize) to an output group
         let required_utxos: Vec<OutputGroup> = required_utxos
@@ -320,7 +325,7 @@ impl<D: Database> CoinSelectionAlgorithm<D> for BranchAndBoundCoinSelection {
             .iter()
             .fold(0, |acc, x| acc + x.effective_value);
 
-        let actual_target = fee_amount + amount_needed;
+        let actual_target = fee_amount.ceil() as u64 + amount_needed;
         let cost_of_change = self.size_of_change as f32 * fee_rate.as_sat_vb();
 
         let expected = (curr_available_value + curr_value)
@@ -340,14 +345,6 @@ impl<D: Database> CoinSelectionAlgorithm<D> for BranchAndBoundCoinSelection {
             .try_into()
             .expect("Bitcoin amount to fit into i64");
 
-        if curr_value > actual_target {
-            return Ok(BranchAndBoundCoinSelection::calculate_cs_result(
-                vec![],
-                required_utxos,
-                fee_amount,
-            ));
-        }
-
         Ok(self
             .bnb(
                 required_utxos.clone(),
@@ -372,7 +369,7 @@ impl<D: Database> CoinSelectionAlgorithm<D> for BranchAndBoundCoinSelection {
 
 impl BranchAndBoundCoinSelection {
     // TODO: make this more Rust-onic :)
-    // (And perhaps refactor with less arguments?)
+    // (And perhpaps refactor with less arguments?)
     #[allow(clippy::too_many_arguments)]
     fn bnb(
         &self,
@@ -381,7 +378,7 @@ impl BranchAndBoundCoinSelection {
         mut curr_value: i64,
         mut curr_available_value: i64,
         actual_target: i64,
-        fee_amount: u64,
+        fee_amount: f32,
         cost_of_change: f32,
     ) -> Result<CoinSelectionResult, Error> {
         // current_selection[i] will contain true if we are using optional_utxos[i],
@@ -489,7 +486,7 @@ impl BranchAndBoundCoinSelection {
         mut optional_utxos: Vec<OutputGroup>,
         curr_value: i64,
         actual_target: i64,
-        fee_amount: u64,
+        fee_amount: f32,
     ) -> CoinSelectionResult {
         #[cfg(not(test))]
         optional_utxos.shuffle(&mut thread_rng());
@@ -518,10 +515,10 @@ impl BranchAndBoundCoinSelection {
     fn calculate_cs_result(
         mut selected_utxos: Vec<OutputGroup>,
         mut required_utxos: Vec<OutputGroup>,
-        mut fee_amount: u64,
+        mut fee_amount: f32,
     ) -> CoinSelectionResult {
         selected_utxos.append(&mut required_utxos);
-        fee_amount += selected_utxos.iter().map(|u| u.fee).sum::<u64>();
+        fee_amount += selected_utxos.iter().map(|u| u.fee).sum::<f32>();
         let selected = selected_utxos
             .into_iter()
             .map(|u| u.weighted_utxo.utxo)
@@ -543,7 +540,6 @@ mod test {
     use super::*;
     use crate::database::MemoryDatabase;
     use crate::types::*;
-    use crate::wallet::Vbytes;
 
     use rand::rngs::StdRng;
     use rand::seq::SliceRandom;
@@ -551,33 +547,52 @@ mod test {
 
     const P2WPKH_WITNESS_SIZE: usize = 73 + 33 + 2;
 
-    const FEE_AMOUNT: u64 = 50;
-
-    fn utxo(value: u64, index: u32) -> WeightedUtxo {
-        assert!(index < 10);
-        let outpoint = OutPoint::from_str(&format!(
-            "000000000000000000000000000000000000000000000000000000000000000{}:0",
-            index
-        ))
-        .unwrap();
-        WeightedUtxo {
-            satisfaction_weight: P2WPKH_WITNESS_SIZE,
-            utxo: Utxo::Local(LocalUtxo {
-                outpoint,
-                txout: TxOut {
-                    value,
-                    script_pubkey: Script::new(),
-                },
-                keychain: KeychainKind::External,
-            }),
-        }
-    }
+    const FEE_AMOUNT: f32 = 50.0;
 
     fn get_test_utxos() -> Vec<WeightedUtxo> {
         vec![
-            utxo(100_000, 0),
-            utxo(FEE_AMOUNT as u64 - 40, 1),
-            utxo(200_000, 2),
+            WeightedUtxo {
+                satisfaction_weight: P2WPKH_WITNESS_SIZE,
+                utxo: Utxo::Local(LocalUtxo {
+                    outpoint: OutPoint::from_str(
+                        "0000000000000000000000000000000000000000000000000000000000000000:0",
+                    )
+                    .unwrap(),
+                    txout: TxOut {
+                        value: 100_000,
+                        script_pubkey: Script::new(),
+                    },
+                    keychain: KeychainKind::External,
+                }),
+            },
+            WeightedUtxo {
+                satisfaction_weight: P2WPKH_WITNESS_SIZE,
+                utxo: Utxo::Local(LocalUtxo {
+                    outpoint: OutPoint::from_str(
+                        "0000000000000000000000000000000000000000000000000000000000000001:0",
+                    )
+                    .unwrap(),
+                    txout: TxOut {
+                        value: FEE_AMOUNT as u64 - 40,
+                        script_pubkey: Script::new(),
+                    },
+                    keychain: KeychainKind::External,
+                }),
+            },
+            WeightedUtxo {
+                satisfaction_weight: P2WPKH_WITNESS_SIZE,
+                utxo: Utxo::Local(LocalUtxo {
+                    outpoint: OutPoint::from_str(
+                        "0000000000000000000000000000000000000000000000000000000000000002:0",
+                    )
+                    .unwrap(),
+                    txout: TxOut {
+                        value: 200_000,
+                        script_pubkey: Script::new(),
+                    },
+                    keychain: KeychainKind::Internal,
+                }),
+            },
         ]
     }
 
@@ -641,13 +656,13 @@ mod test {
                 vec![],
                 FeeRate::from_sat_per_vb(1.0),
                 250_000,
-                FEE_AMOUNT,
+                50.0,
             )
             .unwrap();
 
         assert_eq!(result.selected.len(), 3);
         assert_eq!(result.selected_amount(), 300_010);
-        assert_eq!(result.fee_amount, 254)
+        assert!((result.fee_amount - 254.0).abs() < f32::EPSILON);
     }
 
     #[test]
@@ -662,13 +677,13 @@ mod test {
                 vec![],
                 FeeRate::from_sat_per_vb(1.0),
                 20_000,
-                FEE_AMOUNT,
+                50.0,
             )
             .unwrap();
 
         assert_eq!(result.selected.len(), 3);
         assert_eq!(result.selected_amount(), 300_010);
-        assert_eq!(result.fee_amount, 254);
+        assert!((result.fee_amount - 254.0).abs() < f32::EPSILON);
     }
 
     #[test]
@@ -683,13 +698,13 @@ mod test {
                 utxos,
                 FeeRate::from_sat_per_vb(1.0),
                 20_000,
-                FEE_AMOUNT,
+                50.0,
             )
             .unwrap();
 
         assert_eq!(result.selected.len(), 1);
         assert_eq!(result.selected_amount(), 200_000);
-        assert_eq!(result.fee_amount, 118);
+        assert!((result.fee_amount - 118.0).abs() < f32::EPSILON);
     }
 
     #[test]
@@ -705,7 +720,7 @@ mod test {
                 utxos,
                 FeeRate::from_sat_per_vb(1.0),
                 500_000,
-                FEE_AMOUNT,
+                50.0,
             )
             .unwrap();
     }
@@ -723,7 +738,7 @@ mod test {
                 utxos,
                 FeeRate::from_sat_per_vb(1000.0),
                 250_000,
-                FEE_AMOUNT,
+                50.0,
             )
             .unwrap();
     }
@@ -743,13 +758,13 @@ mod test {
                 utxos,
                 FeeRate::from_sat_per_vb(1.0),
                 250_000,
-                FEE_AMOUNT,
+                50.0,
             )
             .unwrap();
 
         assert_eq!(result.selected.len(), 3);
         assert_eq!(result.selected_amount(), 300_000);
-        assert_eq!(result.fee_amount, 254);
+        assert!((result.fee_amount - 254.0).abs() < f32::EPSILON);
     }
 
     #[test]
@@ -770,7 +785,7 @@ mod test {
 
         assert_eq!(result.selected.len(), 3);
         assert_eq!(result.selected_amount(), 300_010);
-        assert_eq!(result.fee_amount, 254);
+        assert!((result.fee_amount - 254.0).abs() < f32::EPSILON);
     }
 
     #[test]
@@ -791,38 +806,7 @@ mod test {
 
         assert_eq!(result.selected.len(), 3);
         assert_eq!(result.selected_amount(), 300010);
-        assert_eq!(result.fee_amount, 254);
-    }
-
-    #[test]
-    fn test_bnb_coin_selection_required_not_enough() {
-        let utxos = get_test_utxos();
-        let database = MemoryDatabase::default();
-
-        let required = vec![utxos[0].clone()];
-        let mut optional = utxos[1..].to_vec();
-        optional.push(utxo(500_000, 3));
-
-        // Defensive assertions, for sanity and in case someone changes the test utxos vector.
-        let amount: u64 = required.iter().map(|u| u.utxo.txout().value).sum();
-        assert_eq!(amount, 100_000);
-        let amount: u64 = optional.iter().map(|u| u.utxo.txout().value).sum();
-        assert!(amount > 150_000);
-
-        let result = BranchAndBoundCoinSelection::default()
-            .coin_select(
-                &database,
-                required,
-                optional,
-                FeeRate::from_sat_per_vb(1.0),
-                150_000,
-                FEE_AMOUNT,
-            )
-            .unwrap();
-
-        assert_eq!(result.selected.len(), 3);
-        assert_eq!(result.selected_amount(), 300_010);
-        assert!((result.fee_amount as f32 - 254.0).abs() < f32::EPSILON);
+        assert!((result.fee_amount - 254.0).abs() < f32::EPSILON);
     }
 
     #[test]
@@ -838,7 +822,7 @@ mod test {
                 utxos,
                 FeeRate::from_sat_per_vb(1.0),
                 500_000,
-                FEE_AMOUNT,
+                50.0,
             )
             .unwrap();
     }
@@ -856,7 +840,7 @@ mod test {
                 utxos,
                 FeeRate::from_sat_per_vb(1000.0),
                 250_000,
-                FEE_AMOUNT,
+                50.0,
             )
             .unwrap();
     }
@@ -873,7 +857,7 @@ mod test {
                 utxos,
                 FeeRate::from_sat_per_vb(1.0),
                 99932, // first utxo's effective value
-                0,
+                0.0,
             )
             .unwrap();
 
@@ -881,7 +865,7 @@ mod test {
         assert_eq!(result.selected_amount(), 100_000);
         let input_size = (TXIN_BASE_WEIGHT + P2WPKH_WITNESS_SIZE).vbytes();
         let epsilon = 0.5;
-        assert!((1.0 - (result.fee_amount as f32 / input_size as f32)).abs() < epsilon);
+        assert!((1.0 - (result.fee_amount / input_size)).abs() < epsilon);
     }
 
     #[test]
@@ -900,7 +884,7 @@ mod test {
                     optional_utxos,
                     FeeRate::from_sat_per_vb(0.0),
                     target_amount,
-                    0,
+                    0.0,
                 )
                 .unwrap();
             assert_eq!(result.selected_amount(), target_amount);
@@ -927,7 +911,7 @@ mod test {
                 0,
                 curr_available_value,
                 20_000,
-                FEE_AMOUNT,
+                50.0,
                 cost_of_change,
             )
             .unwrap();
@@ -954,7 +938,7 @@ mod test {
                 0,
                 curr_available_value,
                 20_000,
-                FEE_AMOUNT,
+                50.0,
                 cost_of_change,
             )
             .unwrap();
@@ -966,6 +950,7 @@ mod test {
         let fee_rate = FeeRate::from_sat_per_vb(1.0);
         let size_of_change = 31;
         let cost_of_change = size_of_change as f32 * fee_rate.as_sat_vb();
+        let fee_amount = 50.0;
 
         let utxos: Vec<_> = generate_same_value_utxos(50_000, 10)
             .into_iter()
@@ -987,12 +972,12 @@ mod test {
                 curr_value,
                 curr_available_value,
                 target_amount,
-                FEE_AMOUNT,
+                fee_amount,
                 cost_of_change,
             )
             .unwrap();
+        assert!((result.fee_amount - 186.0).abs() < f32::EPSILON);
         assert_eq!(result.selected_amount(), 100_000);
-        assert_eq!(result.fee_amount, 186);
     }
 
     // TODO: bnb() function should be optimized, and this test should be done with more utxos
@@ -1024,7 +1009,7 @@ mod test {
                     curr_value,
                     curr_available_value,
                     target_amount,
-                    0,
+                    0.0,
                     0.0,
                 )
                 .unwrap();
@@ -1050,10 +1035,12 @@ mod test {
             utxos,
             0,
             target_amount as i64,
-            FEE_AMOUNT,
+            50.0,
         );
 
         assert!(result.selected_amount() > target_amount);
-        assert_eq!(result.fee_amount, (50 + result.selected.len() * 68) as u64);
+        assert!(
+            (result.fee_amount - (50.0 + result.selected.len() as f32 * 68.0)).abs() < f32::EPSILON
+        );
     }
 }

src/wallet/export.rs

@@ -30,7 +30,7 @@
 //! }"#;
 //!
 //! let import = WalletExport::from_str(import)?;
-//! let wallet = Wallet::new(
+//! let wallet = Wallet::new_offline(
 //!     &import.descriptor(),
 //!     import.change_descriptor().as_ref(),
 //!     Network::Testnet,
@@ -45,7 +45,7 @@
 //! # use bdk::database::*;
 //! # use bdk::wallet::export::*;
 //! # use bdk::*;
-//! let wallet = Wallet::new(
+//! let wallet = Wallet::new_offline(
 //!     "wpkh([c258d2e4/84h/1h/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe/0/*)",
 //!     Some("wpkh([c258d2e4/84h/1h/0h]tpubDD3ynpHgJQW8VvWRzQ5WFDCrs4jqVFGHB3vLC3r49XHJSqP8bHKdK4AriuUKLccK68zfzowx7YhmDN8SiSkgCDENUFx9qVw65YyqM78vyVe/1/*)"),
 //!     Network::Testnet,
@@ -111,8 +111,8 @@ impl WalletExport {
     ///
     /// If the database is empty or `include_blockheight` is false, the `blockheight` field
     /// returned will be `0`.
-    pub fn export_wallet<D: BatchDatabase>(
-        wallet: &Wallet<D>,
+    pub fn export_wallet<B, D: BatchDatabase>(
+        wallet: &Wallet<B, D>,
         label: &str,
         include_blockheight: bool,
     ) -> Result<Self, &'static str> {
@@ -212,7 +212,7 @@ mod test {
     use crate::database::{memory::MemoryDatabase, BatchOperations};
     use crate::types::TransactionDetails;
     use crate::wallet::Wallet;
-    use crate::BlockTime;
+    use crate::ConfirmationTime;
 
     fn get_test_db() -> MemoryDatabase {
         let mut db = MemoryDatabase::new();
@@ -226,10 +226,11 @@ mod test {
             received: 100_000,
             sent: 0,
             fee: Some(500),
-            confirmation_time: Some(BlockTime {
+            confirmation_time: Some(ConfirmationTime {
                 timestamp: 12345678,
                 height: 5000,
             }),
+            verified: true,
         })
         .unwrap();
 
@@ -241,7 +242,7 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
-        let wallet = Wallet::new(
+        let wallet = Wallet::new_offline(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
@@ -265,7 +266,8 @@ mod test {
 
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
 
-        let wallet = Wallet::new(descriptor, None, Network::Bitcoin, get_test_db()).unwrap();
+        let wallet =
+            Wallet::new_offline(descriptor, None, Network::Bitcoin, get_test_db()).unwrap();
         WalletExport::export_wallet(&wallet, "Test Label", true).unwrap();
     }
 
@@ -278,7 +280,7 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/50'/0'/1/*)";
 
-        let wallet = Wallet::new(
+        let wallet = Wallet::new_offline(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,
@@ -301,7 +303,7 @@ mod test {
                                        [c98b1535/48'/0'/0'/2']tpubDCDi5W4sP6zSnzJeowy8rQDVhBdRARaPhK1axABi8V1661wEPeanpEXj4ZLAUEoikVtoWcyK26TKKJSecSfeKxwHCcRrge9k1ybuiL71z4a/1/*\
                                  ))";
 
-        let wallet = Wallet::new(
+        let wallet = Wallet::new_offline(
             descriptor,
             Some(change_descriptor),
             Network::Testnet,
@@ -321,7 +323,7 @@ mod test {
         let descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/0/*)";
         let change_descriptor = "wpkh(xprv9s21ZrQH143K4CTb63EaMxja1YiTnSEWKMbn23uoEnAzxjdUJRQkazCAtzxGm4LSoTSVTptoV9RbchnKPW9HxKtZumdyxyikZFDLhogJ5Uj/44'/0'/0'/1/*)";
 
-        let wallet = Wallet::new(
+        let wallet = Wallet::new_offline(
             descriptor,
             Some(change_descriptor),
             Network::Bitcoin,

src/wallet/signer.rs

@@ -72,7 +72,7 @@
 //! let custom_signer = CustomSigner::connect();
 //!
 //! let descriptor = "wpkh(tpubD6NzVbkrYhZ4Xferm7Pz4VnjdcDPFyjVu5K4iZXQ4pVN8Cks4pHVowTBXBKRhX64pkRyJZJN5xAKj4UDNnLPb5p2sSKXhewoYx5GbTdUFWq/*)";
-//! let mut wallet = Wallet::new(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
+//! let mut wallet = Wallet::new_offline(descriptor, None, Network::Testnet, MemoryDatabase::default())?;
 //! wallet.add_signer(
 //!     KeychainKind::External,
 //!     SignerOrdering(200),
@@ -143,7 +143,7 @@ pub enum SignerError {
     InvalidNonWitnessUtxo,
     /// The `witness_utxo` field of the transaction is required to sign this input
     MissingWitnessUtxo,
-    /// The `witness_script` field of the transaction is required to sign this input
+    /// The `witness_script` field of the transaction is requied to sign this input
     MissingWitnessScript,
     /// The fingerprint and derivation path are missing from the psbt input
     MissingHdKeypath,
@@ -289,7 +289,7 @@ impl Signer for PrivateKey {
         }
 
         // FIXME: use the presence of `witness_utxo` as an indication that we should make a bip143
-        // sig. Does this make sense? Should we add an extra argument to explicitly switch between
+        // sig. Does this make sense? Should we add an extra argument to explicitly swith between
         // these? The original idea was to declare sign() as sign<Ctx: ScriptContex>() and use Ctx,
         // but that violates the rules for trait-objects, so we can't do it.
         let (hash, sighash) = match psbt.inputs[input_index].witness_utxo {

src/wallet/tx_builder.rs

@@ -120,8 +120,8 @@ impl TxBuilderContext for BumpFee {}
 /// [`finish`]: Self::finish
 /// [`coin_selection`]: Self::coin_selection
 #[derive(Debug)]
-pub struct TxBuilder<'a, D, Cs, Ctx> {
-    pub(crate) wallet: &'a Wallet<D>,
+pub struct TxBuilder<'a, B, D, Cs, Ctx> {
+    pub(crate) wallet: &'a Wallet<B, D>,
     pub(crate) params: TxParams,
     pub(crate) coin_selection: Cs,
     pub(crate) phantom: PhantomData<Ctx>,
@@ -170,7 +170,7 @@ impl std::default::Default for FeePolicy {
     }
 }
 
-impl<'a, Cs: Clone, Ctx, D> Clone for TxBuilder<'a, D, Cs, Ctx> {
+impl<'a, Cs: Clone, Ctx, B, D> Clone for TxBuilder<'a, B, D, Cs, Ctx> {
     fn clone(&self) -> Self {
         TxBuilder {
             wallet: self.wallet,
@@ -182,8 +182,8 @@ impl<'a, Cs: Clone, Ctx, D> Clone for TxBuilder<'a, D, Cs, Ctx> {
 }
 
 // methods supported by both contexts, for any CoinSelectionAlgorithm
-impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
-    TxBuilder<'a, D, Cs, Ctx>
+impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
+    TxBuilder<'a, B, D, Cs, Ctx>
 {
     /// Set a custom fee rate
     pub fn fee_rate(&mut self, fee_rate: FeeRate) -> &mut Self {
@@ -310,7 +310,7 @@ impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
     /// 2. `psbt_input`: To know the value.
     /// 3. `satisfaction_weight`: To know how much weight/vbytes the input will add to the transaction for fee calculation.
     ///
-    /// There are several security concerns about adding foreign UTXOs that application
+    /// There are several security concerns about adding foregin UTXOs that application
     /// developers should consider. First, how do you know the value of the input is correct? If a
     /// `non_witness_utxo` is provided in the `psbt_input` then this method implicitly verifies the
     /// value by checking it against the transaction. If only a `witness_utxo` is provided then this
@@ -508,7 +508,7 @@ impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
     pub fn coin_selection<P: CoinSelectionAlgorithm<D>>(
         self,
         coin_selection: P,
-    ) -> TxBuilder<'a, D, P, Ctx> {
+    ) -> TxBuilder<'a, B, D, P, Ctx> {
         TxBuilder {
             wallet: self.wallet,
             params: self.params,
@@ -547,7 +547,7 @@ impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>, Ctx: TxBuilderContext>
     }
 }
 
-impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, D, Cs, CreateTx> {
+impl<'a, B, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, B, D, Cs, CreateTx> {
     /// Replace the recipients already added with a new list
     pub fn set_recipients(&mut self, recipients: Vec<(Script, u64)>) -> &mut Self {
         self.params.recipients = recipients;
@@ -560,13 +560,6 @@ impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, D, Cs, C
         self
     }
 
-    /// Add data as an output, using OP_RETURN
-    pub fn add_data(&mut self, data: &[u8]) -> &mut Self {
-        let script = Script::new_op_return(data);
-        self.add_recipient(script, 0u64);
-        self
-    }
-
     /// Sets the address to *drain* excess coins to.
     ///
     /// Usually, when there are excess coins they are sent to a change address generated by the
@@ -614,7 +607,7 @@ impl<'a, D: BatchDatabase, Cs: CoinSelectionAlgorithm<D>> TxBuilder<'a, D, Cs, C
 }
 
 // methods supported only by bump_fee
-impl<'a, D: BatchDatabase> TxBuilder<'a, D, DefaultCoinSelectionAlgorithm, BumpFee> {
+impl<'a, B, D: BatchDatabase> TxBuilder<'a, B, D, DefaultCoinSelectionAlgorithm, BumpFee> {
     /// Explicitly tells the wallet that it is allowed to reduce the fee of the output matching this
     /// `script_pubkey` in order to bump the transaction fee. Without specifying this the wallet
     /// will attempt to find a change output to shrink instead.

src/wallet/utils.rs

@@ -9,11 +9,13 @@
 // You may not use this file except in accordance with one or both of these
 // licenses.
 
-use bitcoin::blockdata::script::Script;
 use bitcoin::secp256k1::{All, Secp256k1};
 
 use miniscript::{MiniscriptKey, Satisfier, ToPublicKey};
 
+// De-facto standard "dust limit" (even though it should change based on the output type)
+pub const DUST_LIMIT_SATOSHI: u64 = 546;
+
 // MSB of the nSequence. If set there's no consensus-constraint, so it must be disabled when
 // spending using CSV in order to enforce CSV rules
 pub(crate) const SEQUENCE_LOCKTIME_DISABLE_FLAG: u32 = 1 << 31;
@@ -26,19 +28,18 @@ pub(crate) const SEQUENCE_LOCKTIME_MASK: u32 = 0x0000FFFF;
 // Threshold for nLockTime to be considered a block-height-based timelock rather than time-based
 pub(crate) const BLOCKS_TIMELOCK_THRESHOLD: u32 = 500000000;
 
-/// Trait to check if a value is below the dust limit.
-/// We are performing dust value calculation for a given script public key using rust-bitcoin to
-/// keep it compatible with network dust rate
+/// Trait to check if a value is below the dust limit
 // we implement this trait to make sure we don't mess up the comparison with off-by-one like a <
-// instead of a <= etc.
+// instead of a <= etc. The constant value for the dust limit is not public on purpose, to
+// encourage the usage of this trait.
 pub trait IsDust {
     /// Check whether or not a value is below dust limit
-    fn is_dust(&self, script: &Script) -> bool;
+    fn is_dust(&self) -> bool;
 }
 
 impl IsDust for u64 {
-    fn is_dust(&self, script: &Script) -> bool {
-        *self < script.dust_value().as_sat()
+    fn is_dust(&self) -> bool {
+        *self <= DUST_LIMIT_SATOSHI
     }
 }
 
@@ -137,32 +138,47 @@ impl<Pk: MiniscriptKey + ToPublicKey> Satisfier<Pk> for Older {
 
 pub(crate) type SecpCtx = Secp256k1<All>;
 
+pub struct ChunksIterator<I: Iterator> {
+    iter: I,
+    size: usize,
+}
+
+#[cfg(any(feature = "electrum", feature = "esplora"))]
+impl<I: Iterator> ChunksIterator<I> {
+    pub fn new(iter: I, size: usize) -> Self {
+        ChunksIterator { iter, size }
+    }
+}
+
+impl<I: Iterator> Iterator for ChunksIterator<I> {
+    type Item = Vec<<I as std::iter::Iterator>::Item>;
+
+    fn next(&mut self) -> Option<Self::Item> {
+        let mut v = Vec::new();
+        for _ in 0..self.size {
+            let e = self.iter.next();
+
+            match e {
+                None => break,
+                Some(val) => v.push(val),
+            }
+        }
+
+        if v.is_empty() {
+            return None;
+        }
+
+        Some(v)
+    }
+}
+
 #[cfg(test)]
 mod test {
     use super::{
-        check_nlocktime, check_nsequence_rbf, IsDust, BLOCKS_TIMELOCK_THRESHOLD,
+        check_nlocktime, check_nsequence_rbf, BLOCKS_TIMELOCK_THRESHOLD,
         SEQUENCE_LOCKTIME_TYPE_FLAG,
     };
-    use crate::bitcoin::Address;
     use crate::types::FeeRate;
-    use std::str::FromStr;
-
-    #[test]
-    fn test_is_dust() {
-        let script_p2pkh = Address::from_str("1GNgwA8JfG7Kc8akJ8opdNWJUihqUztfPe")
-            .unwrap()
-            .script_pubkey();
-        assert!(script_p2pkh.is_p2pkh());
-        assert!(545.is_dust(&script_p2pkh));
-        assert!(!546.is_dust(&script_p2pkh));
-
-        let script_p2wpkh = Address::from_str("bc1qxlh2mnc0yqwas76gqq665qkggee5m98t8yskd8")
-            .unwrap()
-            .script_pubkey();
-        assert!(script_p2wpkh.is_v0_p2wpkh());
-        assert!(293.is_dust(&script_p2wpkh));
-        assert!(!294.is_dust(&script_p2wpkh));
-    }
 
     #[test]
     fn test_fee_from_btc_per_kb() {

src/wallet/verify.rs

@@ -17,7 +17,7 @@ use std::fmt;
 use bitcoin::consensus::serialize;
 use bitcoin::{OutPoint, Transaction, Txid};
 
-use crate::blockchain::GetTx;
+use crate::blockchain::Blockchain;
 use crate::database::Database;
 use crate::error::Error;
 
@@ -29,7 +29,7 @@ use crate::error::Error;
 /// Depending on the [capabilities](crate::blockchain::Blockchain::get_capabilities) of the
 /// [`Blockchain`] backend, the method could fail when called with old "historical" transactions or
 /// with unconfirmed transactions that have been evicted from the backend's memory.
-pub fn verify_tx<D: Database, B: GetTx>(
+pub fn verify_tx<D: Database, B: Blockchain>(
     tx: &Transaction,
     database: &D,
     blockchain: &B,
@@ -104,18 +104,43 @@ impl_error!(bitcoinconsensus::Error, Consensus, VerifyError);
 
 #[cfg(test)]
 mod test {
-    use super::*;
-    use crate::database::{BatchOperations, MemoryDatabase};
+    use std::collections::HashSet;
+
     use bitcoin::consensus::encode::deserialize;
     use bitcoin::hashes::hex::FromHex;
     use bitcoin::{Transaction, Txid};
 
+    use crate::blockchain::{Blockchain, Capability, Progress};
+    use crate::database::{BatchDatabase, BatchOperations, MemoryDatabase};
+    use crate::FeeRate;
+
+    use super::*;
+
     struct DummyBlockchain;
 
-    impl GetTx for DummyBlockchain {
+    impl Blockchain for DummyBlockchain {
+        fn get_capabilities(&self) -> HashSet<Capability> {
+            Default::default()
+        }
+        fn setup<D: BatchDatabase, P: 'static + Progress>(
+            &self,
+            _database: &mut D,
+            _progress_update: P,
+        ) -> Result<(), Error> {
+            Ok(())
+        }
         fn get_tx(&self, _txid: &Txid) -> Result<Option<Transaction>, Error> {
             Ok(None)
         }
+        fn broadcast(&self, _tx: &Transaction) -> Result<(), Error> {
+            Ok(())
+        }
+        fn get_height(&self) -> Result<u32, Error> {
+            Ok(42)
+        }
+        fn estimate_fee(&self, _target: usize) -> Result<FeeRate, Error> {
+            Ok(FeeRate::default_min_relay_fee())
+        }
     }
 
     #[test]