Bump version to 0.23.0

d7bfe68e2d Fix broken nightly docs (Alekos Filini)
b11c86d074 Rename internal esplora modules, fix docs (Alekos Filini)
b5b92248c7 Rename esplora features to -async and -blocking (Alekos Filini)
cf2bc388f2 Re-export `esplora_client` (Elias Rohrer)
5baf46f84d Use the external esplora client library (Alekos Filini)

Pull request description:

  ### Description

  Use the external esplora client crate now that it's published

  ### Changelog notice

  - Start using the external esplora client crate
  - Deprecate the `use-esplora-reqwest` and `use-esplora-ureq` features in favor of `use-esplora-async` and `use-esplora-blocking`

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### New Features:

  * [ ] I've added tests for the new feature
  * [ ] I've added docs for the new feature

  #### Bugfixes:

  * [ ] This pull request breaks the existing API
  * [ ] I've added tests to reproduce the issue which are now passing
  * [ ] I'm linking the issue being fixed by this PR

ACKs for top commit:
  notmandatory:
    ACK d7bfe68e2d

Tree-SHA512: 23bd47536fe6f723602cbcc51d909eb9aed28376430f4453eea832e30a587be3d312cdca993d114391132bfb39c48637030f974ab1a742f7defe44f40a82ef8b

.github/ISSUE_TEMPLATE/minor_release.md

@@ -21,9 +21,9 @@ assignees: ''
 
 <--add notices from PRs merged since the prior release, see ["keep a changelog"]-->
 
-### Checklist 
+### Checklist
 
-Release numbering must follow [Semantic Versioning]. These steps assume the current `master` 
+Release numbering must follow [Semantic Versioning]. These steps assume the current `master`
 branch **development** version is *MAJOR.MINOR.0*.
 
 #### On the day of the feature freeze
@@ -37,34 +37,43 @@ Change the `master` branch to the next MINOR+1 version:
   - The commit message should be "Bump version to MAJOR.MINOR+1.0".
 - [ ] Create PR and merge the `bump_dev_MAJOR_MINOR+1` branch to `master`.
   - Title PR "Bump version to MAJOR.MINOR+1.0".
-  
+
 Create a new release branch and release candidate tag:
 
 - [ ] Double check that your local `master` is up-to-date with the upstream repo.
 - [ ] Create a new branch called `release/MAJOR.MINOR+1` from `master`.
+- [ ] Bump the `release/MAJOR.MINOR+1` branch to `MAJOR.MINOR+1.0-rc.1` version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR+1.0-rc.1`.
+  - The commit message should be "Bump version to MAJOR.MINOR+1.0-rc.1".
 - [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR+1` branch.
-  - The tag name should be `vMAJOR.MINOR+1.0-RC.1`
+  - The tag name should be `vMAJOR.MINOR+1.0-rc.1`
-  - Use message "Release MAJOR.MINOR+1.0 RC.1".  
+  - Use message "Release MAJOR.MINOR+1.0 rc.1".
   - Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
 - [ ] Push the `release/MAJOR.MINOR` branch and new tag to the `bitcoindevkit/bdk` repo.
-  - Use `git push --tags` option to push the new `vMAJOR.MINOR+1.0-RC.1` tag.
+  - Use `git push --tags` option to push the new `vMAJOR.MINOR+1.0-rc.1` tag.
-  
+
 If any issues need to be fixed before the *MAJOR.MINOR+1.0* version is released:
 
 - [ ] Merge fix PRs to the `master` branch.
 - [ ] Git cherry-pick fix commits to the `release/MAJOR.MINOR+1` branch.
 - [ ] Verify fixes in `release/MAJOR.MINOR+1` branch.
+- [ ] Bump the `release/MAJOR.MINOR+1` branch to `MAJOR.MINOR+1.0-rc.x+1` version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR+1.0-rc.x+1`.
+  - The commit message should be "Bump version to MAJOR.MINOR+1.0-rc.x+1".
 - [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR+1` branch.
-  - The tag name should be `vMAJOR.MINOR+1.0-RC.x+1`, where x is the current release candidate number.
+  - The tag name should be `vMAJOR.MINOR+1.0-rc.x+1`, where x is the current release candidate number.
-  - Use tag message "Release MAJOR.MINOR+1.0 RC.x+1".  
+  - Use tag message "Release MAJOR.MINOR+1.0 rc.x+1".
   - Make sure the tag is signed, for extra safety use the explicit `--sign` flag.
 - [ ] Push the new tag to the `bitcoindevkit/bdk` repo.
-  - Use `git push --tags` option to push the new `vMAJOR.MINOR+1.0-RC.x+1` tag.
+  - Use `git push --tags` option to push the new `vMAJOR.MINOR+1.0-rc.x+1` tag.
 
 #### On the day of the release
 
 Tag and publish new release:
 
+- [ ] Bump the `release/MAJOR.MINOR+1` branch to `MAJOR.MINOR+1.0` version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR+1.0`.
+  - The commit message should be "Bump version to MAJOR.MINOR+1.0".
 - [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR+1` branch.
   - The tag name should be `vMAJOR.MINOR+1.0`
   - The first line of the tag message should be "Release MAJOR.MINOR+1.0".

.github/ISSUE_TEMPLATE/patch_release.md

@@ -21,9 +21,9 @@ assignees: ''
 
 <--add notices from PRs merged since the prior release, see ["keep a changelog"]-->
 
-### Checklist  
+### Checklist
 
-Release numbering must follow [Semantic Versioning]. These steps assume the current `master` 
+Release numbering must follow [Semantic Versioning]. These steps assume the current `master`
 branch **development** version is *MAJOR.MINOR.PATCH*.
 
 ### On the day of the patch release
@@ -43,6 +43,9 @@ Cherry-pick, tag and publish new PATCH+1 release:
 - [ ] Merge fix PRs to the `master` branch.
 - [ ] Git cherry-pick fix commits to the `release/MAJOR.MINOR` branch to be patched.
 - [ ] Verify fixes in `release/MAJOR.MINOR` branch.
+- [ ] Bump the `release/MAJOR.MINOR.PATCH+1` branch to `MAJOR.MINOR.PATCH+1` version.
+  - Change the `Cargo.toml` version value to `MAJOR.MINOR.MINOR.PATCH+1`.
+  - The commit message should be "Bump version to MAJOR.MINOR.PATCH+1".
 - [ ] Add a tag to the `HEAD` commit in the `release/MAJOR.MINOR` branch.
   - The tag name should be `vMAJOR.MINOR.PATCH+1`
   - The first line of the tag message should be "Release MAJOR.MINOR.PATCH+1".

.github/workflows/code_coverage.yml

@@ -1,4 +1,4 @@
-on: [push]
+on: [push, pull_request]
 
 name: Code Coverage
 
@@ -38,7 +38,13 @@ jobs:
       - name: Install grcov
         run: if [[ ! -e ~/.cargo/bin/grcov ]]; then cargo install grcov; fi
       - name: Test
-        run: cargo test --features default,minimal,all-keys,compact_filters,key-value-db,compiler,sqlite,sqlite-bundled,test-electrum,verify,test-rpc
+        # WARNING: this is not testing the following features: test-esplora, test-hardware-signer, async-interface
+        # This is because some of our features are mutually exclusive, and generating various reports and
+        # merging them doesn't seem to be working very well.
+        # For more info, see:
+        # - https://github.com/bitcoindevkit/bdk/issues/696
+        # - https://github.com/bitcoindevkit/bdk/pull/748#issuecomment-1242721040
+        run: cargo test --features all-keys,compact_filters,compiler,key-value-db,sqlite,sqlite-bundled,test-electrum,test-rpc,verify
       - name: Run grcov
         run: mkdir coverage; grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore '/*' -o ./coverage/lcov.info
       - name: Generate HTML coverage report

.github/workflows/cont_integration.yml

@@ -17,16 +17,16 @@ jobs:
           - default
           - minimal
           - all-keys
-          - minimal,use-esplora-ureq
+          - minimal,use-esplora-blocking
           - key-value-db
           - electrum
           - compact_filters
-          - esplora,ureq,key-value-db,electrum
+          - use-esplora-blocking,key-value-db,electrum
           - compiler
           - rpc
           - verify
           - async-interface
-          - use-esplora-reqwest
+          - use-esplora-async
           - sqlite
           - sqlite-bundled
     steps:
@@ -100,10 +100,10 @@ jobs:
             features: test-rpc-legacy
           - name: esplora
             testprefix: esplora
-            features: test-esplora,use-esplora-reqwest,verify
+            features: test-esplora,use-esplora-async,verify
           - name: esplora
             testprefix: esplora
-            features: test-esplora,use-esplora-ureq,verify
+            features: test-esplora,use-esplora-blocking,verify
     steps:
       - name: Checkout
         uses: actions/checkout@v2
@@ -154,7 +154,7 @@ jobs:
       - name: Update toolchain
         run: rustup update
       - name: Check
-        run: cargo check --target wasm32-unknown-unknown --features use-esplora-reqwest --no-default-features
+        run: cargo check --target wasm32-unknown-unknown --features use-esplora-async --no-default-features
 
   fmt:
     name: Rust fmt

.github/workflows/nightly_docs.yml

@@ -24,7 +24,7 @@ jobs:
       - name: Update toolchain
         run: rustup update
       - name: Build docs
-        run: cargo rustdoc --verbose --features=compiler,electrum,esplora,ureq,compact_filters,key-value-db,all-keys,sqlite -- --cfg docsrs -Dwarnings
+        run: cargo rustdoc --verbose --features=compiler,electrum,esplora,use-esplora-blocking,compact_filters,rpc,key-value-db,sqlite,all-keys,verify,hardware-signer -- --cfg docsrs -Dwarnings
       - name: Upload artifact
         uses: actions/upload-artifact@v2
         with:

.gitignore

@@ -1,5 +1,6 @@
 /target
 Cargo.lock
+/.vscode
 
 *.swp
 .idea

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "bdk"
-version = "0.22.0-rc.2"
+version = "0.23.0"
 edition = "2018"
 authors = ["Alekos Filini <alekos.filini@gmail.com>", "Riccardo Casatta <riccardo@casatta.it>"]
 homepage = "https://bitcoindevkit.org"
@@ -23,16 +23,14 @@ rand = "^0.7"
 # Optional dependencies
 sled = { version = "0.34", optional = true }
 electrum-client = { version = "0.11", optional = true }
+esplora-client = { version = "0.1.1", default-features = false, optional = true }
 rusqlite = { version = "0.27.0", optional = true }
 ahash = { version = "0.7.6", optional = true }
-reqwest = { version = "0.11", optional = true, default-features = false, features = ["json"] }
-ureq = { version = "~2.2.0", features = ["json"], optional = true }
 futures = { version = "0.3", optional = true }
 async-trait = { version = "0.1", optional = true }
 rocksdb = { version = "0.14", default-features = false, features = ["snappy"], optional = true }
 cc = { version = ">=1.0.64", optional = true }
 socks = { version = "0.3", optional = true }
-lazy_static = { version = "1.4", optional = true }
 hwi = { version = "0.2.2", optional = true }
 
 bip39 = { version = "1.0.1", optional = true }
@@ -57,7 +55,7 @@ verify = ["bitcoinconsensus"]
 default = ["key-value-db", "electrum"]
 sqlite = ["rusqlite", "ahash"]
 sqlite-bundled = ["sqlite", "rusqlite/bundled"]
-compact_filters = ["rocksdb", "socks", "lazy_static", "cc"]
+compact_filters = ["rocksdb", "socks", "cc"]
 key-value-db = ["sled"]
 all-keys = ["keys-bip39"]
 keys-bip39 = ["bip39"]
@@ -70,23 +68,26 @@ hardware-signer = ["hwi"]
 #
 # - Users wanting asynchronous HTTP calls should enable `async-interface` to get
 #   access to the asynchronous method implementations. Then, if Esplora is wanted,
-#   enable `esplora` AND `reqwest` (`--features=use-esplora-reqwest`).
+#   enable the `use-esplora-async` feature.
 # - Users wanting blocking HTTP calls can use any of the other blockchain
 #   implementations (`compact_filters`, `electrum`, or `esplora`). Users wanting to
-#   use Esplora should enable `esplora` AND `ureq`  (`--features=use-esplora-ureq`).
+#   use Esplora should enable the `use-esplora-blocking` feature.
 #
 # WARNING: Please take care with the features below, various combinations will
 # fail to build. We cannot currently build `bdk` with `--all-features`.
 async-interface = ["async-trait"]
 electrum = ["electrum-client"]
 # MUST ALSO USE `--no-default-features`.
-use-esplora-reqwest = ["esplora", "reqwest", "reqwest/socks", "futures"]
+use-esplora-async = ["esplora", "esplora-client/async", "futures"]
-use-esplora-ureq = ["esplora", "ureq", "ureq/socks"]
+use-esplora-blocking = ["esplora", "esplora-client/blocking"]
+# Deprecated aliases
+use-esplora-reqwest = ["use-esplora-async"]
+use-esplora-ureq = ["use-esplora-blocking"]
 # Typical configurations will not need to use `esplora` feature directly.
 esplora = []
 
-# Use below feature with `use-esplora-reqwest` to enable reqwest default TLS support
+# Use below feature with `use-esplora-async` to enable reqwest default TLS support
-reqwest-default-tls = ["reqwest/default-tls"]
+reqwest-default-tls = ["esplora-client/async-https"]
 
 # Debug/Test features
 test-blockchains = ["bitcoincore-rpc", "electrum-client"]
@@ -100,7 +101,6 @@ test-hardware-signer = ["hardware-signer"]
 [dev-dependencies]
 lazy_static = "1.4"
 env_logger = "0.7"
-clap = "2.33"
 electrsd = "0.20"
 
 [[example]]
@@ -114,14 +114,23 @@ name = "miniscriptc"
 path = "examples/compiler.rs"
 required-features = ["compiler"]
 
+[[example]]
+name = "policy"
+path = "examples/policy.rs"
+
 [[example]]
 name = "rpcwallet"
 path = "examples/rpcwallet.rs"
 required-features = ["keys-bip39", "key-value-db", "rpc", "electrsd/bitcoind_22_0"]
 
+[[example]]
+name = "psbt_signer"
+path = "examples/psbt_signer.rs"
+required-features = ["electrum"]
+
 [workspace]
 members = ["macros"]
 [package.metadata.docs.rs]
-features = ["compiler", "electrum", "esplora", "use-esplora-ureq", "compact_filters", "rpc", "key-value-db", "sqlite", "all-keys", "verify", "hardware-signer"]
+features = ["compiler", "electrum", "esplora", "use-esplora-blocking", "compact_filters", "rpc", "key-value-db", "sqlite", "all-keys", "verify", "hardware-signer"]
 # defines the configuration attribute `docsrs`
 rustdoc-args = ["--cfg", "docsrs"]

examples/compiler.rs

@@ -11,7 +11,6 @@
 
 extern crate bdk;
 extern crate bitcoin;
-extern crate clap;
 extern crate log;
 extern crate miniscript;
 extern crate serde_json;
@@ -21,8 +20,6 @@ use std::str::FromStr;
 
 use log::info;
 
-use clap::{App, Arg};
-
 use bitcoin::Network;
 use miniscript::policy::Concrete;
 use miniscript::Descriptor;
@@ -31,75 +28,49 @@ use bdk::database::memory::MemoryDatabase;
 use bdk::wallet::AddressIndex::New;
 use bdk::{KeychainKind, Wallet};
 
+/// Miniscript policy is a high level abstraction of spending conditions. Defined in the
+/// rust-miscript library here  https://docs.rs/miniscript/7.0.0/miniscript/policy/index.html
+/// rust-miniscript provides a `compile()` function that can be used to compile any miniscript policy
+/// into a descriptor. This descriptor then in turn can be used in bdk a fully functioning wallet
+/// can be derived from the policy.
+///
+/// This example demonstrates the interaction between a bdk wallet and miniscript policy.
+
 fn main() -> Result<(), Box<dyn Error>> {
     env_logger::init_from_env(
         env_logger::Env::default().filter_or(env_logger::DEFAULT_FILTER_ENV, "info"),
     );
 
-    let matches = App::new("Miniscript Compiler")
+    // We start with a generic miniscript policy string
-        .arg(
+    let policy_str = "or(10@thresh(4,pk(029ffbe722b147f3035c87cb1c60b9a5947dd49c774cc31e94773478711a929ac0),pk(025f05815e3a1a8a83bfbb03ce016c9a2ee31066b98f567f6227df1d76ec4bd143),pk(025625f41e4a065efc06d5019cbbd56fe8c07595af1231e7cbc03fafb87ebb71ec),pk(02a27c8b850a00f67da3499b60562673dcf5fdfb82b7e17652a7ac54416812aefd),pk(03e618ec5f384d6e19ca9ebdb8e2119e5bef978285076828ce054e55c4daf473e2)),1@and(older(4209713),thresh(2,pk(03deae92101c790b12653231439f27b8897264125ecb2f46f48278603102573165),pk(033841045a531e1adf9910a6ec279589a90b3b8a904ee64ffd692bd08a8996c1aa),pk(02aebf2d10b040eb936a6f02f44ee82f8b34f5c1ccb20ff3949c2b28206b7c1068))))";
-            Arg::with_name("POLICY")
+    info!("Compiling policy: \n{}", policy_str);
-                .help("Sets the spending policy to compile")
-                .required(true)
-                .index(1),
-        )
-        .arg(
-            Arg::with_name("TYPE")
-                .help("Sets the script type used to embed the compiled policy")
-                .required(true)
-                .index(2)
-                .possible_values(&["sh", "wsh", "sh-wsh"]),
-        )
-        .arg(
-            Arg::with_name("parsed_policy")
-                .long("parsed_policy")
-                .short("p")
-                .help("Also return the parsed spending policy in JSON format"),
-        )
-        .arg(
-            Arg::with_name("network")
-                .short("n")
-                .long("network")
-                .help("Sets the network")
-                .takes_value(true)
-                .default_value("testnet")
-                .possible_values(&["testnet", "regtest", "bitcoin", "signet"]),
-        )
-        .get_matches();
-
-    let policy_str = matches.value_of("POLICY").unwrap();
-    info!("Compiling policy: {}", policy_str);
 
+    // Parse the string as a [`Concrete`] type miniscript policy.
     let policy = Concrete::<String>::from_str(policy_str)?;
 
-    let descriptor = match matches.value_of("TYPE").unwrap() {
+    // Create a `wsh` type descriptor from the policy.
-        "sh" => Descriptor::new_sh(policy.compile()?)?,
+    // `policy.compile()` returns the resulting miniscript from the policy.
-        "wsh" => Descriptor::new_wsh(policy.compile()?)?,
+    let descriptor = Descriptor::new_wsh(policy.compile()?)?;
-        "sh-wsh" => Descriptor::new_sh_wsh(policy.compile()?)?,
-        _ => panic!("Invalid type"),
-    };
 
-    info!("... Descriptor: {}", descriptor);
+    info!("Compiled into following Descriptor: \n{}", descriptor);
 
     let database = MemoryDatabase::new();
 
-    let network = matches
+    // Create a new wallet from this descriptor
-        .value_of("network")
+    let wallet = Wallet::new(&format!("{}", descriptor), None, Network::Regtest, database)?;
-        .map(Network::from_str)
-        .transpose()
-        .unwrap()
-        .unwrap_or(Network::Testnet);
-    let wallet = Wallet::new(&format!("{}", descriptor), None, network, database)?;
 
-    info!("... First address: {}", wallet.get_address(New)?);
+    info!(
+        "First derived address from the descriptor: \n{}",
+        wallet.get_address(New)?
+    );
 
-    if matches.is_present("parsed_policy") {
+    // BDK also has it's own `Policy` structure to represent the spending condition in a more
-        let spending_policy = wallet.policies(KeychainKind::External)?;
+    // human readable json format.
-        info!(
+    let spending_policy = wallet.policies(KeychainKind::External)?;
-            "... Spending policy:\n{}",
+    info!(
-            serde_json::to_string_pretty(&spending_policy)?
+        "The BDK spending policy: \n{}",
-        );
+        serde_json::to_string_pretty(&spending_policy)?
-    }
+    );
 
     Ok(())
 }

examples/policy.rs

@@ -0,0 +1,66 @@
+// Bitcoin Dev Kit
+// Written in 2020 by Alekos Filini <alekos.filini@gmail.com>
+//
+// Copyright (c) 2020-2021 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+extern crate bdk;
+extern crate env_logger;
+extern crate log;
+use std::error::Error;
+
+use bdk::bitcoin::Network;
+use bdk::descriptor::{policy::BuildSatisfaction, ExtractPolicy, IntoWalletDescriptor};
+use bdk::wallet::signer::SignersContainer;
+
+/// This example describes the use of the BDK's [`bdk::descriptor::policy`] module.
+///
+/// Policy is higher abstraction representation of the wallet descriptor spending condition.
+/// This is useful to express complex miniscript spending conditions into more human readable form.
+/// The resulting `Policy` structure  can be used to derive spending conditions the wallet is capable
+/// to spend from.
+///
+/// This example demos a Policy output for a 2of2 multisig between between 2 parties, where the wallet holds
+/// one of the Extend Private key.
+
+fn main() -> Result<(), Box<dyn Error>> {
+    env_logger::init_from_env(
+        env_logger::Env::default().filter_or(env_logger::DEFAULT_FILTER_ENV, "info"),
+    );
+
+    let secp = bitcoin::secp256k1::Secp256k1::new();
+
+    // The descriptor used in the example
+    // The form is "wsh(multi(2, <privkey>, <pubkey>))"
+    let desc = "wsh(multi(2,tprv8ZgxMBicQKsPdpkqS7Eair4YxjcuuvDPNYmKX3sCniCf16tHEVrjjiSXEkFRnUH77yXc6ZcwHHcLNfjdi5qUvw3VDfgYiH5mNsj5izuiu2N/1/*,tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/1/*))";
+
+    // Use the descriptor string to derive the full descriptor and a keymap.
+    // The wallet descriptor can be used to create a new bdk::wallet.
+    // While the `keymap` can be used to create a `SignerContainer`.
+    //
+    // The `SignerContainer` can sign for `PSBT`s.
+    // a bdk::wallet internally uses these to handle transaction signing.
+    // But they can be used as independent tools also.
+    let (wallet_desc, keymap) = desc.into_wallet_descriptor(&secp, Network::Testnet)?;
+
+    log::info!("Example Descriptor for policy analysis : {}", wallet_desc);
+
+    // Create the signer with the keymap and descriptor.
+    let signers_container = SignersContainer::build(keymap, &wallet_desc, &secp);
+
+    // Extract the Policy from the given descriptor and signer.
+    // Note that Policy is a wallet specific structure. It depends on the the descriptor, and
+    // what the concerned wallet with a given signer can sign for.
+    let policy = wallet_desc
+        .extract_policy(&signers_container, BuildSatisfaction::None, &secp)?
+        .expect("We expect a policy");
+
+    log::info!("Derived Policy for the descriptor {:#?}", policy);
+
+    Ok(())
+}

examples/psbt_signer.rs

@@ -0,0 +1,102 @@
+// Copyright (c) 2020-2022 Bitcoin Dev Kit Developers
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+use bdk::blockchain::{Blockchain, ElectrumBlockchain};
+use bdk::database::MemoryDatabase;
+use bdk::wallet::AddressIndex;
+use bdk::SyncOptions;
+use bdk::{FeeRate, SignOptions, Wallet};
+use bitcoin::{Address, Network};
+use electrum_client::Client;
+use std::error::Error;
+use std::str::FromStr;
+
+fn main() -> Result<(), Box<dyn Error>> {
+    // test keys created with `bdk-cli key generate` and `bdk-cli key derive` commands
+    let signing_external_descriptor = "wpkh([e9824965/84'/1'/0']tprv8fvem7qWxY3SGCQczQpRpqTKg455wf1zgixn6MZ4ze8gRfHjov5gXBQTadNfDgqs9ERbZZ3Bi1PNYrCCusFLucT39K525MWLpeURjHwUsfX/0/*)";
+    let signing_internal_descriptor = "wpkh([e9824965/84'/1'/0']tprv8fvem7qWxY3SGCQczQpRpqTKg455wf1zgixn6MZ4ze8gRfHjov5gXBQTadNfDgqs9ERbZZ3Bi1PNYrCCusFLucT39K525MWLpeURjHwUsfX/1/*)";
+
+    let watch_only_external_descriptor = "wpkh([e9824965/84'/1'/0']tpubDCcguXsm6uj79fSQt4V2EF7SF5b26zCuG2ZZNsbNQuw5G9YWSJuGhg2KknQBywRq4VGTu41zYTCh3QeVFyBdbsymgRX9Mrts94SW7obEdqs/0/*)";
+    let watch_only_internal_descriptor = "wpkh([e9824965/84'/1'/0']tpubDCcguXsm6uj79fSQt4V2EF7SF5b26zCuG2ZZNsbNQuw5G9YWSJuGhg2KknQBywRq4VGTu41zYTCh3QeVFyBdbsymgRX9Mrts94SW7obEdqs/1/*)";
+
+    // create client for Blockstream's testnet electrum server
+    let blockchain =
+        ElectrumBlockchain::from(Client::new("ssl://electrum.blockstream.info:60002")?);
+
+    // create watch only wallet
+    let watch_only_wallet: Wallet<MemoryDatabase> = Wallet::new(
+        watch_only_external_descriptor,
+        Some(watch_only_internal_descriptor),
+        Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    // create signing wallet
+    let signing_wallet: Wallet<MemoryDatabase> = Wallet::new(
+        signing_external_descriptor,
+        Some(signing_internal_descriptor),
+        Network::Testnet,
+        MemoryDatabase::default(),
+    )?;
+
+    println!("Syncing watch only wallet.");
+    watch_only_wallet.sync(&blockchain, SyncOptions::default())?;
+
+    // get deposit address
+    let deposit_address = watch_only_wallet.get_address(AddressIndex::New)?;
+
+    let balance = watch_only_wallet.get_balance()?;
+    println!("Watch only wallet balances in SATs: {}", balance);
+
+    if balance.get_total() < 10000 {
+        println!(
+            "Send at least 10000 SATs (0.0001 BTC) from the u01.net testnet faucet to address '{addr}'.\nFaucet URL: https://bitcoinfaucet.uo1.net/?to={addr}",
+            addr = deposit_address.address
+        );
+    } else if balance.get_spendable() < 10000 {
+        println!(
+            "Wait for at least 10000 SATs of your wallet transactions to be confirmed...\nBe patient, this could take 10 mins or longer depending on how testnet is behaving."
+        );
+        for tx_details in watch_only_wallet
+            .list_transactions(false)?
+            .iter()
+            .filter(|txd| txd.received > 0 && txd.confirmation_time.is_none())
+        {
+            println!(
+                "See unconfirmed tx for {} SATs: https://mempool.space/testnet/tx/{}",
+                tx_details.received, tx_details.txid
+            );
+        }
+    } else {
+        println!("Creating a PSBT sending 9800 SATs plus fee to the u01.net testnet faucet return address 'tb1ql7w62elx9ucw4pj5lgw4l028hmuw80sndtntxt'.");
+        let return_address = Address::from_str("tb1ql7w62elx9ucw4pj5lgw4l028hmuw80sndtntxt")?;
+        let mut builder = watch_only_wallet.build_tx();
+        builder
+            .add_recipient(return_address.script_pubkey(), 9_800)
+            .enable_rbf()
+            .fee_rate(FeeRate::from_sat_per_vb(1.0));
+
+        let (mut psbt, details) = builder.finish()?;
+        println!("Transaction details: {:#?}", details);
+        println!("Unsigned PSBT: {}", psbt);
+
+        // Sign and finalize the PSBT with the signing wallet
+        let finalized = signing_wallet.sign(&mut psbt, SignOptions::default())?;
+        assert!(finalized, "The PSBT was not finalized!");
+        println!("The PSBT has been signed and finalized.");
+
+        // Broadcast the transaction
+        let raw_transaction = psbt.extract_tx();
+        let txid = raw_transaction.txid();
+
+        blockchain.broadcast(&raw_transaction)?;
+        println!("Transaction broadcast! TXID: {txid}.\nExplorer URL: https://mempool.space/testnet/tx/{txid}", txid = txid);
+    }
+
+    Ok(())
+}

src/blockchain/compact_filters/store.rs

@@ -13,7 +13,6 @@ use std::convert::TryInto;
 use std::fmt;
 use std::io::{Read, Write};
 use std::marker::PhantomData;
-use std::ops::Deref;
 use std::sync::Arc;
 use std::sync::RwLock;
 
@@ -22,9 +21,9 @@ use rand::{thread_rng, Rng};
 
 use rocksdb::{Direction, IteratorMode, ReadOptions, WriteBatch, DB};
 
+use bitcoin::blockdata::constants::genesis_block;
 use bitcoin::consensus::{deserialize, encode::VarInt, serialize, Decodable, Encodable};
 use bitcoin::hash_types::{FilterHash, FilterHeader};
-use bitcoin::hashes::hex::FromHex;
 use bitcoin::hashes::Hash;
 use bitcoin::util::bip158::BlockFilter;
 use bitcoin::util::uint::Uint256;
@@ -33,17 +32,8 @@ use bitcoin::BlockHash;
 use bitcoin::BlockHeader;
 use bitcoin::Network;
 
-use lazy_static::lazy_static;
-
 use super::CompactFiltersError;
 
-lazy_static! {
-    static ref MAINNET_GENESIS: Block = deserialize(&Vec::<u8>::from_hex("0100000000000000000000000000000000000000000000000000000000000000000000003BA3EDFD7A7B12B27AC72C3E67768F617FC81BC3888A51323A9FB8AA4B1E5E4A29AB5F49FFFF001D1DAC2B7C0101000000010000000000000000000000000000000000000000000000000000000000000000FFFFFFFF4D04FFFF001D0104455468652054696D65732030332F4A616E2F32303039204368616E63656C6C6F72206F6E206272696E6B206F66207365636F6E64206261696C6F757420666F722062616E6B73FFFFFFFF0100F2052A01000000434104678AFDB0FE5548271967F1A67130B7105CD6A828E03909A67962E0EA1F61DEB649F6BC3F4CEF38C4F35504E51EC112DE5C384DF7BA0B8D578A4C702B6BF11D5FAC00000000").unwrap()).unwrap();
-    static ref TESTNET_GENESIS: Block = deserialize(&Vec::<u8>::from_hex("0100000000000000000000000000000000000000000000000000000000000000000000003BA3EDFD7A7B12B27AC72C3E67768F617FC81BC3888A51323A9FB8AA4B1E5E4ADAE5494DFFFF001D1AA4AE180101000000010000000000000000000000000000000000000000000000000000000000000000FFFFFFFF4D04FFFF001D0104455468652054696D65732030332F4A616E2F32303039204368616E63656C6C6F72206F6E206272696E6B206F66207365636F6E64206261696C6F757420666F722062616E6B73FFFFFFFF0100F2052A01000000434104678AFDB0FE5548271967F1A67130B7105CD6A828E03909A67962E0EA1F61DEB649F6BC3F4CEF38C4F35504E51EC112DE5C384DF7BA0B8D578A4C702B6BF11D5FAC00000000").unwrap()).unwrap();
-    static ref REGTEST_GENESIS: Block = deserialize(&Vec::<u8>::from_hex("0100000000000000000000000000000000000000000000000000000000000000000000003BA3EDFD7A7B12B27AC72C3E67768F617FC81BC3888A51323A9FB8AA4B1E5E4ADAE5494DFFFF7F20020000000101000000010000000000000000000000000000000000000000000000000000000000000000FFFFFFFF4D04FFFF001D0104455468652054696D65732030332F4A616E2F32303039204368616E63656C6C6F72206F6E206272696E6B206F66207365636F6E64206261696C6F757420666F722062616E6B73FFFFFFFF0100F2052A01000000434104678AFDB0FE5548271967F1A67130B7105CD6A828E03909A67962E0EA1F61DEB649F6BC3F4CEF38C4F35504E51EC112DE5C384DF7BA0B8D578A4C702B6BF11D5FAC00000000").unwrap()).unwrap();
-    static ref SIGNET_GENESIS: Block = deserialize(&Vec::<u8>::from_hex("0100000000000000000000000000000000000000000000000000000000000000000000003BA3EDFD7A7B12B27AC72C3E67768F617FC81BC3888A51323A9FB8AA4B1E5E4A008F4D5FAE77031E8AD222030101000000010000000000000000000000000000000000000000000000000000000000000000FFFFFFFF4D04FFFF001D0104455468652054696D65732030332F4A616E2F32303039204368616E63656C6C6F72206F6E206272696E6B206F66207365636F6E64206261696C6F757420666F722062616E6B73FFFFFFFF0100F2052A01000000434104678AFDB0FE5548271967F1A67130B7105CD6A828E03909A67962E0EA1F61DEB649F6BC3F4CEF38C4F35504E51EC112DE5C384DF7BA0B8D578A4C702B6BF11D5FAC00000000").unwrap()).unwrap();
-}
-
 pub trait StoreType: Default + fmt::Debug {}
 
 #[derive(Default, Debug)]
@@ -224,12 +214,7 @@ pub struct ChainStore<T: StoreType> {
 
 impl ChainStore<Full> {
     pub fn new(store: DB, network: Network) -> Result<Self, CompactFiltersError> {
-        let genesis = match network {
+        let genesis = genesis_block(network);
-            Network::Bitcoin => MAINNET_GENESIS.deref(),
-            Network::Testnet => TESTNET_GENESIS.deref(),
-            Network::Regtest => REGTEST_GENESIS.deref(),
-            Network::Signet => SIGNET_GENESIS.deref(),
-        };
 
         let cf_name = "default".to_string();
         let cf_handle = store.cf_handle(&cf_name).unwrap();
@@ -647,14 +632,9 @@ impl CfStore {
             filter_type,
         };
 
-        let genesis = match headers_store.network {
+        let genesis = genesis_block(headers_store.network);
-            Network::Bitcoin => MAINNET_GENESIS.deref(),
-            Network::Testnet => TESTNET_GENESIS.deref(),
-            Network::Regtest => REGTEST_GENESIS.deref(),
-            Network::Signet => SIGNET_GENESIS.deref(),
-        };
 
-        let filter = BlockFilter::new_script_filter(genesis, |utxo| {
+        let filter = BlockFilter::new_script_filter(&genesis, |utxo| {
             Err(bitcoin::util::bip158::Error::UtxoMissing(*utxo))
         })?;
         let first_key = StoreEntry::CFilterTable((filter_type, Some(0))).get_key();

src/blockchain/esplora/api.rs

@@ -1,117 +0,0 @@
-//! structs from the esplora API
-//!
-//! see: <https://github.com/Blockstream/esplora/blob/master/API.md>
-use crate::BlockTime;
-use bitcoin::{OutPoint, Script, Transaction, TxIn, TxOut, Txid, Witness};
-
-#[derive(serde::Deserialize, Clone, Debug)]
-pub struct PrevOut {
-    pub value: u64,
-    pub scriptpubkey: Script,
-}
-
-#[derive(serde::Deserialize, Clone, Debug)]
-pub struct Vin {
-    pub txid: Txid,
-    pub vout: u32,
-    // None if coinbase
-    pub prevout: Option<PrevOut>,
-    pub scriptsig: Script,
-    #[serde(deserialize_with = "deserialize_witness", default)]
-    pub witness: Vec<Vec<u8>>,
-    pub sequence: u32,
-    pub is_coinbase: bool,
-}
-
-#[derive(serde::Deserialize, Clone, Debug)]
-pub struct Vout {
-    pub value: u64,
-    pub scriptpubkey: Script,
-}
-
-#[derive(serde::Deserialize, Clone, Debug)]
-pub struct TxStatus {
-    pub confirmed: bool,
-    pub block_height: Option<u32>,
-    pub block_time: Option<u64>,
-}
-
-#[derive(serde::Deserialize, Clone, Debug)]
-pub struct Tx {
-    pub txid: Txid,
-    pub version: i32,
-    pub locktime: u32,
-    pub vin: Vec<Vin>,
-    pub vout: Vec<Vout>,
-    pub status: TxStatus,
-    pub fee: u64,
-}
-
-impl Tx {
-    pub fn to_tx(&self) -> Transaction {
-        Transaction {
-            version: self.version,
-            lock_time: self.locktime,
-            input: self
-                .vin
-                .iter()
-                .cloned()
-                .map(|vin| TxIn {
-                    previous_output: OutPoint {
-                        txid: vin.txid,
-                        vout: vin.vout,
-                    },
-                    script_sig: vin.scriptsig,
-                    sequence: vin.sequence,
-                    witness: Witness::from_vec(vin.witness),
-                })
-                .collect(),
-            output: self
-                .vout
-                .iter()
-                .cloned()
-                .map(|vout| TxOut {
-                    value: vout.value,
-                    script_pubkey: vout.scriptpubkey,
-                })
-                .collect(),
-        }
-    }
-
-    pub fn confirmation_time(&self) -> Option<BlockTime> {
-        match self.status {
-            TxStatus {
-                confirmed: true,
-                block_height: Some(height),
-                block_time: Some(timestamp),
-            } => Some(BlockTime { timestamp, height }),
-            _ => None,
-        }
-    }
-
-    pub fn previous_outputs(&self) -> Vec<Option<TxOut>> {
-        self.vin
-            .iter()
-            .cloned()
-            .map(|vin| {
-                vin.prevout.map(|po| TxOut {
-                    script_pubkey: po.scriptpubkey,
-                    value: po.value,
-                })
-            })
-            .collect()
-    }
-}
-
-fn deserialize_witness<'de, D>(d: D) -> Result<Vec<Vec<u8>>, D::Error>
-where
-    D: serde::de::Deserializer<'de>,
-{
-    use crate::serde::Deserialize;
-    use bitcoin::hashes::hex::FromHex;
-    let list = Vec::<String>::deserialize(d)?;
-    list.into_iter()
-        .map(|hex_str| Vec::<u8>::from_hex(&hex_str))
-        .collect::<Result<Vec<Vec<u8>>, _>>()
-        .map_err(serde::de::Error::custom)
-}

src/blockchain/esplora/async.rs

@@ -14,49 +14,36 @@
 use std::collections::{HashMap, HashSet};
 use std::ops::Deref;
 
-use bitcoin::consensus::{deserialize, serialize};
+use bitcoin::{Transaction, Txid};
-use bitcoin::hashes::hex::{FromHex, ToHex};
-use bitcoin::hashes::{sha256, Hash};
-use bitcoin::{BlockHeader, Script, Transaction, Txid};
 
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
-use ::reqwest::{Client, StatusCode};
+use esplora_client::{convert_fee_rate, AsyncClient, Builder, Tx};
 use futures::stream::{FuturesOrdered, TryStreamExt};
 
-use super::api::Tx;
-use crate::blockchain::esplora::EsploraError;
 use crate::blockchain::*;
 use crate::database::BatchDatabase;
 use crate::error::Error;
 use crate::FeeRate;
 
-/// Structure encapsulates Esplora client
-#[derive(Debug)]
-pub struct UrlClient {
-    url: String,
-    // We use the async client instead of the blocking one because it automatically uses `fetch`
-    // when the target platform is wasm32.
-    client: Client,
-    concurrency: u8,
-}
-
 /// Structure that implements the logic to sync with Esplora
 ///
 /// ## Example
 /// See the [`blockchain::esplora`](crate::blockchain::esplora) module for a usage example.
 #[derive(Debug)]
 pub struct EsploraBlockchain {
-    url_client: UrlClient,
+    url_client: AsyncClient,
     stop_gap: usize,
+    concurrency: u8,
 }
 
-impl std::convert::From<UrlClient> for EsploraBlockchain {
+impl std::convert::From<AsyncClient> for EsploraBlockchain {
-    fn from(url_client: UrlClient) -> Self {
+    fn from(url_client: AsyncClient) -> Self {
         EsploraBlockchain {
             url_client,
             stop_gap: 20,
+            concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
         }
     }
 }
@@ -64,19 +51,25 @@ impl std::convert::From<UrlClient> for EsploraBlockchain {
 impl EsploraBlockchain {
     /// Create a new instance of the client from a base URL and `stop_gap`.
     pub fn new(base_url: &str, stop_gap: usize) -> Self {
+        let url_client = Builder::new(base_url)
+            .build_async()
+            .expect("Should never fail with no proxy and timeout");
+
+        Self::from_client(url_client, stop_gap)
+    }
+
+    /// Build a new instance given a client
+    pub fn from_client(url_client: AsyncClient, stop_gap: usize) -> Self {
         EsploraBlockchain {
-            url_client: UrlClient {
+            url_client,
-                url: base_url.to_string(),
-                client: Client::new(),
-                concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
-            },
             stop_gap,
+            concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
         }
     }
 
     /// Set the concurrency to use when doing batch queries against the Esplora instance.
     pub fn with_concurrency(mut self, concurrency: u8) -> Self {
-        self.url_client.concurrency = concurrency;
+        self.concurrency = concurrency;
         self
     }
 }
@@ -94,17 +87,19 @@ impl Blockchain for EsploraBlockchain {
     }
 
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        Ok(await_or_block!(self.url_client._broadcast(tx))?)
+        Ok(await_or_block!(self.url_client.broadcast(tx))?)
     }
 
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
-        let estimates = await_or_block!(self.url_client._get_fee_estimates())?;
+        let estimates = await_or_block!(self.url_client.get_fee_estimates())?;
-        super::into_fee_rate(target, estimates)
+        Ok(FeeRate::from_sat_per_vb(convert_fee_rate(
+            target, estimates,
+        )?))
     }
 }
 
 impl Deref for EsploraBlockchain {
-    type Target = UrlClient;
+    type Target = AsyncClient;
 
     fn deref(&self) -> &Self::Target {
         &self.url_client
@@ -116,21 +111,21 @@ impl StatelessBlockchain for EsploraBlockchain {}
 #[maybe_async]
 impl GetHeight for EsploraBlockchain {
     fn get_height(&self) -> Result<u32, Error> {
-        Ok(await_or_block!(self.url_client._get_height())?)
+        Ok(await_or_block!(self.url_client.get_height())?)
     }
 }
 
 #[maybe_async]
 impl GetTx for EsploraBlockchain {
     fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(await_or_block!(self.url_client._get_tx(txid))?)
+        Ok(await_or_block!(self.url_client.get_tx(txid))?)
     }
 }
 
 #[maybe_async]
 impl GetBlockHash for EsploraBlockchain {
     fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
-        let block_header = await_or_block!(self.url_client._get_header(height as u32))?;
+        let block_header = await_or_block!(self.url_client.get_header(height as u32))?;
         Ok(block_header.block_hash())
     }
 }
@@ -151,10 +146,10 @@ impl WalletSync for EsploraBlockchain {
                 Request::Script(script_req) => {
                     let futures: FuturesOrdered<_> = script_req
                         .request()
-                        .take(self.url_client.concurrency as usize)
+                        .take(self.concurrency as usize)
                         .map(|script| async move {
                             let mut related_txs: Vec<Tx> =
-                                self.url_client._scripthash_txs(script, None).await?;
+                                self.url_client.scripthash_txs(script, None).await?;
 
                             let n_confirmed =
                                 related_txs.iter().filter(|tx| tx.status.confirmed).count();
@@ -164,7 +159,7 @@ impl WalletSync for EsploraBlockchain {
                                 loop {
                                     let new_related_txs: Vec<Tx> = self
                                         .url_client
-                                        ._scripthash_txs(
+                                        .scripthash_txs(
                                             script,
                                             Some(related_txs.last().unwrap().txid),
                                         )
@@ -204,6 +199,7 @@ impl WalletSync for EsploraBlockchain {
                                 .get(txid)
                                 .expect("must be in index")
                                 .confirmation_time()
+                                .map(Into::into)
                         })
                         .collect();
                     conftime_req.satisfy(conftimes)?
@@ -227,132 +223,26 @@ impl WalletSync for EsploraBlockchain {
     }
 }
 
-impl UrlClient {
-    async fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
-        let resp = self
-            .client
-            .get(&format!("{}/tx/{}/raw", self.url, txid))
-            .send()
-            .await?;
-
-        if let StatusCode::NOT_FOUND = resp.status() {
-            return Ok(None);
-        }
-
-        Ok(Some(deserialize(&resp.error_for_status()?.bytes().await?)?))
-    }
-
-    async fn _get_tx_no_opt(&self, txid: &Txid) -> Result<Transaction, EsploraError> {
-        match self._get_tx(txid).await {
-            Ok(Some(tx)) => Ok(tx),
-            Ok(None) => Err(EsploraError::TransactionNotFound(*txid)),
-            Err(e) => Err(e),
-        }
-    }
-
-    async fn _get_header(&self, block_height: u32) -> Result<BlockHeader, EsploraError> {
-        let resp = self
-            .client
-            .get(&format!("{}/block-height/{}", self.url, block_height))
-            .send()
-            .await?;
-
-        if let StatusCode::NOT_FOUND = resp.status() {
-            return Err(EsploraError::HeaderHeightNotFound(block_height));
-        }
-        let bytes = resp.bytes().await?;
-        let hash = std::str::from_utf8(&bytes)
-            .map_err(|_| EsploraError::HeaderHeightNotFound(block_height))?;
-
-        let resp = self
-            .client
-            .get(&format!("{}/block/{}/header", self.url, hash))
-            .send()
-            .await?;
-
-        let header = deserialize(&Vec::from_hex(&resp.text().await?)?)?;
-
-        Ok(header)
-    }
-
-    async fn _broadcast(&self, transaction: &Transaction) -> Result<(), EsploraError> {
-        self.client
-            .post(&format!("{}/tx", self.url))
-            .body(serialize(transaction).to_hex())
-            .send()
-            .await?
-            .error_for_status()?;
-
-        Ok(())
-    }
-
-    async fn _get_height(&self) -> Result<u32, EsploraError> {
-        let req = self
-            .client
-            .get(&format!("{}/blocks/tip/height", self.url))
-            .send()
-            .await?;
-
-        Ok(req.error_for_status()?.text().await?.parse()?)
-    }
-
-    async fn _scripthash_txs(
-        &self,
-        script: &Script,
-        last_seen: Option<Txid>,
-    ) -> Result<Vec<Tx>, EsploraError> {
-        let script_hash = sha256::Hash::hash(script.as_bytes()).into_inner().to_hex();
-        let url = match last_seen {
-            Some(last_seen) => format!(
-                "{}/scripthash/{}/txs/chain/{}",
-                self.url, script_hash, last_seen
-            ),
-            None => format!("{}/scripthash/{}/txs", self.url, script_hash),
-        };
-        Ok(self
-            .client
-            .get(url)
-            .send()
-            .await?
-            .error_for_status()?
-            .json::<Vec<Tx>>()
-            .await?)
-    }
-
-    async fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
-        Ok(self
-            .client
-            .get(&format!("{}/fee-estimates", self.url,))
-            .send()
-            .await?
-            .error_for_status()?
-            .json::<HashMap<String, f64>>()
-            .await?)
-    }
-}
-
 impl ConfigurableBlockchain for EsploraBlockchain {
     type Config = super::EsploraBlockchainConfig;
 
     fn from_config(config: &Self::Config) -> Result<Self, Error> {
-        let map_e = |e: reqwest::Error| Error::Esplora(Box::new(e.into()));
+        let mut builder = Builder::new(config.base_url.as_str());
 
-        let mut blockchain = EsploraBlockchain::new(config.base_url.as_str(), config.stop_gap);
-        if let Some(concurrency) = config.concurrency {
-            blockchain.url_client.concurrency = concurrency;
-        }
-        let mut builder = Client::builder();
-        #[cfg(not(target_arch = "wasm32"))]
-        if let Some(proxy) = &config.proxy {
-            builder = builder.proxy(reqwest::Proxy::all(proxy).map_err(map_e)?);
-        }
-
-        #[cfg(not(target_arch = "wasm32"))]
         if let Some(timeout) = config.timeout {
-            builder = builder.timeout(core::time::Duration::from_secs(timeout));
+            builder = builder.timeout(timeout);
         }
 
-        blockchain.url_client.client = builder.build().map_err(map_e)?;
+        if let Some(proxy) = &config.proxy {
+            builder = builder.proxy(proxy);
+        }
+
+        let mut blockchain =
+            EsploraBlockchain::from_client(builder.build_async()?, config.stop_gap);
+
+        if let Some(concurrency) = config.concurrency {
+            blockchain = blockchain.with_concurrency(concurrency);
+        }
 
         Ok(blockchain)
     }

src/blockchain/esplora/blocking.rs

@@ -12,42 +12,26 @@
 //! Esplora by way of `ureq` HTTP client.
 
 use std::collections::{HashMap, HashSet};
-use std::io;
-use std::io::Read;
-use std::ops::Deref;
-use std::time::Duration;
 
 #[allow(unused_imports)]
 use log::{debug, error, info, trace};
 
-use ureq::{Agent, Proxy, Response};
+use bitcoin::{Transaction, Txid};
 
-use bitcoin::consensus::{deserialize, serialize};
+use esplora_client::{convert_fee_rate, BlockingClient, Builder, Tx};
-use bitcoin::hashes::hex::{FromHex, ToHex};
-use bitcoin::hashes::{sha256, Hash};
-use bitcoin::{BlockHeader, Script, Transaction, Txid};
 
-use super::api::Tx;
-use crate::blockchain::esplora::EsploraError;
 use crate::blockchain::*;
 use crate::database::BatchDatabase;
 use crate::error::Error;
 use crate::FeeRate;
 
-/// Structure encapsulates ureq Esplora client
-#[derive(Debug, Clone)]
-pub struct UrlClient {
-    url: String,
-    agent: Agent,
-}
-
 /// Structure that implements the logic to sync with Esplora
 ///
 /// ## Example
 /// See the [`blockchain::esplora`](crate::blockchain::esplora) module for a usage example.
 #[derive(Debug)]
 pub struct EsploraBlockchain {
-    url_client: UrlClient,
+    url_client: BlockingClient,
     stop_gap: usize,
     concurrency: u8,
 }
@@ -55,22 +39,22 @@ pub struct EsploraBlockchain {
 impl EsploraBlockchain {
     /// Create a new instance of the client from a base URL and the `stop_gap`.
     pub fn new(base_url: &str, stop_gap: usize) -> Self {
+        let url_client = Builder::new(base_url)
+            .build_blocking()
+            .expect("Should never fail with no proxy and timeout");
+
+        Self::from_client(url_client, stop_gap)
+    }
+
+    /// Build a new instance given a client
+    pub fn from_client(url_client: BlockingClient, stop_gap: usize) -> Self {
         EsploraBlockchain {
-            url_client: UrlClient {
+            url_client,
-                url: base_url.to_string(),
-                agent: Agent::new(),
-            },
             concurrency: super::DEFAULT_CONCURRENT_REQUESTS,
             stop_gap,
         }
     }
 
-    /// Set the inner `ureq` agent.
-    pub fn with_agent(mut self, agent: Agent) -> Self {
-        self.url_client.agent = agent;
-        self
-    }
-
     /// Set the number of parallel requests the client can make.
     pub fn with_concurrency(mut self, concurrency: u8) -> Self {
         self.concurrency = concurrency;
@@ -90,18 +74,20 @@ impl Blockchain for EsploraBlockchain {
     }
 
     fn broadcast(&self, tx: &Transaction) -> Result<(), Error> {
-        self.url_client._broadcast(tx)?;
+        self.url_client.broadcast(tx)?;
         Ok(())
     }
 
     fn estimate_fee(&self, target: usize) -> Result<FeeRate, Error> {
-        let estimates = self.url_client._get_fee_estimates()?;
+        let estimates = self.url_client.get_fee_estimates()?;
-        super::into_fee_rate(target, estimates)
+        Ok(FeeRate::from_sat_per_vb(convert_fee_rate(
+            target, estimates,
+        )?))
     }
 }
 
 impl Deref for EsploraBlockchain {
-    type Target = UrlClient;
+    type Target = BlockingClient;
 
     fn deref(&self) -> &Self::Target {
         &self.url_client
@@ -112,19 +98,19 @@ impl StatelessBlockchain for EsploraBlockchain {}
 
 impl GetHeight for EsploraBlockchain {
     fn get_height(&self) -> Result<u32, Error> {
-        Ok(self.url_client._get_height()?)
+        Ok(self.url_client.get_height()?)
     }
 }
 
 impl GetTx for EsploraBlockchain {
     fn get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, Error> {
-        Ok(self.url_client._get_tx(txid)?)
+        Ok(self.url_client.get_tx(txid)?)
     }
 }
 
 impl GetBlockHash for EsploraBlockchain {
     fn get_block_hash(&self, height: u64) -> Result<BlockHash, Error> {
-        let block_header = self.url_client._get_header(height as u32)?;
+        let block_header = self.url_client.get_header(height as u32)?;
         Ok(block_header.block_hash())
     }
 }
@@ -151,7 +137,7 @@ impl WalletSync for EsploraBlockchain {
                         let client = self.url_client.clone();
                         // make each request in its own thread.
                         handles.push(std::thread::spawn(move || {
-                            let mut related_txs: Vec<Tx> = client._scripthash_txs(&script, None)?;
+                            let mut related_txs: Vec<Tx> = client.scripthash_txs(&script, None)?;
 
                             let n_confirmed =
                                 related_txs.iter().filter(|tx| tx.status.confirmed).count();
@@ -159,7 +145,7 @@ impl WalletSync for EsploraBlockchain {
                             // keep requesting to see if there's more.
                             if n_confirmed >= 25 {
                                 loop {
-                                    let new_related_txs: Vec<Tx> = client._scripthash_txs(
+                                    let new_related_txs: Vec<Tx> = client.scripthash_txs(
                                         &script,
                                         Some(related_txs.last().unwrap().txid),
                                     )?;
@@ -202,6 +188,7 @@ impl WalletSync for EsploraBlockchain {
                                 .get(txid)
                                 .expect("must be in index")
                                 .confirmation_time()
+                                .map(Into::into)
                         })
                         .collect();
                     conftime_req.satisfy(conftimes)?
@@ -226,159 +213,22 @@ impl WalletSync for EsploraBlockchain {
     }
 }
 
-impl UrlClient {
-    fn _get_tx(&self, txid: &Txid) -> Result<Option<Transaction>, EsploraError> {
-        let resp = self
-            .agent
-            .get(&format!("{}/tx/{}/raw", self.url, txid))
-            .call();
-
-        match resp {
-            Ok(resp) => Ok(Some(deserialize(&into_bytes(resp)?)?)),
-            Err(ureq::Error::Status(code, _)) => {
-                if is_status_not_found(code) {
-                    return Ok(None);
-                }
-                Err(EsploraError::HttpResponse(code))
-            }
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }
-    }
-
-    fn _get_tx_no_opt(&self, txid: &Txid) -> Result<Transaction, EsploraError> {
-        match self._get_tx(txid) {
-            Ok(Some(tx)) => Ok(tx),
-            Ok(None) => Err(EsploraError::TransactionNotFound(*txid)),
-            Err(e) => Err(e),
-        }
-    }
-
-    fn _get_header(&self, block_height: u32) -> Result<BlockHeader, EsploraError> {
-        let resp = self
-            .agent
-            .get(&format!("{}/block-height/{}", self.url, block_height))
-            .call();
-
-        let bytes = match resp {
-            Ok(resp) => Ok(into_bytes(resp)?),
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }?;
-
-        let hash = std::str::from_utf8(&bytes)
-            .map_err(|_| EsploraError::HeaderHeightNotFound(block_height))?;
-
-        let resp = self
-            .agent
-            .get(&format!("{}/block/{}/header", self.url, hash))
-            .call();
-
-        match resp {
-            Ok(resp) => Ok(deserialize(&Vec::from_hex(&resp.into_string()?)?)?),
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }
-    }
-
-    fn _broadcast(&self, transaction: &Transaction) -> Result<(), EsploraError> {
-        let resp = self
-            .agent
-            .post(&format!("{}/tx", self.url))
-            .send_string(&serialize(transaction).to_hex());
-
-        match resp {
-            Ok(_) => Ok(()), // We do not return the txid?
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }
-    }
-
-    fn _get_height(&self) -> Result<u32, EsploraError> {
-        let resp = self
-            .agent
-            .get(&format!("{}/blocks/tip/height", self.url))
-            .call();
-
-        match resp {
-            Ok(resp) => Ok(resp.into_string()?.parse()?),
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }
-    }
-
-    fn _get_fee_estimates(&self) -> Result<HashMap<String, f64>, EsploraError> {
-        let resp = self
-            .agent
-            .get(&format!("{}/fee-estimates", self.url,))
-            .call();
-
-        let map = match resp {
-            Ok(resp) => {
-                let map: HashMap<String, f64> = resp.into_json()?;
-                Ok(map)
-            }
-            Err(ureq::Error::Status(code, _)) => Err(EsploraError::HttpResponse(code)),
-            Err(e) => Err(EsploraError::Ureq(e)),
-        }?;
-
-        Ok(map)
-    }
-
-    fn _scripthash_txs(
-        &self,
-        script: &Script,
-        last_seen: Option<Txid>,
-    ) -> Result<Vec<Tx>, EsploraError> {
-        let script_hash = sha256::Hash::hash(script.as_bytes()).into_inner().to_hex();
-        let url = match last_seen {
-            Some(last_seen) => format!(
-                "{}/scripthash/{}/txs/chain/{}",
-                self.url, script_hash, last_seen
-            ),
-            None => format!("{}/scripthash/{}/txs", self.url, script_hash),
-        };
-        Ok(self.agent.get(&url).call()?.into_json()?)
-    }
-}
-
-fn is_status_not_found(status: u16) -> bool {
-    status == 404
-}
-
-fn into_bytes(resp: Response) -> Result<Vec<u8>, io::Error> {
-    const BYTES_LIMIT: usize = 10 * 1_024 * 1_024;
-
-    let mut buf: Vec<u8> = vec![];
-    resp.into_reader()
-        .take((BYTES_LIMIT + 1) as u64)
-        .read_to_end(&mut buf)?;
-    if buf.len() > BYTES_LIMIT {
-        return Err(io::Error::new(
-            io::ErrorKind::Other,
-            "response too big for into_bytes",
-        ));
-    }
-
-    Ok(buf)
-}
-
 impl ConfigurableBlockchain for EsploraBlockchain {
     type Config = super::EsploraBlockchainConfig;
 
     fn from_config(config: &Self::Config) -> Result<Self, Error> {
-        let mut agent_builder = ureq::AgentBuilder::new();
+        let mut builder = Builder::new(config.base_url.as_str());
 
         if let Some(timeout) = config.timeout {
-            agent_builder = agent_builder.timeout(Duration::from_secs(timeout));
+            builder = builder.timeout(timeout);
         }
 
         if let Some(proxy) = &config.proxy {
-            agent_builder = agent_builder
+            builder = builder.proxy(proxy);
-                .proxy(Proxy::new(proxy).map_err(|e| Error::Esplora(Box::new(e.into())))?);
         }
 
-        let mut blockchain = EsploraBlockchain::new(config.base_url.as_str(), config.stop_gap)
+        let mut blockchain =
-            .with_agent(agent_builder.build());
+            EsploraBlockchain::from_client(builder.build_blocking()?, config.stop_gap);
 
         if let Some(concurrency) = config.concurrency {
             blockchain = blockchain.with_concurrency(concurrency);
@@ -387,12 +237,3 @@ impl ConfigurableBlockchain for EsploraBlockchain {
         Ok(blockchain)
     }
 }
-
-impl From<ureq::Error> for EsploraError {
-    fn from(e: ureq::Error) -> Self {
-        match e {
-            ureq::Error::Status(code, _) => EsploraError::HttpResponse(code),
-            e => EsploraError::Ureq(e),
-        }
-    }
-}

src/blockchain/esplora/mod.rs

@@ -15,86 +15,22 @@
 //! depending on your needs (blocking or async respectively).
 //!
 //! Please note, to configure the Esplora HTTP client correctly use one of:
-//! Blocking:  --features='esplora,ureq'
+//! Blocking:  --features='use-esplora-blocking'
-//! Async:     --features='async-interface,esplora,reqwest' --no-default-features
+//! Async:     --features='async-interface,use-esplora-async' --no-default-features
-use std::collections::HashMap;
-use std::fmt;
-use std::io;
 
-use bitcoin::consensus;
+pub use esplora_client::Error as EsploraError;
-use bitcoin::{BlockHash, Txid};
 
-use crate::error::Error;
+#[cfg(feature = "use-esplora-async")]
-use crate::FeeRate;
+mod r#async;
 
-#[cfg(feature = "reqwest")]
+#[cfg(feature = "use-esplora-async")]
-mod reqwest;
+pub use self::r#async::*;
 
-#[cfg(feature = "reqwest")]
+#[cfg(feature = "use-esplora-blocking")]
-pub use self::reqwest::*;
+mod blocking;
 
-#[cfg(feature = "ureq")]
+#[cfg(feature = "use-esplora-blocking")]
-mod ureq;
+pub use self::blocking::*;
-
-#[cfg(feature = "ureq")]
-pub use self::ureq::*;
-
-mod api;
-
-fn into_fee_rate(target: usize, estimates: HashMap<String, f64>) -> Result<FeeRate, Error> {
-    let fee_val = {
-        let mut pairs = estimates
-            .into_iter()
-            .filter_map(|(k, v)| Some((k.parse::<usize>().ok()?, v)))
-            .collect::<Vec<_>>();
-        pairs.sort_unstable_by_key(|(k, _)| std::cmp::Reverse(*k));
-        pairs
-            .into_iter()
-            .find(|(k, _)| k <= &target)
-            .map(|(_, v)| v)
-            .unwrap_or(1.0)
-    };
-    Ok(FeeRate::from_sat_per_vb(fee_val as f32))
-}
-
-/// Errors that can happen during a sync with [`EsploraBlockchain`]
-#[derive(Debug)]
-pub enum EsploraError {
-    /// Error during ureq HTTP request
-    #[cfg(feature = "ureq")]
-    Ureq(::ureq::Error),
-    /// Transport error during the ureq HTTP call
-    #[cfg(feature = "ureq")]
-    UreqTransport(::ureq::Transport),
-    /// Error during reqwest HTTP request
-    #[cfg(feature = "reqwest")]
-    Reqwest(::reqwest::Error),
-    /// HTTP response error
-    HttpResponse(u16),
-    /// IO error during ureq response read
-    Io(io::Error),
-    /// No header found in ureq response
-    NoHeader,
-    /// Invalid number returned
-    Parsing(std::num::ParseIntError),
-    /// Invalid Bitcoin data returned
-    BitcoinEncoding(bitcoin::consensus::encode::Error),
-    /// Invalid Hex data returned
-    Hex(bitcoin::hashes::hex::Error),
-
-    /// Transaction not found
-    TransactionNotFound(Txid),
-    /// Header height not found
-    HeaderHeightNotFound(u32),
-    /// Header hash not found
-    HeaderHashNotFound(BlockHash),
-}
-
-impl fmt::Display for EsploraError {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        write!(f, "{:?}", self)
-    }
-}
 
 /// Configuration for an [`EsploraBlockchain`]
 #[derive(Debug, serde::Deserialize, serde::Serialize, Clone, PartialEq)]
@@ -138,16 +74,11 @@ impl EsploraBlockchainConfig {
     }
 }
 
-impl std::error::Error for EsploraError {}
+impl From<esplora_client::BlockTime> for crate::BlockTime {
-
+    fn from(esplora_client::BlockTime { timestamp, height }: esplora_client::BlockTime) -> Self {
-#[cfg(feature = "ureq")]
+        Self { timestamp, height }
-impl_error!(::ureq::Transport, UreqTransport, EsploraError);
+    }
-#[cfg(feature = "reqwest")]
+}
-impl_error!(::reqwest::Error, Reqwest, EsploraError);
-impl_error!(io::Error, Io, EsploraError);
-impl_error!(std::num::ParseIntError, Parsing, EsploraError);
-impl_error!(consensus::encode::Error, BitcoinEncoding, EsploraError);
-impl_error!(bitcoin::hashes::hex::Error, Hex, EsploraError);
 
 #[cfg(test)]
 #[cfg(feature = "test-esplora")]
@@ -161,58 +92,11 @@ const DEFAULT_CONCURRENT_REQUESTS: u8 = 4;
 
 #[cfg(test)]
 mod test {
-    use super::*;
-
-    #[test]
-    fn feerate_parsing() {
-        let esplora_fees = serde_json::from_str::<HashMap<String, f64>>(
-            r#"{
-  "25": 1.015,
-  "5": 2.3280000000000003,
-  "12": 2.0109999999999997,
-  "15": 1.018,
-  "17": 1.018,
-  "11": 2.0109999999999997,
-  "3": 3.01,
-  "2": 4.9830000000000005,
-  "6": 2.2359999999999998,
-  "21": 1.018,
-  "13": 1.081,
-  "7": 2.2359999999999998,
-  "8": 2.2359999999999998,
-  "16": 1.018,
-  "20": 1.018,
-  "22": 1.017,
-  "23": 1.017,
-  "504": 1,
-  "9": 2.2359999999999998,
-  "14": 1.018,
-  "10": 2.0109999999999997,
-  "24": 1.017,
-  "1008": 1,
-  "1": 4.9830000000000005,
-  "4": 2.3280000000000003,
-  "19": 1.018,
-  "144": 1,
-  "18": 1.018
-}
-"#,
-        )
-        .unwrap();
-        assert_eq!(
-            into_fee_rate(6, esplora_fees.clone()).unwrap(),
-            FeeRate::from_sat_per_vb(2.236)
-        );
-        assert_eq!(
-            into_fee_rate(26, esplora_fees).unwrap(),
-            FeeRate::from_sat_per_vb(1.015),
-            "should inherit from value for 25"
-        );
-    }
-
     #[test]
     #[cfg(feature = "test-esplora")]
     fn test_esplora_with_variable_configs() {
+        use super::*;
+
         use crate::testutils::{
             blockchain_tests::TestClient,
             configurable_blockchain_tests::ConfigurableBlockchainTester,

src/database/keyvalue.rs

@@ -492,4 +492,44 @@ mod test {
     fn test_sync_time() {
         crate::database::test::test_sync_time(get_tree());
     }
+
+    #[test]
+    fn test_iter_raw_txs() {
+        crate::database::test::test_iter_raw_txs(get_tree());
+    }
+
+    #[test]
+    fn test_del_path_from_script_pubkey() {
+        crate::database::test::test_del_path_from_script_pubkey(get_tree());
+    }
+
+    #[test]
+    fn test_iter_script_pubkeys() {
+        crate::database::test::test_iter_script_pubkeys(get_tree());
+    }
+
+    #[test]
+    fn test_del_utxo() {
+        crate::database::test::test_del_utxo(get_tree());
+    }
+
+    #[test]
+    fn test_del_raw_tx() {
+        crate::database::test::test_del_raw_tx(get_tree());
+    }
+
+    #[test]
+    fn test_del_tx() {
+        crate::database::test::test_del_tx(get_tree());
+    }
+
+    #[test]
+    fn test_del_last_index() {
+        crate::database::test::test_del_last_index(get_tree());
+    }
+
+    #[test]
+    fn test_check_descriptor_checksum() {
+        crate::database::test::test_check_descriptor_checksum(get_tree());
+    }
 }

src/database/memory.rs

@@ -647,4 +647,44 @@ mod test {
     fn test_sync_time() {
         crate::database::test::test_sync_time(get_tree());
     }
+
+    #[test]
+    fn test_iter_raw_txs() {
+        crate::database::test::test_iter_raw_txs(get_tree());
+    }
+
+    #[test]
+    fn test_del_path_from_script_pubkey() {
+        crate::database::test::test_del_path_from_script_pubkey(get_tree());
+    }
+
+    #[test]
+    fn test_iter_script_pubkeys() {
+        crate::database::test::test_iter_script_pubkeys(get_tree());
+    }
+
+    #[test]
+    fn test_del_utxo() {
+        crate::database::test::test_del_utxo(get_tree());
+    }
+
+    #[test]
+    fn test_del_raw_tx() {
+        crate::database::test::test_del_raw_tx(get_tree());
+    }
+
+    #[test]
+    fn test_del_tx() {
+        crate::database::test::test_del_tx(get_tree());
+    }
+
+    #[test]
+    fn test_del_last_index() {
+        crate::database::test::test_del_last_index(get_tree());
+    }
+
+    #[test]
+    fn test_check_descriptor_checksum() {
+        crate::database::test::test_check_descriptor_checksum(get_tree());
+    }
 }

src/database/mod.rs

@@ -217,32 +217,33 @@ pub mod test {
     use std::str::FromStr;
 
     use bitcoin::consensus::encode::deserialize;
+    use bitcoin::consensus::serialize;
     use bitcoin::hashes::hex::*;
     use bitcoin::*;
 
     use super::*;
 
-    pub fn test_script_pubkey<D: Database>(mut tree: D) {
+    pub fn test_script_pubkey<D: Database>(mut db: D) {
         let script = Script::from(
             Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
         );
         let path = 42;
         let keychain = KeychainKind::External;
 
-        tree.set_script_pubkey(&script, keychain, path).unwrap();
+        db.set_script_pubkey(&script, keychain, path).unwrap();
 
         assert_eq!(
-            tree.get_script_pubkey_from_path(keychain, path).unwrap(),
+            db.get_script_pubkey_from_path(keychain, path).unwrap(),
             Some(script.clone())
         );
         assert_eq!(
-            tree.get_path_from_script_pubkey(&script).unwrap(),
+            db.get_path_from_script_pubkey(&script).unwrap(),
             Some((keychain, path))
         );
     }
 
-    pub fn test_batch_script_pubkey<D: BatchDatabase>(mut tree: D) {
+    pub fn test_batch_script_pubkey<D: BatchDatabase>(mut db: D) {
-        let mut batch = tree.begin_batch();
+        let mut batch = db.begin_batch();
 
         let script = Script::from(
             Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
@@ -253,50 +254,50 @@ pub mod test {
         batch.set_script_pubkey(&script, keychain, path).unwrap();
 
         assert_eq!(
-            tree.get_script_pubkey_from_path(keychain, path).unwrap(),
+            db.get_script_pubkey_from_path(keychain, path).unwrap(),
             None
         );
-        assert_eq!(tree.get_path_from_script_pubkey(&script).unwrap(), None);
+        assert_eq!(db.get_path_from_script_pubkey(&script).unwrap(), None);
 
-        tree.commit_batch(batch).unwrap();
+        db.commit_batch(batch).unwrap();
 
         assert_eq!(
-            tree.get_script_pubkey_from_path(keychain, path).unwrap(),
+            db.get_script_pubkey_from_path(keychain, path).unwrap(),
             Some(script.clone())
         );
         assert_eq!(
-            tree.get_path_from_script_pubkey(&script).unwrap(),
+            db.get_path_from_script_pubkey(&script).unwrap(),
             Some((keychain, path))
         );
     }
 
-    pub fn test_iter_script_pubkey<D: Database>(mut tree: D) {
+    pub fn test_iter_script_pubkey<D: Database>(mut db: D) {
         let script = Script::from(
             Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
         );
         let path = 42;
         let keychain = KeychainKind::External;
 
-        tree.set_script_pubkey(&script, keychain, path).unwrap();
+        db.set_script_pubkey(&script, keychain, path).unwrap();
 
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 1);
+        assert_eq!(db.iter_script_pubkeys(None).unwrap().len(), 1);
     }
 
-    pub fn test_del_script_pubkey<D: Database>(mut tree: D) {
+    pub fn test_del_script_pubkey<D: Database>(mut db: D) {
         let script = Script::from(
             Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
         );
         let path = 42;
         let keychain = KeychainKind::External;
 
-        tree.set_script_pubkey(&script, keychain, path).unwrap();
+        db.set_script_pubkey(&script, keychain, path).unwrap();
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 1);
+        assert_eq!(db.iter_script_pubkeys(None).unwrap().len(), 1);
 
-        tree.del_script_pubkey_from_path(keychain, path).unwrap();
+        db.del_script_pubkey_from_path(keychain, path).unwrap();
-        assert_eq!(tree.iter_script_pubkeys(None).unwrap().len(), 0);
+        assert_eq!(db.iter_script_pubkeys(None).unwrap().len(), 0);
     }
 
-    pub fn test_utxo<D: Database>(mut tree: D) {
+    pub fn test_utxo<D: Database>(mut db: D) {
         let outpoint = OutPoint::from_str(
             "5df6e0e2761359d30a8275058e299fcc0381534545f55cf43e41983f5d4c9456:0",
         )
@@ -315,24 +316,40 @@ pub mod test {
             is_spent: true,
         };
 
-        tree.set_utxo(&utxo).unwrap();
+        db.set_utxo(&utxo).unwrap();
-        tree.set_utxo(&utxo).unwrap();
+        db.set_utxo(&utxo).unwrap();
-        assert_eq!(tree.iter_utxos().unwrap().len(), 1);
+        assert_eq!(db.iter_utxos().unwrap().len(), 1);
-        assert_eq!(tree.get_utxo(&outpoint).unwrap(), Some(utxo));
+        assert_eq!(db.get_utxo(&outpoint).unwrap(), Some(utxo));
     }
 
-    pub fn test_raw_tx<D: Database>(mut tree: D) {
+    pub fn test_raw_tx<D: Database>(mut db: D) {
-        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
+        let hex_tx = Vec::<u8>::from_hex("02000000000101f58c18a90d7a76b30c7e47d4e817adfdd79a6a589a615ef36e360f913adce2cd0000000000feffffff0210270000000000001600145c9a1816d38db5cbdd4b067b689dc19eb7d930e2cf70aa2b080000001600140f48b63160043047f4f60f7f8f551f80458f693f024730440220413f42b7bc979945489a38f5221e5527d4b8e3aa63eae2099e01945896ad6c10022024ceec492d685c31d8adb64e935a06933877c5ae0e21f32efe029850914c5bad012102361caae96f0e9f3a453d354bb37a5c3244422fb22819bf0166c0647a38de39f21fca2300").unwrap();
-        let tx: Transaction = deserialize(&hex_tx).unwrap();
+        let mut tx: Transaction = deserialize(&hex_tx).unwrap();
 
-        tree.set_raw_tx(&tx).unwrap();
+        db.set_raw_tx(&tx).unwrap();
 
         let txid = tx.txid();
 
-        assert_eq!(tree.get_raw_tx(&txid).unwrap(), Some(tx));
+        assert_eq!(db.get_raw_tx(&txid).unwrap(), Some(tx.clone()));
+
+        // mutate transaction's witnesses
+        for tx_in in tx.input.iter_mut() {
+            tx_in.witness = Witness::new();
+        }
+
+        let updated_hex_tx = serialize(&tx);
+
+        // verify that mutation was successful
+        assert_ne!(hex_tx, updated_hex_tx);
+
+        db.set_raw_tx(&tx).unwrap();
+
+        let txid = tx.txid();
+
+        assert_eq!(db.get_raw_tx(&txid).unwrap(), Some(tx));
     }
 
-    pub fn test_tx<D: Database>(mut tree: D) {
+    pub fn test_tx<D: Database>(mut db: D) {
         let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
         let tx: Transaction = deserialize(&hex_tx).unwrap();
         let txid = tx.txid();
@@ -348,28 +365,28 @@ pub mod test {
             }),
         };
 
-        tree.set_tx(&tx_details).unwrap();
+        db.set_tx(&tx_details).unwrap();
 
         // get with raw tx too
         assert_eq!(
-            tree.get_tx(&tx_details.txid, true).unwrap(),
+            db.get_tx(&tx_details.txid, true).unwrap(),
             Some(tx_details.clone())
         );
         // get only raw_tx
         assert_eq!(
-            tree.get_raw_tx(&tx_details.txid).unwrap(),
+            db.get_raw_tx(&tx_details.txid).unwrap(),
             tx_details.transaction
         );
 
         // now get without raw_tx
         tx_details.transaction = None;
         assert_eq!(
-            tree.get_tx(&tx_details.txid, false).unwrap(),
+            db.get_tx(&tx_details.txid, false).unwrap(),
             Some(tx_details)
         );
     }
 
-    pub fn test_list_transaction<D: Database>(mut tree: D) {
+    pub fn test_list_transaction<D: Database>(mut db: D) {
         let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
         let tx: Transaction = deserialize(&hex_tx).unwrap();
         let txid = tx.txid();
@@ -385,46 +402,43 @@ pub mod test {
             }),
         };
 
-        tree.set_tx(&tx_details).unwrap();
+        db.set_tx(&tx_details).unwrap();
 
         // get raw tx
-        assert_eq!(tree.iter_txs(true).unwrap(), vec![tx_details.clone()]);
+        assert_eq!(db.iter_txs(true).unwrap(), vec![tx_details.clone()]);
 
         // now get without raw tx
         tx_details.transaction = None;
 
         // get not raw tx
-        assert_eq!(tree.iter_txs(false).unwrap(), vec![tx_details.clone()]);
+        assert_eq!(db.iter_txs(false).unwrap(), vec![tx_details.clone()]);
     }
 
-    pub fn test_last_index<D: Database>(mut tree: D) {
+    pub fn test_last_index<D: Database>(mut db: D) {
-        tree.set_last_index(KeychainKind::External, 1337).unwrap();
+        db.set_last_index(KeychainKind::External, 1337).unwrap();
 
         assert_eq!(
-            tree.get_last_index(KeychainKind::External).unwrap(),
+            db.get_last_index(KeychainKind::External).unwrap(),
             Some(1337)
         );
-        assert_eq!(tree.get_last_index(KeychainKind::Internal).unwrap(), None);
+        assert_eq!(db.get_last_index(KeychainKind::Internal).unwrap(), None);
 
-        let res = tree.increment_last_index(KeychainKind::External).unwrap();
+        let res = db.increment_last_index(KeychainKind::External).unwrap();
         assert_eq!(res, 1338);
-        let res = tree.increment_last_index(KeychainKind::Internal).unwrap();
+        let res = db.increment_last_index(KeychainKind::Internal).unwrap();
         assert_eq!(res, 0);
 
         assert_eq!(
-            tree.get_last_index(KeychainKind::External).unwrap(),
+            db.get_last_index(KeychainKind::External).unwrap(),
             Some(1338)
         );
-        assert_eq!(
+        assert_eq!(db.get_last_index(KeychainKind::Internal).unwrap(), Some(0));
-            tree.get_last_index(KeychainKind::Internal).unwrap(),
-            Some(0)
-        );
     }
 
-    pub fn test_sync_time<D: Database>(mut tree: D) {
+    pub fn test_sync_time<D: Database>(mut db: D) {
-        assert!(tree.get_sync_time().unwrap().is_none());
+        assert!(db.get_sync_time().unwrap().is_none());
 
-        tree.set_sync_time(SyncTime {
+        db.set_sync_time(SyncTime {
             block_time: BlockTime {
                 height: 100,
                 timestamp: 1000,
@@ -432,13 +446,211 @@ pub mod test {
         })
         .unwrap();
 
-        let extracted = tree.get_sync_time().unwrap();
+        let extracted = db.get_sync_time().unwrap();
         assert!(extracted.is_some());
         assert_eq!(extracted.as_ref().unwrap().block_time.height, 100);
         assert_eq!(extracted.as_ref().unwrap().block_time.timestamp, 1000);
 
-        tree.del_sync_time().unwrap();
+        db.del_sync_time().unwrap();
-        assert!(tree.get_sync_time().unwrap().is_none());
+        assert!(db.get_sync_time().unwrap().is_none());
+    }
+
+    pub fn test_iter_raw_txs<D: Database>(mut db: D) {
+        let txs = db.iter_raw_txs().unwrap();
+        assert!(txs.is_empty());
+
+        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
+        let first_tx: Transaction = deserialize(&hex_tx).unwrap();
+
+        let hex_tx = Vec::<u8>::from_hex("02000000000101f58c18a90d7a76b30c7e47d4e817adfdd79a6a589a615ef36e360f913adce2cd0000000000feffffff0210270000000000001600145c9a1816d38db5cbdd4b067b689dc19eb7d930e2cf70aa2b080000001600140f48b63160043047f4f60f7f8f551f80458f693f024730440220413f42b7bc979945489a38f5221e5527d4b8e3aa63eae2099e01945896ad6c10022024ceec492d685c31d8adb64e935a06933877c5ae0e21f32efe029850914c5bad012102361caae96f0e9f3a453d354bb37a5c3244422fb22819bf0166c0647a38de39f21fca2300").unwrap();
+        let second_tx: Transaction = deserialize(&hex_tx).unwrap();
+
+        db.set_raw_tx(&first_tx).unwrap();
+        db.set_raw_tx(&second_tx).unwrap();
+
+        let txs = db.iter_raw_txs().unwrap();
+
+        assert!(txs.contains(&first_tx));
+        assert!(txs.contains(&second_tx));
+        assert_eq!(txs.len(), 2);
+    }
+
+    pub fn test_del_path_from_script_pubkey<D: Database>(mut db: D) {
+        let keychain = KeychainKind::External;
+
+        let script = Script::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let path = 42;
+
+        let res = db.del_path_from_script_pubkey(&script).unwrap();
+
+        assert!(res.is_none());
+
+        let _res = db.set_script_pubkey(&script, keychain, path);
+        let (chain, child) = db.del_path_from_script_pubkey(&script).unwrap().unwrap();
+
+        assert_eq!(chain, keychain);
+        assert_eq!(child, path);
+
+        let res = db.get_path_from_script_pubkey(&script).unwrap();
+        assert!(res.is_none());
+    }
+
+    pub fn test_iter_script_pubkeys<D: Database>(mut db: D) {
+        let keychain = KeychainKind::External;
+        let scripts = db.iter_script_pubkeys(Some(keychain)).unwrap();
+        assert!(scripts.is_empty());
+
+        let first_script = Script::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let path = 42;
+
+        db.set_script_pubkey(&first_script, keychain, path).unwrap();
+
+        let second_script = Script::from(
+            Vec::<u8>::from_hex("00145c9a1816d38db5cbdd4b067b689dc19eb7d930e2").unwrap(),
+        );
+        let path = 57;
+
+        db.set_script_pubkey(&second_script, keychain, path)
+            .unwrap();
+        let scripts = db.iter_script_pubkeys(Some(keychain)).unwrap();
+
+        assert!(scripts.contains(&first_script));
+        assert!(scripts.contains(&second_script));
+        assert_eq!(scripts.len(), 2);
+    }
+
+    pub fn test_del_utxo<D: Database>(mut db: D) {
+        let outpoint = OutPoint::from_str(
+            "5df6e0e2761359d30a8275058e299fcc0381534545f55cf43e41983f5d4c9456:0",
+        )
+        .unwrap();
+        let script = Script::from(
+            Vec::<u8>::from_hex("76a91402306a7c23f3e8010de41e9e591348bb83f11daa88ac").unwrap(),
+        );
+        let txout = TxOut {
+            value: 133742,
+            script_pubkey: script,
+        };
+        let utxo = LocalUtxo {
+            txout,
+            outpoint,
+            keychain: KeychainKind::External,
+            is_spent: true,
+        };
+
+        let res = db.del_utxo(&outpoint).unwrap();
+        assert!(res.is_none());
+
+        db.set_utxo(&utxo).unwrap();
+
+        let res = db.del_utxo(&outpoint).unwrap();
+
+        assert_eq!(res.unwrap(), utxo);
+
+        let res = db.get_utxo(&outpoint).unwrap();
+        assert!(res.is_none());
+    }
+
+    pub fn test_del_raw_tx<D: Database>(mut db: D) {
+        let hex_tx = Vec::<u8>::from_hex("02000000000101f58c18a90d7a76b30c7e47d4e817adfdd79a6a589a615ef36e360f913adce2cd0000000000feffffff0210270000000000001600145c9a1816d38db5cbdd4b067b689dc19eb7d930e2cf70aa2b080000001600140f48b63160043047f4f60f7f8f551f80458f693f024730440220413f42b7bc979945489a38f5221e5527d4b8e3aa63eae2099e01945896ad6c10022024ceec492d685c31d8adb64e935a06933877c5ae0e21f32efe029850914c5bad012102361caae96f0e9f3a453d354bb37a5c3244422fb22819bf0166c0647a38de39f21fca2300").unwrap();
+        let tx: Transaction = deserialize(&hex_tx).unwrap();
+
+        let res = db.del_raw_tx(&tx.txid()).unwrap();
+
+        assert!(res.is_none());
+
+        db.set_raw_tx(&tx).unwrap();
+
+        let res = db.del_raw_tx(&tx.txid()).unwrap();
+
+        assert_eq!(res.unwrap(), tx);
+
+        let res = db.get_raw_tx(&tx.txid()).unwrap();
+        assert!(res.is_none());
+    }
+
+    pub fn test_del_tx<D: Database>(mut db: D) {
+        let hex_tx = Vec::<u8>::from_hex("0100000001a15d57094aa7a21a28cb20b59aab8fc7d1149a3bdbcddba9c622e4f5f6a99ece010000006c493046022100f93bb0e7d8db7bd46e40132d1f8242026e045f03a0efe71bbb8e3f475e970d790221009337cd7f1f929f00cc6ff01f03729b069a7c21b59b1736ddfee5db5946c5da8c0121033b9b137ee87d5a812d6f506efdd37f0affa7ffc310711c06c7f3e097c9447c52ffffffff0100e1f505000000001976a9140389035a9225b3839e2bbf32d826a1e222031fd888ac00000000").unwrap();
+        let tx: Transaction = deserialize(&hex_tx).unwrap();
+        let txid = tx.txid();
+        let mut tx_details = TransactionDetails {
+            transaction: Some(tx.clone()),
+            txid,
+            received: 1337,
+            sent: 420420,
+            fee: Some(140),
+            confirmation_time: Some(BlockTime {
+                timestamp: 123456,
+                height: 1000,
+            }),
+        };
+
+        let res = db.del_tx(&tx.txid(), true).unwrap();
+
+        assert!(res.is_none());
+
+        db.set_tx(&tx_details).unwrap();
+
+        let res = db.del_tx(&tx.txid(), false).unwrap();
+        tx_details.transaction = None;
+        assert_eq!(res.unwrap(), tx_details);
+
+        let res = db.get_tx(&tx.txid(), true).unwrap();
+        assert!(res.is_none());
+
+        let res = db.get_raw_tx(&tx.txid()).unwrap();
+        assert_eq!(res.unwrap(), tx);
+
+        db.set_tx(&tx_details).unwrap();
+        let res = db.del_tx(&tx.txid(), true).unwrap();
+        tx_details.transaction = Some(tx.clone());
+        assert_eq!(res.unwrap(), tx_details);
+
+        let res = db.get_tx(&tx.txid(), true).unwrap();
+        assert!(res.is_none());
+
+        let res = db.get_raw_tx(&tx.txid()).unwrap();
+        assert!(res.is_none());
+    }
+
+    pub fn test_del_last_index<D: Database>(mut db: D) {
+        let keychain = KeychainKind::External;
+
+        let _res = db.increment_last_index(keychain);
+
+        let res = db.get_last_index(keychain).unwrap().unwrap();
+
+        assert_eq!(res, 0);
+
+        let _res = db.increment_last_index(keychain);
+
+        let res = db.del_last_index(keychain).unwrap().unwrap();
+
+        assert_eq!(res, 1);
+
+        let res = db.get_last_index(keychain).unwrap();
+        assert!(res.is_none());
+    }
+
+    pub fn test_check_descriptor_checksum<D: Database>(mut db: D) {
+        // insert checksum associated to keychain
+        let checksum = "1cead456".as_bytes();
+        let keychain = KeychainKind::External;
+        let _res = db.check_descriptor_checksum(keychain, checksum);
+
+        // check if `check_descriptor_checksum` throws
+        // `Error::ChecksumMismatch` error if the
+        // function is passed a checksum that does
+        // not match the one initially inserted
+        let checksum = "1cead454".as_bytes();
+        let keychain = KeychainKind::External;
+        let res = db.check_descriptor_checksum(keychain, checksum);
+
+        assert!(res.is_err());
     }
 
     // TODO: more tests...

src/database/sqlite.rs

@@ -52,6 +52,11 @@ static MIGRATIONS: &[&str] = &[
     "DELETE FROM transactions;",
     "DELETE FROM utxos;",
     "DROP INDEX idx_txid_vout;",
+    "CREATE UNIQUE INDEX idx_utxos_txid_vout ON utxos(txid, vout);",
+    "ALTER TABLE utxos RENAME TO utxos_old;",
+    "CREATE TABLE utxos (value INTEGER, keychain TEXT, vout INTEGER, txid BLOB, script BLOB, is_spent BOOLEAN DEFAULT 0);",
+    "INSERT INTO utxos SELECT value, keychain, vout, txid, script, is_spent FROM utxos_old;",
+    "DROP TABLE utxos_old;",
     "CREATE UNIQUE INDEX idx_utxos_txid_vout ON utxos(txid, vout);"
 ];
 
@@ -744,11 +749,13 @@ impl BatchOperations for SqliteDatabase {
         include_raw: bool,
     ) -> Result<Option<TransactionDetails>, Error> {
         match self.select_transaction_details_by_txid(txid)? {
-            Some(transaction_details) => {
+            Some(mut transaction_details) => {
                 self.delete_transaction_details_by_txid(txid)?;
 
                 if include_raw {
                     self.delete_transaction_by_txid(txid)?;
+                } else {
+                    transaction_details.transaction = None;
                 }
                 Ok(Some(transaction_details))
             }
@@ -914,8 +921,8 @@ impl BatchDatabase for SqliteDatabase {
 }
 
 pub fn get_connection<T: AsRef<Path>>(path: &T) -> Result<Connection, Error> {
-    let connection = Connection::open(path)?;
+    let mut connection = Connection::open(path)?;
-    migrate(&connection)?;
+    migrate(&mut connection)?;
     Ok(connection)
 }
 
@@ -950,28 +957,41 @@ pub fn set_schema_version(conn: &Connection, version: i32) -> rusqlite::Result<u
     )
 }
 
-pub fn migrate(conn: &Connection) -> rusqlite::Result<()> {
+pub fn migrate(conn: &mut Connection) -> Result<(), Error> {
     let version = get_schema_version(conn)?;
     let stmts = &MIGRATIONS[(version as usize)..];
-    let mut i: i32 = version;
 
-    if version == MIGRATIONS.len() as i32 {
+    // begin transaction, all migration statements and new schema version commit or rollback
+    let tx = conn.transaction()?;
+
+    // execute every statement and return `Some` new schema version
+    // if execution fails, return `Error::Rusqlite`
+    // if no statements executed returns `None`
+    let new_version = stmts
+        .iter()
+        .enumerate()
+        .map(|version_stmt| {
+            log::info!(
+                "executing db migration {}: `{}`",
+                version + version_stmt.0 as i32 + 1,
+                version_stmt.1
+            );
+            tx.execute(version_stmt.1, [])
+                // map result value to next migration version
+                .map(|_| version_stmt.0 as i32 + version + 1)
+        })
+        .last()
+        .transpose()?;
+
+    // if `Some` new statement version, set new schema version
+    if let Some(version) = new_version {
+        set_schema_version(&tx, version)?;
+    } else {
         log::info!("db up to date, no migration needed");
-        return Ok(());
     }
 
-    for stmt in stmts {
+    // commit transaction
-        let res = conn.execute(stmt, []);
+    tx.commit()?;
-        if res.is_err() {
-            println!("migration failed on:\n{}\n{:?}", stmt, res);
-            break;
-        }
-
-        i += 1;
-    }
-
-    set_schema_version(conn, i)?;
-
     Ok(())
 }
 
@@ -1036,4 +1056,44 @@ pub mod test {
     fn test_txs() {
         crate::database::test::test_list_transaction(get_database());
     }
+
+    #[test]
+    fn test_iter_raw_txs() {
+        crate::database::test::test_iter_raw_txs(get_database());
+    }
+
+    #[test]
+    fn test_del_path_from_script_pubkey() {
+        crate::database::test::test_del_path_from_script_pubkey(get_database());
+    }
+
+    #[test]
+    fn test_iter_script_pubkeys() {
+        crate::database::test::test_iter_script_pubkeys(get_database());
+    }
+
+    #[test]
+    fn test_del_utxo() {
+        crate::database::test::test_del_utxo(get_database());
+    }
+
+    #[test]
+    fn test_del_raw_tx() {
+        crate::database::test::test_del_raw_tx(get_database());
+    }
+
+    #[test]
+    fn test_del_tx() {
+        crate::database::test::test_del_tx(get_database());
+    }
+
+    #[test]
+    fn test_del_last_index() {
+        crate::database::test::test_del_last_index(get_database());
+    }
+
+    #[test]
+    fn test_check_descriptor_checksum() {
+        crate::database::test::test_check_descriptor_checksum(get_database());
+    }
 }

src/descriptor/error.rs

@@ -20,8 +20,6 @@ pub enum Error {
     InvalidDescriptorChecksum,
     /// The descriptor contains hardened derivation steps on public extended keys
     HardenedDerivationXpub,
-    /// The descriptor contains multiple keys with the same BIP32 fingerprint
-    DuplicatedKeys,
 
     /// Error thrown while working with [`keys`](crate::keys)
     Key(crate::keys::KeyError),

src/descriptor/mod.rs

@@ -14,7 +14,7 @@
 //! This module contains generic utilities to work with descriptors, plus some re-exported types
 //! from [`miniscript`].
 
-use std::collections::{BTreeMap, HashSet};
+use std::collections::BTreeMap;
 use std::ops::Deref;
 
 use bitcoin::util::bip32::{ChildNumber, DerivationPath, ExtendedPubKey, Fingerprint, KeySource};
@@ -222,23 +222,9 @@ pub(crate) fn into_wallet_descriptor_checked<T: IntoWalletDescriptor>(
         return Err(DescriptorError::HardenedDerivationXpub);
     }
 
-    // Ensure that there are no duplicated keys
+    // Run miniscript's sanity check, which will look for duplicated keys and other potential
-    let mut found_keys = HashSet::new();
+    // issues
-    let descriptor_contains_duplicated_keys = descriptor.for_any_key(|k| {
+    descriptor.sanity_check()?;
-        if let DescriptorPublicKey::XPub(xkey) = k.as_key() {
-            let fingerprint = xkey.root_fingerprint(secp);
-            if found_keys.contains(&fingerprint) {
-                return true;
-            }
-
-            found_keys.insert(fingerprint);
-        }
-
-        false
-    });
-    if descriptor_contains_duplicated_keys {
-        return Err(DescriptorError::DuplicatedKeys);
-    }
 
     Ok((descriptor, keymap))
 }
@@ -923,14 +909,10 @@ mod test {
             DescriptorError::HardenedDerivationXpub
         ));
 
-        let descriptor = "wsh(multi(2,tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/0/*,tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/1/*))";
+        let descriptor = "wsh(multi(2,tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/0/*,tpubD6NzVbkrYhZ4XHndKkuB8FifXm8r5FQHwrN6oZuWCz13qb93rtgKvD4PQsqC4HP4yhV3tA2fqr2RbY5mNXfM7RxXUoeABoDtsFUq2zJq6YK/0/*))";
         let result = into_wallet_descriptor_checked(descriptor, &secp, Network::Testnet);
 
         assert!(result.is_err());
-        assert!(matches!(
-            result.unwrap_err(),
-            DescriptorError::DuplicatedKeys
-        ));
     }
 
     #[test]

src/lib.rs

@@ -231,15 +231,15 @@ extern crate async_trait;
 #[macro_use]
 extern crate bdk_macros;
 
-#[cfg(feature = "compact_filters")]
-extern crate lazy_static;
-
 #[cfg(feature = "rpc")]
 pub extern crate bitcoincore_rpc;
 
 #[cfg(feature = "electrum")]
 pub extern crate electrum_client;
 
+#[cfg(feature = "esplora")]
+pub extern crate esplora_client;
+
 #[cfg(feature = "key-value-db")]
 pub extern crate sled;
 

src/psbt/mod.rs

@@ -9,11 +9,22 @@
 // You may not use this file except in accordance with one or both of these
 // licenses.
 
+use crate::FeeRate;
 use bitcoin::util::psbt::PartiallySignedTransaction as Psbt;
 use bitcoin::TxOut;
 
 pub trait PsbtUtils {
     fn get_utxo_for(&self, input_index: usize) -> Option<TxOut>;
+
+    /// The total transaction fee amount, sum of input amounts minus sum of output amounts, in Sats.
+    /// If the PSBT is missing a TxOut for an input returns None.
+    fn fee_amount(&self) -> Option<u64>;
+
+    /// The transaction's fee rate. This value will only be accurate if calculated AFTER the
+    /// `PartiallySignedTransaction` is finalized and all witness/signature data is added to the
+    /// transaction.
+    /// If the PSBT is missing a TxOut for an input returns None.
+    fn fee_rate(&self) -> Option<FeeRate>;
 }
 
 impl PsbtUtils for Psbt {
@@ -37,6 +48,27 @@ impl PsbtUtils for Psbt {
             None
         }
     }
+
+    fn fee_amount(&self) -> Option<u64> {
+        let tx = &self.unsigned_tx;
+        let utxos: Option<Vec<TxOut>> = (0..tx.input.len()).map(|i| self.get_utxo_for(i)).collect();
+
+        utxos.map(|inputs| {
+            let input_amount: u64 = inputs.iter().map(|i| i.value).sum();
+            let output_amount: u64 = self.unsigned_tx.output.iter().map(|o| o.value).sum();
+            input_amount
+                .checked_sub(output_amount)
+                .expect("input amount must be greater than output amount")
+        })
+    }
+
+    fn fee_rate(&self) -> Option<FeeRate> {
+        let fee_amount = self.fee_amount();
+        fee_amount.map(|fee| {
+            let weight = self.clone().extract_tx().weight();
+            FeeRate::from_wu(fee, weight)
+        })
+    }
 }
 
 #[cfg(test)]
@@ -44,8 +76,9 @@ mod test {
     use crate::bitcoin::TxIn;
     use crate::psbt::Psbt;
     use crate::wallet::AddressIndex;
+    use crate::wallet::AddressIndex::New;
     use crate::wallet::{get_funded_wallet, test::get_test_wpkh};
-    use crate::SignOptions;
+    use crate::{psbt, FeeRate, SignOptions};
     use std::str::FromStr;
 
     // from bip 174
@@ -118,4 +151,83 @@ mod test {
 
         let _ = wallet.sign(&mut psbt, SignOptions::default()).unwrap();
     }
+
+    #[test]
+    fn test_psbt_fee_rate_with_witness_utxo() {
+        use psbt::PsbtUtils;
+
+        let expected_fee_rate = 1.2345;
+
+        let (wallet, _, _) = get_funded_wallet("wpkh(tprv8ZgxMBicQKsPd3EupYiPRhaMooHKUHJxNsTfYuScep13go8QFfHdtkG9nRkFGb7busX4isf6X9dURGCoKgitaApQ6MupRhZMcELAxTBRJgS/*)");
+        let addr = wallet.get_address(New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.drain_to(addr.script_pubkey()).drain_wallet();
+        builder.fee_rate(FeeRate::from_sat_per_vb(expected_fee_rate));
+        let (mut psbt, _) = builder.finish().unwrap();
+        let fee_amount = psbt.fee_amount();
+        assert!(fee_amount.is_some());
+
+        let unfinalized_fee_rate = psbt.fee_rate().unwrap();
+
+        let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
+        assert!(finalized);
+
+        let finalized_fee_rate = psbt.fee_rate().unwrap();
+        assert!(finalized_fee_rate.as_sat_per_vb() >= expected_fee_rate);
+        assert!(finalized_fee_rate.as_sat_per_vb() < unfinalized_fee_rate.as_sat_per_vb());
+    }
+
+    #[test]
+    fn test_psbt_fee_rate_with_nonwitness_utxo() {
+        use psbt::PsbtUtils;
+
+        let expected_fee_rate = 1.2345;
+
+        let (wallet, _, _) = get_funded_wallet("pkh(tprv8ZgxMBicQKsPd3EupYiPRhaMooHKUHJxNsTfYuScep13go8QFfHdtkG9nRkFGb7busX4isf6X9dURGCoKgitaApQ6MupRhZMcELAxTBRJgS/*)");
+        let addr = wallet.get_address(New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.drain_to(addr.script_pubkey()).drain_wallet();
+        builder.fee_rate(FeeRate::from_sat_per_vb(expected_fee_rate));
+        let (mut psbt, _) = builder.finish().unwrap();
+        let fee_amount = psbt.fee_amount();
+        assert!(fee_amount.is_some());
+        let unfinalized_fee_rate = psbt.fee_rate().unwrap();
+
+        let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
+        assert!(finalized);
+
+        let finalized_fee_rate = psbt.fee_rate().unwrap();
+        assert!(finalized_fee_rate.as_sat_per_vb() >= expected_fee_rate);
+        assert!(finalized_fee_rate.as_sat_per_vb() < unfinalized_fee_rate.as_sat_per_vb());
+    }
+
+    #[test]
+    fn test_psbt_fee_rate_with_missing_txout() {
+        use psbt::PsbtUtils;
+
+        let expected_fee_rate = 1.2345;
+
+        let (wpkh_wallet, _, _) = get_funded_wallet("wpkh(tprv8ZgxMBicQKsPd3EupYiPRhaMooHKUHJxNsTfYuScep13go8QFfHdtkG9nRkFGb7busX4isf6X9dURGCoKgitaApQ6MupRhZMcELAxTBRJgS/*)");
+        let addr = wpkh_wallet.get_address(New).unwrap();
+        let mut builder = wpkh_wallet.build_tx();
+        builder.drain_to(addr.script_pubkey()).drain_wallet();
+        builder.fee_rate(FeeRate::from_sat_per_vb(expected_fee_rate));
+        let (mut wpkh_psbt, _) = builder.finish().unwrap();
+
+        wpkh_psbt.inputs[0].witness_utxo = None;
+        wpkh_psbt.inputs[0].non_witness_utxo = None;
+        assert!(wpkh_psbt.fee_amount().is_none());
+        assert!(wpkh_psbt.fee_rate().is_none());
+
+        let (pkh_wallet, _, _) = get_funded_wallet("pkh(tprv8ZgxMBicQKsPd3EupYiPRhaMooHKUHJxNsTfYuScep13go8QFfHdtkG9nRkFGb7busX4isf6X9dURGCoKgitaApQ6MupRhZMcELAxTBRJgS/*)");
+        let addr = pkh_wallet.get_address(New).unwrap();
+        let mut builder = pkh_wallet.build_tx();
+        builder.drain_to(addr.script_pubkey()).drain_wallet();
+        builder.fee_rate(FeeRate::from_sat_per_vb(expected_fee_rate));
+        let (mut pkh_psbt, _) = builder.finish().unwrap();
+
+        pkh_psbt.inputs[0].non_witness_utxo = None;
+        assert!(pkh_psbt.fee_amount().is_none());
+        assert!(pkh_psbt.fee_rate().is_none());
+    }
 }

src/wallet/mod.rs

@@ -1196,7 +1196,12 @@ where
         }
     }
 
-    /// Try to finalize a PSBT
+    /// Finalize a PSBT, i.e., for each input determine if sufficient data is available to pass
+    /// validation and construct the respective `scriptSig` or `scriptWitness`. Please refer to
+    /// [BIP174](https://github.com/bitcoin/bips/blob/master/bip-0174.mediawiki#Input_Finalizer)
+    /// for further information.
+    ///
+    /// Returns `true` if the PSBT could be finalized, and `false` otherwise.
     ///
     /// The [`SignOptions`] can be used to tweak the behavior of the finalizer.
     pub fn finalize_psbt(
@@ -1837,7 +1842,7 @@ where
             .to_string()
             .split_once('#')
             .unwrap()
-            .0
+            .1
             .to_string()
     }
 }
@@ -1933,6 +1938,22 @@ pub(crate) mod test {
     // OP_PUSH.
     const P2WPKH_FAKE_WITNESS_SIZE: usize = 106;
 
+    #[test]
+    fn test_descriptor_checksum() {
+        let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
+        let checksum = wallet.descriptor_checksum(KeychainKind::External);
+        assert_eq!(checksum.len(), 8);
+
+        let raw_descriptor = wallet
+            .descriptor
+            .to_string()
+            .split_once('#')
+            .unwrap()
+            .0
+            .to_string();
+        assert_eq!(get_checksum(&raw_descriptor).unwrap(), checksum);
+    }
+
     #[test]
     fn test_get_funded_wallet_balance() {
         let (wallet, _, _) = get_funded_wallet(get_test_wpkh());
@@ -2072,6 +2093,10 @@ pub(crate) mod test {
         "tr(cNJmN3fH9DDbDt131fQNkVakkpzawJBSeybCUNmP1BovpmGQ45xG,{pk(tprv8ZgxMBicQKsPdDArR4xSAECuVxeX1jwwSXR4ApKbkYgZiziDc4LdBy2WvJeGDfUSE4UT4hHhbgEwbdq8ajjUHiKDegkwrNU6V55CxcxonVN/*),pk(8aee2b8120a5f157f1223f72b5e62b825831a27a9fdf427db7cc697494d4a642)})"
     }
 
+    pub(crate) fn get_test_tr_dup_keys() -> &'static str {
+        "tr(cNJmN3fH9DDbDt131fQNkVakkpzawJBSeybCUNmP1BovpmGQ45xG,{pk(8aee2b8120a5f157f1223f72b5e62b825831a27a9fdf427db7cc697494d4a642),pk(8aee2b8120a5f157f1223f72b5e62b825831a27a9fdf427db7cc697494d4a642)})"
+    }
+
     macro_rules! assert_fee_rate {
         ($psbt:expr, $fees:expr, $fee_rate:expr $( ,@dust_change $( $dust_change:expr )* )* $( ,@add_signature $( $add_signature:expr )* )* ) => ({
             let psbt = $psbt.clone();
@@ -4959,6 +4984,29 @@ pub(crate) mod test {
         );
     }
 
+    #[test]
+    fn test_taproot_sign_using_non_witness_utxo() {
+        let (wallet, _, prev_txid) = get_funded_wallet(get_test_tr_single_sig());
+        let addr = wallet.get_address(New).unwrap();
+        let mut builder = wallet.build_tx();
+        builder.drain_to(addr.script_pubkey()).drain_wallet();
+        let (mut psbt, _) = builder.finish().unwrap();
+
+        psbt.inputs[0].witness_utxo = None;
+        psbt.inputs[0].non_witness_utxo = wallet.database().get_raw_tx(&prev_txid).unwrap();
+        assert!(
+            psbt.inputs[0].non_witness_utxo.is_some(),
+            "Previous tx should be present in the database"
+        );
+
+        let result = wallet.sign(&mut psbt, Default::default());
+        assert!(result.is_ok(), "Signing should have worked");
+        assert!(
+            result.unwrap(),
+            "Should finalize the input since we can produce signatures"
+        );
+    }
+
     #[test]
     fn test_taproot_foreign_utxo() {
         let (wallet1, _, _) = get_funded_wallet(get_test_wpkh());
@@ -5526,4 +5574,19 @@ pub(crate) mod test {
         let finalized = wallet.sign(&mut psbt, Default::default()).unwrap();
         assert!(finalized);
     }
+
+    #[test]
+    fn test_taproot_load_descriptor_duplicated_keys() {
+        // Added after issue https://github.com/bitcoindevkit/bdk/issues/760
+        //
+        // Having the same key in multiple taproot leaves is safe and should be accepted by BDK
+
+        let (wallet, _, _) = get_funded_wallet(get_test_tr_dup_keys());
+        let addr = wallet.get_address(New).unwrap();
+
+        assert_eq!(
+            addr.to_string(),
+            "bcrt1pvysh4nmh85ysrkpwtrr8q8gdadhgdejpy6f9v424a8v9htjxjhyqw9c5s5"
+        );
+    }
 }

src/wallet/signer.rs

@@ -103,6 +103,7 @@ use miniscript::{Legacy, MiniscriptKey, Segwitv0, Tap};
 
 use super::utils::SecpCtx;
 use crate::descriptor::{DescriptorMeta, XKeyUtils};
+use crate::psbt::PsbtUtils;
 
 /// Identifier of a signer in the `SignersContainers`. Used as a key to find the right signer among
 /// multiple of them
@@ -696,14 +697,14 @@ pub struct SignOptions {
     /// Defaults to `false` which will only allow signing using `SIGHASH_ALL`.
     pub allow_all_sighashes: bool,
 
-    /// Whether to remove partial_sigs from psbt inputs while finalizing psbt.
+    /// Whether to remove partial signatures from the PSBT inputs while finalizing PSBT.
     ///
-    /// Defaults to `true` which will remove partial_sigs after finalizing.
+    /// Defaults to `true` which will remove partial signatures during finalization.
     pub remove_partial_sigs: bool,
 
-    /// Whether to try finalizing psbt input after the inputs are signed.
+    /// Whether to try finalizing the PSBT after the inputs are signed.
     ///
-    /// Defaults to `true` which will try fianlizing psbt after inputs are signed.
+    /// Defaults to `true` which will try finalizing PSBT after inputs are signed.
     pub try_finalize: bool,
 
     /// Specifies which Taproot script-spend leaves we should sign for. This option is
@@ -921,11 +922,8 @@ impl ComputeSighash for Tap {
             .unwrap_or_else(|| SchnorrSighashType::Default.into())
             .schnorr_hash_ty()
             .map_err(|_| SignerError::InvalidSighash)?;
-        let witness_utxos = psbt
+        let witness_utxos = (0..psbt.inputs.len())
-            .inputs
+            .map(|i| psbt.get_utxo_for(i))
-            .iter()
-            .cloned()
-            .map(|i| i.witness_utxo)
             .collect::<Vec<_>>();
         let mut all_witness_utxos = vec![];
 

src/wallet/verify.rs

@@ -29,6 +29,8 @@ use crate::error::Error;
 /// Depending on the [capabilities](crate::blockchain::Blockchain::get_capabilities) of the
 /// [`Blockchain`] backend, the method could fail when called with old "historical" transactions or
 /// with unconfirmed transactions that have been evicted from the backend's memory.
+///
+/// [`Blockchain`]: crate::blockchain::Blockchain
 pub fn verify_tx<D: Database, B: GetTx>(
     tx: &Transaction,
     database: &D,