frost/bdk
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge bitcoindevkit/bdk#1380: Simplified EsploraExt API

Browse Source 
96a9aa6e63 feat(chain): refactor `merge_chains` (志宇)
2f22987c9e chore(chain): fix comment (志宇)
daf588f016 feat(chain): optimize `merge_chains` (志宇)
77d35954c1 feat(chain)!: rm `local_chain::Update` (志宇)
1269b0610e test(chain): fix incorrect test case (志宇)
72fe65b65f feat(esplora)!: simplify chain update logic (志宇)
eded1a7ea0 feat(chain): introduce `CheckPoint::insert` (志宇)
519cd75d23 test(esplora): move esplora tests into src files (志宇)
a6e613e6b9 test(esplora): add `test_finalize_chain_update` (志宇)
494d253493 feat(testenv): add `genesis_hash` method (志宇)
886d72e3d5 chore(chain)!: rm `missing_heights` and `missing_heights_from` methods (志宇)
bd62aa0fe1 feat(esplora)!: remove `EsploraExt::update_local_chain` (志宇)
1e99793983 feat(testenv): add `make_checkpoint_tip` (志宇)

Pull request description:

  Fixes #1354

  ### Description

  Built on top of both #1369 and #1373, we simplify the `EsploraExt` API by removing the `update_local_chain` method and having `full_scan` and `sync` update the local chain in the same call. The `full_scan` and `sync` methods now takes in an additional input (`local_tip`) which provides us with the view of the `LocalChain` before the update. These methods now return structs `FullScanUpdate` and `SyncUpdate`.

  The examples are updated to use this new API. `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` are no longer needed, therefore they are removed.

  Additionally, we used this opportunity to simplify the logic which updates `LocalChain`. We got rid of the `local_chain::Update` struct (which contained the update `CheckPoint` tip and a `bool` which signaled whether we want to introduce blocks below point of agreement). It turns out we can use something like `CheckPoint::insert` so the chain source can craft an update based on the old tip. This way, we can make better use of `merge_chains`' optimization that compares the `Arc` pointers of the local and update chain (before we were crafting the update chain NOT based on top of the previous local chain). With this, we no longer need the `Update::introduce_older_block` field since the logic will naturally break when we reach a matching `Arc` pointer.

  ### Notes to the reviewers

  * Obtaining the `LocalChain`'s update now happens within `EsploraExt::full_scan` and `EsploraExt::sync`. Creating the `LocalChain` update is now split into two methods (`fetch_latest_blocks` and `chain_update`) that are called before and after fetching transactions and anchors.
  * We need to duplicate code for `bdk_esplora`. One for blocking and one for async.

  ### Changelog notice

  * Changed `EsploraExt` API so that sync only requires one round of fetching data. The `local_chain_update` method is removed and the `local_tip` parameter is added to the `full_scan` and `sync` methods.
  * Removed `TxGraph::missing_heights` and `tx_graph::ChangeSet::missing_heights_from` methods.
  * Introduced `CheckPoint::insert` which allows convenient checkpoint-insertion. This is intended for use by chain-sources when crafting an update.
  * Refactored `merge_chains` to also return the resultant `CheckPoint` tip.
  * Optimized the update `LocalChain` logic - use the update `CheckPoint` as the new `CheckPoint` tip when possible.

  ### Checklists

  #### All Submissions:

  * [x] I've signed all my commits
  * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
  * [x] I ran `cargo fmt` and `cargo clippy` before committing

  #### New Features:

  * [x] I've added tests for the new feature
  * [x] I've added docs for the new feature

ACKs for top commit:
  LLFourn:
    ACK 96a9aa6e63

Tree-SHA512: 3d4f2eab08a1fe94eb578c594126e99679f72e231680b2edd4bfb018ba1d998ca123b07acb2d19c644d5887fc36b8e42badba91cd09853df421ded04de45bf69
This commit is contained in:
志宇
2024-04-22 17:44:44 +08:00
parent 9800f8d88e 96a9aa6e63
commit 8e73998cfa
19 changed files with 1609 additions and 1120 deletions
Download Patch File Download Diff File Expand all files Collapse all files

225
crates/chain/src/local_chain.rs
View File

@@ -96,16 +96,6 @@ impl CheckPoint {
.expect("must construct checkpoint")
}
/// Convenience method to convert the [`CheckPoint`] into an [`Update`].
///
/// For more information, refer to [`Update`].
pub fn into_update(self, introduce_older_blocks: bool) -> Update {
Update {
tip: self,
introduce_older_blocks,
}
}
/// Puts another checkpoint onto the linked list representing the blockchain.
///
/// Returns an `Err(self)` if the block you are pushing on is not at a greater height that the one you
@@ -187,6 +177,82 @@ impl CheckPoint {
core::ops::Bound::Unbounded => true,
})
}
/// Inserts `block_id` at its height within the chain.
///
/// The effect of `insert` depends on whether a height already exists. If it doesn't the
/// `block_id` we inserted and all pre-existing blocks higher than it will be re-inserted after
/// it. If the height already existed and has a conflicting block hash then it will be purged
/// along with all block followin it. The returned chain will have a tip of the `block_id`
/// passed in. Of course, if the `block_id` was already present then this just returns `self`.
#[must_use]
pub fn insert(self, block_id: BlockId) -> Self {
assert_ne!(block_id.height, 0, "cannot insert the genesis block");
let mut cp = self.clone();
let mut tail = vec![];
let base = loop {
if cp.height() == block_id.height {
if cp.hash() == block_id.hash {
return self;
}
// if we have a conflict we just return the inserted block because the tail is by
// implication invalid.
tail = vec![];
break cp.prev().expect("can't be called on genesis block");
}
if cp.height() < block_id.height {
break cp;
}
tail.push(cp.block_id());
cp = cp.prev().expect("will break before genesis block");
};
base.extend(core::iter::once(block_id).chain(tail.into_iter().rev()))
.expect("tail is in order")
}
/// Apply `changeset` to the checkpoint.
fn apply_changeset(mut self, changeset: &ChangeSet) -> Result<CheckPoint, MissingGenesisError> {
if let Some(start_height) = changeset.keys().next().cloned() {
// changes after point of agreement
let mut extension = BTreeMap::default();
// point of agreement
let mut base: Option<CheckPoint> = None;
for cp in self.iter() {
if cp.height() >= start_height {
extension.insert(cp.height(), cp.hash());
} else {
base = Some(cp);
break;
}
}
for (&height, &hash) in changeset {
match hash {
Some(hash) => {
extension.insert(height, hash);
}
None => {
extension.remove(&height);
}
};
}
let new_tip = match base {
Some(base) => base
.extend(extension.into_iter().map(BlockId::from))
.expect("extension is strictly greater than base"),
None => LocalChain::from_blocks(extension)?.tip(),
};
self = new_tip;
}
Ok(self)
}
}
/// Iterates over checkpoints backwards.
@@ -215,31 +281,6 @@ impl IntoIterator for CheckPoint {
}
}
/// Used to update [`LocalChain`].
///
/// This is used as input for [`LocalChain::apply_update`]. It contains the update's chain `tip` and
/// a flag `introduce_older_blocks` which signals whether this update intends to introduce missing
/// blocks to the original chain.
///
/// Block-by-block syncing mechanisms would typically create updates that builds upon the previous
/// tip. In this case, `introduce_older_blocks` would be `false`.
///
/// Script-pubkey based syncing mechanisms may not introduce transactions in a chronological order
/// so some updates require introducing older blocks (to anchor older transactions). For
/// script-pubkey based syncing, `introduce_older_blocks` would typically be `true`.
#[derive(Debug, Clone, PartialEq)]
pub struct Update {
/// The update chain's new tip.
pub tip: CheckPoint,
/// Whether the update allows for introducing older blocks.
///
/// Refer to [struct-level documentation] for more.
///
/// [struct-level documentation]: Update
pub introduce_older_blocks: bool,
}
/// This is a local implementation of [`ChainOracle`].
#[derive(Debug, Clone, PartialEq)]
pub struct LocalChain {
@@ -347,36 +388,22 @@ impl LocalChain {
/// Applies the given `update` to the chain.
///
/// The method returns [`ChangeSet`] on success. This represents the applied changes to `self`.
/// The method returns [`ChangeSet`] on success. This represents the changes applied to `self`.
///
/// There must be no ambiguity about which of the existing chain's blocks are still valid and
/// which are now invalid. That is, the new chain must implicitly connect to a definite block in
/// the existing chain and invalidate the block after it (if it exists) by including a block at
/// the same height but with a different hash to explicitly exclude it as a connection point.
///
/// Additionally, an empty chain can be updated with any chain, and a chain with a single block
/// can have it's block invalidated by an update chain with a block at the same height but
/// different hash.
///
/// # Errors
///
/// An error will occur if the update does not correctly connect with `self`.
///
/// Refer to [`Update`] for more about the update struct.
///
/// [module-level documentation]: crate::local_chain
pub fn apply_update(&mut self, update: Update) -> Result<ChangeSet, CannotConnectError> {
let changeset = merge_chains(
self.tip.clone(),
update.tip.clone(),
update.introduce_older_blocks,
)?;
// `._check_index_is_consistent_with_tip` and `._check_changeset_is_applied` is called in
// `.apply_changeset`
self.apply_changeset(&changeset)
.map_err(|_| CannotConnectError {
try_include_height: 0,
})?;
pub fn apply_update(&mut self, update: CheckPoint) -> Result<ChangeSet, CannotConnectError> {
let (new_tip, changeset) = merge_chains(self.tip.clone(), update)?;
self.tip = new_tip;
self._check_changeset_is_applied(&changeset);
Ok(changeset)
}
@@ -428,11 +455,8 @@ impl LocalChain {
conn => Some(conn),
};
let update = Update {
tip: CheckPoint::from_block_ids([conn, prev, Some(this)].into_iter().flatten())
.expect("block ids must be in order"),
introduce_older_blocks: false,
};
let update = CheckPoint::from_block_ids([conn, prev, Some(this)].into_iter().flatten())
.expect("block ids must be in order");
self.apply_update(update)
.map_err(ApplyHeaderError::CannotConnect)
@@ -471,43 +495,10 @@ impl LocalChain {
/// Apply the given `changeset`.
pub fn apply_changeset(&mut self, changeset: &ChangeSet) -> Result<(), MissingGenesisError> {
if let Some(start_height) = changeset.keys().next().cloned() {
// changes after point of agreement
let mut extension = BTreeMap::default();
// point of agreement
let mut base: Option<CheckPoint> = None;
for cp in self.iter_checkpoints() {
if cp.height() >= start_height {
extension.insert(cp.height(), cp.hash());
} else {
base = Some(cp);
break;
}
}
for (&height, &hash) in changeset {
match hash {
Some(hash) => {
extension.insert(height, hash);
}
None => {
extension.remove(&height);
}
};
}
let new_tip = match base {
Some(base) => base
.extend(extension.into_iter().map(BlockId::from))
.expect("extension is strictly greater than base"),
None => LocalChain::from_blocks(extension)?.tip(),
};
self.tip = new_tip;
debug_assert!(self._check_changeset_is_applied(changeset));
}
let old_tip = self.tip.clone();
let new_tip = old_tip.apply_changeset(changeset)?;
self.tip = new_tip;
debug_assert!(self._check_changeset_is_applied(changeset));
Ok(())
}
@@ -730,14 +721,17 @@ impl core::fmt::Display for ApplyHeaderError {
#[cfg(feature = "std")]
impl std::error::Error for ApplyHeaderError {}
/// Applies `update_tip` onto `original_tip`.
///
/// On success, a tuple is returned `(changeset, can_replace)`. If `can_replace` is true, then the
/// `update_tip` can replace the `original_tip`.
fn merge_chains(
original_tip: CheckPoint,
update_tip: CheckPoint,
introduce_older_blocks: bool,
) -> Result<ChangeSet, CannotConnectError> {
) -> Result<(CheckPoint, ChangeSet), CannotConnectError> {
let mut changeset = ChangeSet::default();
let mut orig = original_tip.into_iter();
let mut update = update_tip.into_iter();
let mut orig = original_tip.iter();
let mut update = update_tip.iter();
let mut curr_orig = None;
let mut curr_update = None;
let mut prev_orig: Option<CheckPoint> = None;
@@ -746,6 +740,12 @@ fn merge_chains(
let mut prev_orig_was_invalidated = false;
let mut potentially_invalidated_heights = vec![];
// If we can, we want to return the update tip as the new tip because this allows checkpoints
// in multiple locations to keep the same `Arc` pointers when they are being updated from each
// other using this function. We can do this as long as long as the update contains every
// block's height of the original chain.
let mut is_update_height_superset_of_original = true;
// To find the difference between the new chain and the original we iterate over both of them
// from the tip backwards in tandem. We always dealing with the highest one from either chain
// first and move to the next highest. The crucial logic is applied when they have blocks at the
@@ -771,6 +771,8 @@ fn merge_chains(
prev_orig_was_invalidated = false;
prev_orig = curr_orig.take();
is_update_height_superset_of_original = false;
// OPTIMIZATION: we have run out of update blocks so we don't need to continue
// iterating because there's no possibility of adding anything to changeset.
if u.is_none() {
@@ -793,12 +795,20 @@ fn merge_chains(
}
point_of_agreement_found = true;
prev_orig_was_invalidated = false;
// OPTIMIZATION 1 -- If we know that older blocks cannot be introduced without
// invalidation, we can break after finding the point of agreement.
// OPTIMIZATION 2 -- if we have the same underlying pointer at this point, we
// can guarantee that no older blocks are introduced.
if !introduce_older_blocks || Arc::as_ptr(&o.0) == Arc::as_ptr(&u.0) {
return Ok(changeset);
if Arc::as_ptr(&o.0) == Arc::as_ptr(&u.0) {
if is_update_height_superset_of_original {
return Ok((update_tip, changeset));
} else {
let new_tip =
original_tip.apply_changeset(&changeset).map_err(|_| {
CannotConnectError {
try_include_height: 0,
}
})?;
return Ok((new_tip, changeset));
}
}
} else {
// We have an invalidation height so we set the height to the updated hash and
@@ -832,5 +842,10 @@ fn merge_chains(
}
}
Ok(changeset)
let new_tip = original_tip
.apply_changeset(&changeset)
.map_err(|_| CannotConnectError {
try_include_height: 0,
})?;
Ok((new_tip, changeset))
}

87
crates/chain/src/tx_graph.rs
View File

@@ -89,8 +89,8 @@
//! [`insert_txout`]: TxGraph::insert_txout
use crate::{
collections::*, keychain::Balance, local_chain::LocalChain, Anchor, Append, BlockId,
ChainOracle, ChainPosition, FullTxOut,
collections::*, keychain::Balance, Anchor, Append, BlockId, ChainOracle, ChainPosition,
FullTxOut,
};
use alloc::collections::vec_deque::VecDeque;
use alloc::sync::Arc;
@@ -759,69 +759,6 @@ impl<A: Clone + Ord> TxGraph<A> {
}
impl<A: Anchor> TxGraph<A> {
/// Find missing block heights of `chain`.
///
/// This works by scanning through anchors, and seeing whether the anchor block of the anchor
/// exists in the [`LocalChain`]. The returned iterator does not output duplicate heights.
pub fn missing_heights<'a>(&'a self, chain: &'a LocalChain) -> impl Iterator<Item = u32> + 'a {
// Map of txids to skip.
//
// Usually, if a height of a tx anchor is missing from the chain, we would want to return
// this height in the iterator. The exception is when the tx is confirmed in chain. All the
// other missing-height anchors of this tx can be skipped.
//
// * Some(true) => skip all anchors of this txid
// * Some(false) => do not skip anchors of this txid
// * None => we do not know whether we can skip this txid
let mut txids_to_skip = HashMap::<Txid, bool>::new();
// Keeps track of the last height emitted so we don't double up.
let mut last_height_emitted = Option::<u32>::None;
self.anchors
.iter()
.filter(move |(_, txid)| {
let skip = *txids_to_skip.entry(*txid).or_insert_with(|| {
let tx_anchors = match self.txs.get(txid) {
Some((_, anchors, _)) => anchors,
None => return true,
};
let mut has_missing_height = false;
for anchor_block in tx_anchors.iter().map(Anchor::anchor_block) {
match chain.get(anchor_block.height) {
None => {
has_missing_height = true;
continue;
}
Some(chain_cp) => {
if chain_cp.hash() == anchor_block.hash {
return true;
}
}
}
}
!has_missing_height
});
#[cfg(feature = "std")]
debug_assert!({
println!("txid={} skip={}", txid, skip);
true
});
!skip
})
.filter_map(move |(a, _)| {
let anchor_block = a.anchor_block();
if Some(anchor_block.height) != last_height_emitted
&& chain.get(anchor_block.height).is_none()
{
last_height_emitted = Some(anchor_block.height);
Some(anchor_block.height)
} else {
None
}
})
}
/// Get the position of the transaction in `chain` with tip `chain_tip`.
///
/// Chain data is fetched from `chain`, a [`ChainOracle`] implementation.
@@ -1330,8 +1267,6 @@ impl<A> ChangeSet<A> {
///
/// This is useful if you want to find which heights you need to fetch data about in order to
/// confirm or exclude these anchors.
///
/// See also: [`TxGraph::missing_heights`]
pub fn anchor_heights(&self) -> impl Iterator<Item = u32> + '_
where
A: Anchor,
@@ -1346,24 +1281,6 @@ impl<A> ChangeSet<A> {
!duplicate
})
}
/// Returns an iterator for the [`anchor_heights`] in this changeset that are not included in
/// `local_chain`. This tells you which heights you need to include in `local_chain` in order
/// for it to conclusively act as a [`ChainOracle`] for the transaction anchors this changeset
/// will add.
///
/// [`ChainOracle`]: crate::ChainOracle
/// [`anchor_heights`]: Self::anchor_heights
pub fn missing_heights_from<'a>(
&'a self,
local_chain: &'a LocalChain,
) -> impl Iterator<Item = u32> + 'a
where
A: Anchor,
{
self.anchor_heights()
.filter(move |&height| local_chain.get(height).is_none())
}
}
impl<A: Ord> Append for ChangeSet<A> {