Roadmap
Milestones
chore: remove dead code
Skip-Changelog
feat(ci): track nightly failures in GitHub issues
Make sure failures during nightlies do not get lost by creating issues for them. Those issues will be updated/closed if/when associated nightly run changes/resolves.
fix(node): default peer listener to port 3001
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
Bomanaps/forging cli todos
I addressed two TODOs that are from #1505 the envelope loaders now take the protocol version as a parameter instead of hard-coding v12, and I added a VrfVerificationKey newtype so mixing VRF and Ed25519 keys is a compile error while wire formats stay unchanged, for the remaining kes::PublicKey TODO, should I move only the kes module to the kernel or vrf too, given it puts the vendored crypto crates into the riscv32 amaru-ledger build?
feat(amaru): add `amaru peer probe` for N2N peer diagnostics
## Summary - Add `amaru peer probe <TARGETS> [OPTIONS]` for one-shot N2N checks: TCP connect, handshake (default max version 15), tip, peer-share, keep-alive ping, and optional `--peer-check` of shared peers - Targets may be `IP:PORT`, `HOSTNAME:PORT`, or a CIP-0155 SRV DNS name (all usable `_cardano._tcp.<name>` records); comma-separated lists are probed in order - Operator controls: wall-clock `--timeout` (default 5s, fresh budget per `--retry`), step timeouts, live ping output in human mode, `--json`, and docs at `docs/peer-probe.md` (linked from README Monitoring)
fix(ci): fail nightly bootstrap without sync progress
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
fix(ci): improve nightly bootstrap outcomes and diagnostics
* keep all three network tests independent and prevent newer runs from cancelling active runs * print all warnings and errors in Actions logs without making them fail the test * make sure timeouts are reported as failures
ci: clean-up the creation of reports
This PR:
ci: add a manual workflow that republishes the conformance reports
This PR add a manual workflow for republishing conformance reports manually. This is useful when testing a new report format for example.
Basic but non-naive mempool
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 -->
NTP Check on Unix Systems
Closes #1369
test: update the cbor conformance dataset
This PR updates the dataset used to validate the conformance of the Amaru decoders.
feat: allow to import historical blocks
Adds `node bootstrap --historical-blocks` to import historical blocks from Mithril and serve it over N2N BlockFetch.
fix(release): pin zig 0.15.2 for musl builds
Fixes #1558
BlockFetch responder refuses ranges over 1000 blocks with MsgNoBlocks, which breaks clients built against cardano-node
## Summary The BlockFetch responder answers `MsgNoBlocks` to any `MsgRequestRange` covering more than 1000 blocks (`MAX_FETCHED_BLOCKS`, `crates/amaru-protocols/src/blockfetch/responder.rs`). - The refusal is all-or-nothing: a 1001-block range gets nothing, not the first 1000. - It's logged at `debug` level only (`RANGE_REFUSED`, `reason = "exceeds_max_blocks"`). - The mini-protocol spec defines no such limit.
Correct is_valid=false validation
In the Haskell node the "UTXOW" checks (validating witnesses) run regardless of the phase 2 validity. In Amaru, currently, we only "require witnesses" when `is_valid=true`, which means there are blocks we accept that Haskell rejects, and vice versa.
Merge Test Suites
We currently have (at least) two different test suites for the ledger; the `evaluate_ledger_state` tests which partially run CBOR encoded test vectors used by the Haskell implementation, and the JSON fixtures we produce and use ourselves.
Prevent panics when accumulating value
`Balance` stored `coin` as a `u64`, which allowed you to build a transaction which, even though it would be invalid, would cause Amaru to crash. This PR resolves that by accumulating value into an `i128`.
Update packages metadata for v10.11.20261029
Updates packages metadata generated from the published release [`v10.11.20261029`](https://github.com/pragma-org/amaru/releases/tag/v10.11.20261029).
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4){kind=small}
CIP-203 Amaru Payload and Usage
Closes #1360
feat: add cli to manage vrf keys
Added vrf related CLI commands:
test: remove the demos directory
This PR removes the demo showing an Amaru node synchronizing with other Amaru and Haskell nodes.
remainingKESPeriods metric is always 0: computed from the absolute KES period, not the op.cert
### What revision are you using?
Extend ledger-rules conformance coverage.
I took the old conformance corpus, and adapted some of the scenarios to the new format. Interestingly, this still found a few discrepancies that we weren't catching before 🤨...
fix: world race
Fix https://github.com/pragma-org/amaru/actions/runs/40678948630/job/122784412012?pr=1548
add block forging CLI commands
Adds three new subcommands under amaru keys to help operators manage block forging key materials: opcert inspect, kes period, and pool-id closes #1368.
Uplc memory leak fix and further optimisations
Some background work while busy doing other stuff. The main intent was to investigate some of the memory usage in the uplc VM identified by the benchmark; found one memory leak in some of the builtins. Then pursued some "low hanging" optimisations that minimize code changes and maintenance overhead.
Fix donation handling for invalid transactions
This PR corrects the behavior around donation handling with `is_valid ` is false.
Fix Bootstrap Address Witness Checks
Previously, Amaru would skip validating inputs which spend from a bootstrap address (sometimes called a redemption address), and it would panic if such an input was provided as collateral.
feat(stores): add resumable immutable block ranges
Add resumable immutable block-range reads to the chain store and expose them through `NodeRunning::block_reader()`, allowing embedders to process blocks in bounded, cancellable batches from a saved slot/hash checkpoint.
VRF Key Uniqueness Validation - Take 2
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 -->
test: prevent reverse sync in boot connection simulation
Set the serving node's upstream target to zero to prevent it from syncing from the receiving node. Add the failing seed as a regression.
Inexplicable failure with amaru as lib
### What revision are you using?
fix(changelog): typo
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
Fix Collateral Witness Verification
This PR corrects the validation of signatures in collateral; previously Amaru would skip validating the collateral's witness without a redeemer. This was a conformance bug.
chore: use Unreleased as top heading
Make sure PR writes just have to maintain their changelog changes under `## Unreleased`. Consequently modify the release script to query the tagged changelog so that further changes to the unreleased section to not get accidentally added to the release chngelog.
Fix credential validation
This PR ensures we check credentials exist when delegating, with test fixtures to prove it.
fix: cleanup changelog
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
Don't wrap ExUnits values during addition
This PR fixes an issue where the total `ExUnits` value in a transaction, or block, could exceed the maximum value of `u64`, causing the value to wrap, and thus allowing transactions that should be rejected.
Bech32 PoolId Logging
This PR introduces a new `PoolId` type wrapper, which allows us to `Display` a `PoolId` in bech32, which is the standard encoding in the ecosystem.
chore: remove unused commands
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
Ci and readme tweaks
make TUI snappy
Currently, the TUI takes hundreds of milliseconds to react to clicks or key presses for me.
make sure that downstream promotion is done responsibly
If we just always prefer existing connections then there may be a heightened risk of eclipse attacks. We should audit this behaviour
make sure that stopping the node gracefully stops all mini-protoocols
Not sure what the current state is: we should try to cleanly shut down all bearers in consensus before terminating the process (unless of course a second kill signal comes in).
don’t log cardano-cli ping as “WARN mux.failed …”
Currently, such a ping looks like this: ``` 14:42:22 INFO amaru::protocols manager.message.process manager.peer.handshake_completed advertisable=false conn_id=154 full_duplex=false message_type=HandshakeComplete peer=5.135.141.64:59754 role=responder 14:42:22 INFO amaru::protocols manager.peer.local_use_applied conn_id=154 local_use=none peer=5.135.141.64:59754 14:42:22 INFO amaru::protocols peer_selection.peer.remote_use conn_id=154 peer=5.135.141.64:59754 remote_use=diffusion
use connection reset to remove adversarial peers
Currently, ending a peer connection uses orderly TCP shutdown, flushing all data. A misbehaving peer can use this to hold the Manager ransom.
figure out sync slowness with many peers
Currently, sync speed is much faster with one peer than with 100 on my Hetzner box. This is quite likely not limited by network bandwidth, although I haven’t yet done a proper analysis.
tone down ERROR and WARN that shouldn’t be
There’s an ERROR when a peer disconnects (which due to the impolite disconnection process might remain a WARN), and there’s a WARN when we get an invalid transaction (which can happen for entirely legitimate reasons).
allow multiple listeners to be opened
Currently, only one `--listen-address` is allowed, which may prevent listening on IPv4 and IPv6 at the same time.
stage messaging deadlock detection
A deadlocked cycle of stages waiting to send a message to the respective next stage should be recognised by the runtime and kill the node with a clear diagnostic including the involved stages and the types of messages in their mailboxes.
make sure (and document) that TraceBuffer is usable in production
It should basically work already, but has been disabled due to high overhead observed initially.
stop TUI logs from moving when not at most recent position
Currently, scrolling to the desired place in the logs does not stabilize that view, especially DEBUG logging is unusable that way because it keeps scrolling. The first line on display should be fixed unless changed by explicit user interaction (moving the slider, using cursor keys, etc). This applies both to normal and copy modes.
move peer tracking into the PeerPerformance resource
Today, there are lots of messages flowing back and forth between the `manager` (and the network stack) and the `peer_selection` stage. This led to the ad hoc bug fix of grossly enlarging the mailbox sizes of both stages, while the proper fix is to perform the management and querying of shared state without sending messages:
deploy typestate API across amaru-consensus and protocols
Changing the communication behaviour of stages in the graph usually has an influence on the minimum required mailbox size of recipient stages, and since those need careful design, we need to both control changes as well as make the actual communication behaviour accessible to static analysis. Capturing it as typestate allows the construction of typegraphs in code (usually in tests, but possibly even when starting the node with a particular configuration — number of upstreams/downstreams will have an influence on the analysis) to ensure that we don’t make mistakes too easily. It can also guide …
write proper consensus design documentation
We have bits and pieces everywhere, the design is basically working and will need a few tweaks here and there, so it is time now for an honest review and a write-up of the final shape of the consensus and protocols implementation architecture. This includes choices on how to
make pure-stage internals concise and understandable
Currently, `pure-stage` still has almost all features from the beginning with v1 internal structure. Usage has matured enough to pare down the API to what is actually being used outside the crate, then come up with a nice and simple internal implementation that humans can review and work on with confidence, keep the tests (of the retained API) working as is, and in general make it fit for release.
add back-pressure on accepting inbound connections
Currently, one connection is accepted every 100ms and sent to the PeerSelection without flow control. With the PeerSelection mailbox being used by many other purposes, a dedicated back-pressure mechanism should be implemented, e.g. by PeerSelection giving permission for receiving a new connection.
Extend the CBOR conformance coverage
### Abstract
check quality of peers shared by peer_sharing
Markus found the following when looking at the peers shared by Amaru:
Check the conformance of flat decoding
### Abstract
should remove unproductive peers
We measure some things about peers (like RTT, announcements). We are not yet quantifying the badness of a peer, like using our bandwidth and CPU without producing value.
properly fix blockfetch typestate tracking
Once `eff.call(...)` returns a result that may state “the message was not actually sent”, this will require more thought on how to match the resulting type graph to the session spec: there’s a new edge from the “request sent to muxer” state back to the initial state, and the “pull” needs to be injected not when leaving the switch state `Idle` but when leaving the “Idle or Request Sent” conglomerate state. Not sure yet how to work this out.
TUI: Forging View
### Abstract
Trace Schemas for Forging
### Abstract
Discrepancy with Haskell's node on pools fallback drep
### What revision are you using?
check and improve inbound connection limit handling
Currently, the handshake concludes successfully and then the bearer is dropped if the inbound connection limit is exceeded. We should investigate how the Haskell implementation signals this to the peer and possibly improve the behaviour accordingly.
Optimize reward payouts using RocksDB merge operators
### Abstract
Add rust docs to TraceContext
As a way to solidify my understanding and softening the learning curve of future devs, the interplay between `tracing`, `opentelemetry` and serialization/deserialization (esp. with the `TraceBuffer`) should be spelt out.
Correct behavior around invalid phase two transactions
Currently, Amaru runs all validation rules when a transaction has `is_valid: false`. This is a divergence from the Haskell implementation, where only the `UTXOW` state transition rule runs (which includes the `UTXO and UTXOS` rules). The `UTXOS` rule branches on `isValid`. The state update is also different, only the UTxO state changes are persisted (and here, it is the collateral spend and return only).
Persist findings of consensus behaviour in Cardano blueprints
As discussed in Porto last week, any gaps found in the Ouroboros network spec shall eventually be filled with information on https://github.com/cardano-scaling/cardano-blueprint. This ticket serves as a reminder to myself that I should start with the notes taken in Porto from very fruitful discussions with Marcin M.
Ledger predicate: VRFKeyHashAlreadyRegistered
### Abstract
Consider memory + cpu execution units when computing the available mempool capacity
### Abstract
Support chained transactions in the mempool
### Abstract
Use a separate column family for `BEST_CHAIN_PREFIX`
### Abstract
Monitor RocksDB metrics
### Abstract
fix amaru-sim test termination
### Abstract
Cardano Node Diversity platform
### Abstract
Update the animations for the simulation
### Abstract
Add ledger checks to the simulation
### Abstract
Add the txsubmission protocol to the simulation
### Abstract
Promote downstream connections to initiator mode on incoming connections (warm -> hot peer transition)
### Abstract
Update packages metadata for v10.11.20261022
Updates packages metadata generated from the published release [`v10.11.20261022`](https://github.com/pragma-org/amaru/releases/tag/v10.11.20261022).
detect stage messaging deadlocks
Closes #1480 the Tokio runtime now detects a cycle of stages deadlocked sending to each other and kills the node with a clear diagnostic that names the stages, marks the cycle, and shows each blocked message and mailbox occupancy, while the bookkeeping stays off the uncontended send path.
chore: schedule release-begin workflow every Thursday, with 'release: true'
Typed observer hooks for embedders
Embedders have to reconstruct node health from telemetry. This is a proposition to extend `LedgerObservers` to facilitate and make more robust amaru integration.
fix: conformance reports CI pipeline
Remove `Address::Stake` Variant
This PR addresses a longstanding `TODO`, removing the `Address:Stake` variant which isn't needed. the `RewardAccount` type should be used instead, which removes some `unreachable` branches in the code. It also prevents illegal state, such as specifying a reward account in a transaction output.
chore: do not fail sync when an error is shown
Sync should fail only in case of amaru crashing. Still, display error and warnings so that run can be investigated.
chore: lib fixes
Improve lib usages, do not assume all crates always are enabled.
UPLC and ledger conformance (polish, fixes, rework)
Supersedes #1522
TUI Docs Update
This PR updates the TUI docs to reflect the latest state of the TUI, with updated images.
chore: remove dead code
Removed some dead code.
test: update the cbor dataset and fix conformance gaps
This PR selects a CBOR conformance corpus with a better coverage for the `HuddleSpec` specification and fixes a few gaps revealed by this new corpus:
Zero-denominator Conway governance parameter update is admitted to the Amaru mempool
## What revision are you using?
Support mempool submission and monitoring for library consumers
Applications embedding Amaru need to submit transactions and track mempool contents without reconstructing state from traces.
chore: wait for proper node shutdown
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
fix(node): retain shutdown ownership and allow cleanup retries
Improve shutdown API for embedders: * allow to customize shutdown timeout * allow to retry a failed shutdown * share details about why shutdown failed
chore: general bootstrap/mithril cleanup
Cleanup / leftover removals from `amaru-bootstrap` / `amaru-mithril` dureing recent refactorings.
TUI: copy text and fix on large screen
- **allow copying text from the TUI outside copy mode** - **restrict TUI to less than 65536 cells**
chore: allow to see fetch error details
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
Amaru Watermark in Protocol Minor Version
### Abstract
KES key number check
Before accepting a new KES key from the chain database, the node must check that the opcert sequence number is exactly one greater than that of the previous key.
Removes the long-lived in-memory cache of two full stake distributions; only one is needed.
Supersedes #1339 (cc @bomanaps)
fix(ledger): skip storage removal for transient DRep registrations
Reg/unreg of dreps during a single blocks shouldn't lead to persistency modification.
chore(amaru-ledger): rename option operator-cert
Rename `--operator-operational-certificate`/ `AMARU_OPERATOR_OPERATIONAL_CERTIFICATE` to `--operator-cert`/`AMARU_OPERATOR_CERT`
chore: better mithril sync
Improve both UI (CLI) and performance during mithril sync.
blockfetch initiator terminates on Done
Currently, demoted peers remain in Diffusion because the blockfetch initiator doesn’t terminate, then the group transition timeout kills the bearer after 5min. This fixes it and adds an integration test for this behaviour.
test: allow competing blocks with different parents
Fix https://github.com/pragma-org/amaru/actions/runs/38631677282/job/116084601598
fix: use ephemeral outbound ports in transaction submission E2E
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
fix: make sure amaru has proper time to cleanly shutdown
Skip-changelog
monitor remote_use and improve sharing our socket address
This change records which initiator mini-protocols the remote has opened on an established bearer, derives a temperature of none, maintenance, or diffusion, emits a remote_use trace only when that temperature changes, and shows it in the TUI. Share replies append this node’s connected-socket IP and listen port, dropping the last sampled address when the reply is already full, and staying within the requested amount and a cap of 10. Maintenance now starts keep-alive and, when the peer is advertisable, the peer-sharing initiator on an inbound duplex bearer as well as an outbound one. Peer select…
TUI improvements: 1s/3s/5s percentiles, peer scores
The score is not yet used in peer selection, that part is still awaiting a proper overhaul.
feat: tastyer keys
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit
proper linking of events and spans for header and block perf telemetry
including world loop test to assert proper OTel structure
parent span requirements can be formulated in trace schema
second step of #1422
trace schema distinguishes spans from events
first step of #1422
add facilities for logging effect timings and using them
- also adds --log-output command line argument for adding separate output streams
rework peer sharing resource to remove pure-stage messaging cycles
This change makes the consensus performance worker the single owner of peer-population state. Protocols record handshakes, closes, failed dials, applied local use, keep-alive round trips, shared addresses, and answered share requests on that worker through a new PeerTracking trait. track_peers records an intersection miss the same way and clears that
Synthetic bench for roll forward: simple transaction (1 input -> 1 output)
Exercise `State::roll_forward` against a real RocksDB-backed ledger with 10,000 seeded UTxOs and a saturated volatile window. Benchmark empty blocks and blocks with increasing counts of signed ADA-only transactions so the delta isolates per-transaction application cost from fixed block overhead. ``` roll_forward fastest │ slowest │ median │ mean │ samples │ iters ╰─ simple_transactions │ │ │ │ │
clean cli and align options with EDR-019 - guidelines for cli
This is a bit painful, but waiting longer for this is not going to help. With adoption growing and more and more docs being produced, we need to anchor the "final" api surface for users real soon. The best moment to do this was 6 months ago. The second best moment is now.
feat: add KES key generation CLI
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit
chore: logs
Some more log cleanup
chore: lower internal log level
Skip-changelog
More UPLC Improvements
This PR introduces more UPLC fixes, mostly conformance issues. All introduced test cases were run against `plutus 1.70`.
Puzzling error message
I frequently see this error message:
feat: allow to run longer nightly tests
Allow to run nightly syncs from initial bootstrap snapshot available. Introduced `amaru snapshot epochs` command to list all snapshots available.
fix: make sure that in a multi assets bundle the policies and assets are sorted
This PR fixes a `MultiAssets` regression introduced by #1400.
chore: stop sync when error happens
Skip-changelog
fix: wrong koios host for mainnet
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
make pure-stage mailbox size configurable
- enlarge for Manager and PeerSelection - remove the 10'000 override for simulation tests
chore: also test mainnet sync nightly
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
fix: lifecycle improvements
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
ensure equal semantics between Tokio and Simulation runtime
- require a timeout when the callee terminates during a call - deliver a queued call when its mailbox slot frees - time out calls that cannot enqueue
fix: make decoders conform to the cbor dataset
This PR fixes the decoding gaps exposed by the cbor dataset generated in the [`cardano-cbor-dataset` repository](https://github.com/r2rationality/cardano-cbor-dataset/pull/1).
chore: improve store errors
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit
feat: make sure progress always ends with a summary
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit
Prevent some panics in UPLC
This PR fixes 5 explicit panics that were reachable on chain, and one typo that resulted in a panic. This is not every explicit panic (`unreachable!`, `panic!`, etc.), but it is a bundle of ones that I easily reasoned about, and moves us in the right direction.
Harden some UPLC builtins and switch some dependencies in the UPLC virtual machine
Why?
Purge Mempool Transactions on Block Adoption
### Abstract
Mempool: For Some Parent and Slot, Provide a Block Body
### Abstract
remove generic_const_exprs feature
fixes #1406
Expose typed mempool services for embedders
### Abstract
Remove "Nightly Synchronization" job and simplify e2e test setup
### Abstract
Amaru for epoch-end ledger audits
First, let me thank you for Amaru. I'm currently working on [Tessera](https://tessera-preview.matthieu-pizenberg.workers.dev/), a tool implementing [CIP-179 on-chain surveys](https://github.com/cardano-foundation/CIPs/blob/master/CIP-0179/README.md). We want survey results to be reproducible and independently verifiable, so Amaru is one of the options I've explored. This is an experience report about what I did, and where I struggled.
Compilation warning
When compiling locally
bench epoch transition at scale
This pr adds a divan benchmark that seeds a RocksDB with realistic pools, UTxOs, and accounts, drives a State to the epoch boundary, and measures the full transition including the background rewards thread, pool updates, and stable flush.
TUI: clarify when tui mode shall exit and how
Currently, `amaru node run` may open the TUI and immediately close it, leaving nothing on the console.
Automatic release process for crates
In preparation for a release of the `amaru` crate on crates.io we need to install the following mechanisms:
Bootstrap should fill db with blocks
### Abstract
Optimize startup time edge case
### Abstract
Witness Set v12 Decoding
### Abstract
Flat-decoding optimizations
## Abstract
Local transaction is not being diffused, despite peers asking?
### What revision are you using?
Disentangle ledger store
### Abstract
amaru snapshot create fails with "Permission denied (os error 13)" when no --dist-dir and --snapshot-dir is given
### What revision are you using?
Optimize the download of transactions in the txsubmission protocol across several peers
### Abstract
Do not revalidate transactions that are already in the Mempool
### Abstract
Perform integrity/sanity check of the ledger & chain db
### Abstract
Enable the TraceBuffer in production
### Abstract